base64 1.0.1 → 2.0
raw patch · 24 files changed
+1974/−388 lines, 24 filesdep ~deepseqdep ~textPVP ok
version bump matches the API change (PVP)
Dependency ranges changed: deepseq, text
API changes (from Hackage documentation)
- Data.ByteString.Base64: isBase64 :: ByteString -> Bool
- Data.ByteString.Base64: isValidBase64 :: ByteString -> Bool
- Data.ByteString.Lazy.Base64: isBase64 :: ByteString -> Bool
- Data.ByteString.Lazy.Base64: isValidBase64 :: ByteString -> Bool
- Data.ByteString.Short.Base64: isBase64 :: ShortByteString -> Bool
- Data.ByteString.Short.Base64: isValidBase64 :: ShortByteString -> Bool
- Data.Text.Encoding.Base64: isBase64 :: Text -> Bool
- Data.Text.Encoding.Base64: isValidBase64 :: Text -> Bool
- Data.Text.Lazy.Encoding.Base64: isBase64 :: Text -> Bool
- Data.Text.Lazy.Encoding.Base64: isValidBase64 :: Text -> Bool
- Data.Text.Short.Encoding.Base64: isBase64 :: ShortText -> Bool
- Data.Text.Short.Encoding.Base64: isValidBase64 :: ShortText -> Bool
+ Data.ByteString.Base64: parseBase64 :: ByteString -> Either Text (Base64 'StdPadded ByteString)
+ Data.ByteString.Lazy.Base64: parseBase64 :: ByteString -> Either Text (Base64 'StdPadded ByteString)
+ Data.ByteString.Short.Base64: parseBase64 :: ShortByteString -> Either Text (Base64 'StdPadded ShortByteString)
+ Data.Text.Encoding.Base64: parseBase64 :: Text -> Either Text (Base64 'StdPadded Text)
+ Data.Text.Lazy.Encoding.Base64: parseBase64 :: Text -> Either Text (Base64 'StdPadded Text)
+ Data.Text.Short.Encoding.Base64: parseBase64 :: ShortText -> Either Text (Base64 'StdPadded ShortText)
Files
- CHANGELOG.md +22/−0
- MIGRATION-2.0.md +110/−0
- README.md +82/−1
- base64.cabal +26/−4
- benchmarks/Base64Bench.hs +88/−97
- benchmarks/simd-native.c +162/−0
- cbits/base64_simd.c +443/−0
- cbits/base64_simd.h +44/−0
- src/Data/Base64/Types.hs +3/−0
- src/Data/ByteString/Base64.hs +19/−42
- src/Data/ByteString/Base64/Internal.hs +1/−23
- src/Data/ByteString/Base64/Internal/Head.hs +29/−10
- src/Data/ByteString/Base64/Internal/SIMD.hs +46/−0
- src/Data/ByteString/Base64/Internal/W16/Loop.hs +36/−4
- src/Data/ByteString/Base64/Internal/W64/Loop.hs +24/−0
- src/Data/ByteString/Lazy/Base64.hs +22/−47
- src/Data/ByteString/Short/Base64.hs +19/−38
- src/Data/Text/Encoding/Base64.hs +22/−34
- src/Data/Text/Lazy/Encoding/Base64.hs +24/−33
- src/Data/Text/Short/Encoding/Base64.hs +22/−34
- test/Internal.hs +7/−14
- test/Main.hs +108/−7
- test/Simd.hs +235/−0
- test/simd-native.c +380/−0
CHANGELOG.md view
@@ -1,5 +1,27 @@ # Revision history for base64 +## 2.0++This release removes two public functions. See [MIGRATION-2.0.md](MIGRATION-2.0.md)+for replacement examples and the stricter validation behavior.++* Replace `isBase64` and `isValidBase64` with `parseBase64` in all six+ standard-alphabet modules. The parser returns+ `Either Text (Base64 'StdPadded a)`, preserving the original encoded input+ after checking its alphabet, length, padding, and canonicity.+* For a Boolean check, migrate to `isRight . parseBase64`. This is stricter+ than the old shape-only `isValidBase64` check. URL-specific validators+ are unchanged.+* Remove the predicates without a deprecation period. A source scan of 45+ published direct dependents found no occurrences of either function;+ private projects and other historical versions were not covered.+* Add an optional, manual `simd` flag (off by default) for bundled ARM64 NEON+ and runtime-detected x86-64 AVX2/SSSE3 encoding and strict decoding.+* The SIMD path retains scalar fallbacks, encoding and decoding behavior,+ padding/canonicity checks, and decoding error offsets.+* Add flag-on/off test and benchmark variants, SIMD boundary regressions,+ and native memory checks in CI.+ ## 1.0.1 * Support GHC 9.6 through 9.14, including `base-4.22`.
+ MIGRATION-2.0.md view
@@ -0,0 +1,110 @@+# Migrating to base64 2.0++Version 2.0 replaces the standard-alphabet predicates `isBase64` and+`isValidBase64` with `parseBase64`. The new function checks the encoded input+and returns a typed `Base64` value, or an error explaining why validation+failed. The old predicates are removed without a deprecation period.++The release also adds optional SIMD support. It is off by default, so existing+builds continue to use the scalar implementation.++## Parse externally supplied Base64++Each of these modules exports `parseBase64`, specialized to its string type:++- `Data.ByteString.Base64`+- `Data.ByteString.Lazy.Base64`+- `Data.ByteString.Short.Base64`+- `Data.Text.Encoding.Base64`+- `Data.Text.Lazy.Encoding.Base64`+- `Data.Text.Short.Encoding.Base64`++For example, the strict ByteString signature is:++```haskell+parseBase64 :: ByteString -> Either Text (Base64 'StdPadded ByteString)+```++Use the returned wrapper directly with `decodeBase64` instead of checking a+Boolean and then calling `assertBase64`:++```haskell+{-# LANGUAGE OverloadedStrings #-}++import qualified Data.ByteString.Base64 as B64++-- Right "U3Vu": the result contains the original encoded bytes.+encoded = B64.parseBase64 "U3Vu"++-- Right "Sun": decoding consumes the validated wrapper.+decoded = fmap B64.decodeBase64 encoded++-- Left "non-canonical encoding detected at offset: 2"+rejected = B64.parseBase64 "U3V="+```++Parsing preserves the input, including lazy chunk boundaries, and validates+the whole value before returning `Right`. It checks the alphabet, length,+padding, and unused padding bits. Error messages match `decodeBase64Untyped`.++## Replace Boolean checks++Import `isRight` from `Data.Either` and replace a call to `B64.isBase64 input`+with `isRight (B64.parseBase64 input)`. For a composed predicate, use+`isRight . B64.parseBase64`.++The same replacement is appropriate for `isValidBase64` when you want to check+for a decodable, canonical value. It is stricter than that function's former+alphabet-and-shape check:++| Input | Former `isBase64` | Former `isValidBase64` | `isRight . parseBase64` |+| --- | --- | --- | --- |+| `"U3Vu"` | `True` | `True` | `True` |+| `"U3V"` | `False` | `True` | `False` |+| `"U3V="` | `False` | `True` | `False` |+| `"%"` | `False` | `False` | `False` |+| `""` | `True` | `True` | `True` |++If your application intentionally accepts incomplete or non-canonical input,+keep that policy explicit; `parseBase64` does not reproduce the permissive+behavior of `isValidBase64`.++## Text and URL-safe encodings++The Text parsers validate the Base64 encoding, not the UTF-8 validity of the+decoded payload. For example, `parseBase64 "/w=="` succeeds even though its+decoded byte is not valid UTF-8. Use a ByteString decoder for arbitrary binary+payloads, or `decodeBase64UntypedWith` with a checked text conversion such as+`decodeUtf8'` when you need to report conversion errors.++The URL-specific predicates `isBase64Url` and `isValidBase64Url` retain their+existing names and behavior. The encoding and decoding APIs are unchanged.++## Enable SIMD optionally++Enable the manual Cabal flag for bundled ARM64 NEON or runtime-selected+x86-64 AVX2/SSSE3 kernels:++```sh+cabal build -fsimd+cabal test -fsimd --test-show-details=direct+```++For a project dependency, enable the flag in `cabal.project`:++```cabal+package base64+ flags: +simd+```++Unsupported targets and short inputs fall back to the scalar implementation.+No external C library is required. Padding checks, canonicity checks, and+decoding errors retain their existing behavior; lenient decoding remains+scalar. See the [README](README.md#optional-simd) for benchmark commands.++## Update package bounds++After migrating callers, update dependency bounds to admit 2.0, for example+`base64 >= 2.0 && < 2.1` or `base64 ^>= 2.0`. GHC 9.6 through 9.14 remains+the supported compiler range. Applications migrating from a pre-1.0 release+should also follow [MIGRATION-1.0.md](MIGRATION-1.0.md).
README.md view
@@ -22,8 +22,89 @@ - Better performance than `base64-bytestring` for encode and decode. - Optics for handling more complex structures with Base64 representations via the `base64-lens` package-- Checks for both validity and correctness of Base64 and Base64url encodings+- Parses standard Base64 into a validated `Base64` value, and checks Base64url encodings - Rejects non-canonical encodings that do not roundtrip in other base64 libraries like `ZE==`. The library supports GHC 9.6 through 9.14. Its dependencies are `base`, `bytestring`, `deepseq`, `text`, and `text-short`.++Upgrading to 2.0? See the [migration guide](MIGRATION-2.0.md) for the replacement+of `isBase64` and `isValidBase64`, and the [release notes](CHANGELOG.md#20) for+the optional SIMD implementation.++### Parsing existing Base64++Use `parseBase64` to validate externally supplied standard Base64 and retain the+encoded value in a typed wrapper:++```haskell+>>> import qualified Data.ByteString.Base64 as B64+>>> :set -XOverloadedStrings+>>> B64.parseBase64 "U3Vu"+Right "U3Vu"+>>> fmap B64.decodeBase64 (B64.parseBase64 "U3Vu")+Right "Sun"+>>> B64.parseBase64 "U3V="+Left "non-canonical encoding detected at offset: 2"+```++Each standard-alphabet module exports `parseBase64 :: a -> Either Text (Base64+'StdPadded a)`, specialized to its string type. Parsing checks the alphabet,+length, padding, and canonical padding bits, then returns the original encoded+input. The Text variants validate Base64 independently of the decoded payload's+UTF-8 validity; use `decodeBase64UntypedWith` to check that conversion.++`parseBase64` replaces `isBase64` and `isValidBase64`. For a Boolean check, use+`isRight . parseBase64` from `Data.Either`. Unlike the old `isValidBase64`, this+rejects missing padding and non-canonical encodings. The URL-specific validators+`isBase64Url` and `isValidBase64Url` retain their existing behavior.++### Optional SIMD++Enable the manual `simd` Cabal flag to use bundled SIMD kernels for standard+and URL-safe encoding and strict decoding, including the typed decoders:++```sh+cabal build -fsimd+cabal test -fsimd --test-show-details=direct+```++The flag is **off by default**. ARM64 uses NEON; x86-64 selects AVX2 when+the CPU and operating system support it, then SSSE3 on older processors.+Other targets and short inputs use the existing+Haskell loops. No external C library is required. Remainders, padding checks,+canonical-encoding checks, and error messages still use the Haskell code.+Lenient decoding remains scalar.++Test both configurations and compare optimized benchmark builds separately:++```sh+cabal test -f-simd --builddir=dist-scalar --test-show-details=direct+cabal test -fsimd --builddir=dist-simd --test-show-details=direct+cabal bench -f-simd --builddir=dist-scalar -O2 --benchmark-options='--csv scalar.csv'+cabal bench -fsimd --builddir=dist-simd -O2 --benchmark-options='--csv simd.csv'+```++Benchmark names identify the selected flag configuration. The suite covers+standard Base64 and padded/unpadded Base64url, encoding and typed/untyped+decoding, short and large buffers, and sliced inputs. The test suite exercises+block boundaries, scalar tails, slices, invalid input, and wrapper types in+both configurations; CI also runs native memory checks on x86-64 and ARM64,+explicitly requiring AVX2 on Linux x86-64 and checking the SSSE3 and scalar+fallbacks separately.++For native kernel measurements, a standalone benchmark validates every case+against scalar output and checks both alphabets. It times pseudorandom,+cycling, and repeated inputs at two alignments.+It reports the detected backend and writes CSV results:++```sh+cc -O2 -std=c99 -Icbits cbits/base64_simd.c benchmarks/simd-native.c -o base64-native-bench+./base64-native-bench --seconds 0.05 --rounds 5 > native.csv+```++Add `--url` to time the URL alphabet, or `--pattern random` to select one+input distribution. On x86-64, compile a second executable with+`-DHS_BASE64_SIMD_FORCE_SSSE3` to compare the older backend on the same CPU.+These measurements cover native prefixes only; use the Cabal benchmarks above+to include Haskell allocation and scalar tails.
base64.cabal view
@@ -1,6 +1,6 @@ cabal-version: 2.0 name: base64-version: 1.0.1+version: 2.0 synopsis: A modern Base64 library description: A performant, featureful RFC 4648 and 7049-compliant Base64 implementation@@ -18,10 +18,21 @@ CHANGELOG.md README.md MIGRATION-1.0.md+ MIGRATION-2.0.md tested-with: GHC ==9.6.7 || ==9.8.4 || ==9.10.3 || ==9.12.4 || ==9.14.1 +extra-source-files:+ cbits/base64_simd.h+ test/simd-native.c+ benchmarks/simd-native.c++flag simd+ description: Use bundled SIMD kernels where supported, with a scalar fallback+ default: False+ manual: True+ source-repository head type: git location: https://github.com/emilypi/base64.git@@ -53,6 +64,13 @@ Data.ByteString.Base64.Internal.W16.Loop Data.ByteString.Base64.Internal.W64.Loop + if flag(simd)+ other-modules: Data.ByteString.Base64.Internal.SIMD+ cpp-options: -DBASE64_SIMD+ c-sources: cbits/base64_simd.c+ include-dirs: cbits+ cc-options: -O2 -std=c99+ build-depends: base >=4.18 && <4.23 , bytestring >=0.11 && <0.13@@ -69,6 +87,7 @@ type: exitcode-stdio-1.0 hs-source-dirs: test other-modules: Internal+ Simd main-is: Main.hs build-depends: base >=4.18 && <4.23@@ -83,6 +102,9 @@ , text >=2.0 , text-short + if flag(simd)+ cpp-options: -DBASE64_SIMD+ ghc-options: -Wall -threaded -with-rtsopts=-N benchmark bench@@ -96,8 +118,8 @@ , base64-bytestring , bytestring >=0.11 , criterion- , deepseq- , random-bytestring- , text >=2.0++ if flag(simd)+ cpp-options: -DBASE64_SIMD ghc-options: -Wall -rtsopts
benchmarks/Base64Bench.hs view
@@ -1,106 +1,97 @@-{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE CPP #-} {-# LANGUAGE DataKinds #-}-{-# LANGUAGE FlexibleContexts #-}-{-# LANGUAGE FlexibleInstances #-}-{-# LANGUAGE GADTs #-}-{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-} {-# LANGUAGE PackageImports #-}-{-# LANGUAGE ScopedTypeVariables #-}-{-# LANGUAGE RankNTypes #-}-{-# LANGUAGE TypeFamilies #-}-module Main-( main-) where-+-- |+-- Module : Main+-- Copyright : (c) 2019-2026 Emily Pillmore+-- License : BSD-style+--+-- Maintainer : Emily Pillmore <emilypi@cohomolo.gy>+-- Stability : Experimental+-- Portability : portable+--+-- Build once with -f-simd and once with -fsimd to compare the same cases.+-- Sizes describe raw input bytes; sliced inputs also apply to decode buffers.+module Main (main) where -import Criterion import Criterion.Main--import "base64-bytestring" Data.ByteString.Base64 as Bos-import "base64" Data.ByteString.Base64 as B64-import "base64" Data.Base64.Types as B64-import Data.ByteString.Random (random)+import "base64" Data.Base64.Types (Alphabet(StdPadded), assertBase64)+import qualified Data.ByteString as BS+import qualified "base64" Data.ByteString.Base64 as B64+import qualified "base64" Data.ByteString.Base64.URL as B64U+import qualified "base64-bytestring" Data.ByteString.Base64 as Bos+import qualified "base64-bytestring" Data.ByteString.Base64.URL as BosU main :: IO ()-main =- defaultMain- [ env bs $ \ ~(bs25,bs100,bs1k,bs10k,bs100k,bs1mm) ->- bgroup "encode"- [ bgroup "25"- [ bench "base64-bytestring" $ whnf Bos.encode bs25- , bench "base64" $ whnf B64.encodeBase64' bs25- ]- , bgroup "100"- [ bench "base64-bytestring" $ whnf Bos.encode bs100- , bench "base64" $ whnf B64.encodeBase64' bs100- ]- , bgroup "1k"- [ bench "base64-bytestring" $ whnf Bos.encode bs1k- , bench "base64" $ whnf B64.encodeBase64' bs1k- ]- , bgroup "10k"- [ bench "base64-bytestring" $ whnf Bos.encode bs10k- , bench "base64" $ whnf B64.encodeBase64' bs10k- ]- , bgroup "100k"- [ bench "base64-bytestring" $ whnf Bos.encode bs100k- , bench "base64" $ whnf B64.encodeBase64' bs100k- ]- , bgroup "1mm"- [ bench "base64-bytestring" $ whnf Bos.encode bs1mm- , bench "base64" $ whnf B64.encodeBase64' bs1mm- ]- ]- , env bs' $ \ ~(bs25,bs100,bs1k,bs10k,bs100k,bs1mm) ->- bgroup "decode"- [ bgroup "25"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs25)- , bench "base64-typed" $ whnf B64.decodeBase64 bs25- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs25)- ]- , bgroup "100"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs100)- , bench "base64-typed" $ whnf B64.decodeBase64 bs100- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs100)- ]- , bgroup "1k"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs1k)- , bench "base64-typed" $ whnf B64.decodeBase64 bs1k- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs1k)- ]- , bgroup "10k"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs10k)- , bench "base64-typed" $ whnf B64.decodeBase64 bs10k- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs10k)- ]- , bgroup "100k"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs100k)- , bench "base64-typed" $ whnf B64.decodeBase64 bs100k- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs100k)- ]- , bgroup "1mm"- [ bench "base64-bytestring" $ whnf Bos.decode (B64.extractBase64 bs1mm)- , bench "base64-typed" $ whnf B64.decodeBase64 bs1mm- , bench "base64-untyped" $ whnf B64.decodeBase64Untyped (B64.extractBase64 bs1mm)- ]+main = defaultMain+ [ bgroup implementation+ [ bgroup (show size)+ [ env (inputs size offset) $ \ ~(raw, std, url, unpadded) ->+ bgroup ("offset-" ++ show offset)+ [ bgroup "standard"+ [ bgroup "encode"+ [ bench "base64-bytestring" $ nf Bos.encode raw+ , bench "base64" $ nf B64.encodeBase64' raw+ ]+ , bgroup "decode"+ [ bench "base64-bytestring" $ nf Bos.decode std+ , bench "base64-typed" $+ nf (B64.decodeBase64 . assertBase64 @'StdPadded) std+ , bench "base64-untyped" $ nf B64.decodeBase64Untyped std+ ]+ ]+ , bgroup "url-padded"+ [ bgroup "encode"+ [ bench "base64-bytestring" $ nf BosU.encode raw+ , bench "base64" $ nf B64U.encodeBase64' raw+ ]+ , bgroup "decode"+ [ bench "base64-bytestring" $ nf BosU.decodePadded url+ , bench "base64-typed" $+ nf (B64U.decodeBase64Padded . assertBase64) url+ , bench "base64-untyped" $ nf B64U.decodeBase64PaddedUntyped url+ ]+ ]+ , bgroup "url-unpadded"+ [ bgroup "encode"+ [ bench "base64-bytestring" $ nf BosU.encodeUnpadded raw+ , bench "base64" $ nf B64U.encodeBase64Unpadded' raw+ ]+ , bgroup "decode"+ [ bench "base64-bytestring" $ nf BosU.decodeUnpadded unpadded+ , bench "base64-typed" $+ nf (B64U.decodeBase64Unpadded . assertBase64) unpadded+ , bench "base64-untyped" $+ nf B64U.decodeBase64UnpaddedUntyped unpadded+ ]+ ]+ ]+ | offset <- [0, 1]+ ]+ | size <- [11, 12, 24, 32, 48, 64, 128, 4096, 1048576] ]- ]- where- bs = do- a <- random 25- b <- random 100- c <- random 1000- d <- random 10000- e <- random 100000- f <- random 1000000- return (a,b,c,d,e,f)+ ] - bs' = do- a <- B64.encodeBase64' <$> random 25- b <- B64.encodeBase64' <$> random 100- c <- B64.encodeBase64' <$> random 1000- d <- B64.encodeBase64' <$> random 10000- e <- B64.encodeBase64' <$> random 100000- f <- B64.encodeBase64' <$> random 1000000- return (a,b,c,d,e,f)+implementation :: String+#ifdef BASE64_SIMD+implementation = "simd"+#else+implementation = "scalar"+#endif++-- Criterion forces this environment before timing. The exact same deterministic+-- data is used for both flag settings, and encoding is excluded from decoding+-- measurements. Reference encodings supply the known-valid typed inputs.+inputs :: Int -> Int -> IO (BS.ByteString, BS.ByteString, BS.ByteString, BS.ByteString)+inputs size offset = pure+ ( sliced raw+ , sliced (Bos.encode raw)+ , sliced (BosU.encode raw)+ , sliced (BosU.encodeUnpadded raw)+ )+ where+ raw = BS.pack (take size (cycle [0 .. 255]))+ sliced bytes = BS.take (BS.length bytes) . BS.drop offset $+ BS.replicate offset 0xe5 <> bytes <> BS.replicate 32 0xff
+ benchmarks/simd-native.c view
@@ -0,0 +1,162 @@+/* Copyright (c) 2026, Emily Pillmore. BSD-3-Clause; see LICENSE. */+/* Standalone native-prefix timing; no full codec is used in timed regions.+ * Build: cc -O2 -std=c99 -Icbits benchmarks/simd-native.c cbits/base64_simd.c -o simd-native-bench+ * Every case is validated against independent scalar bytes before timing.+ */+#define _POSIX_C_SOURCE 200809L+#include <stdio.h>+#include <stdlib.h>+#include <stdint.h>+#include <string.h>+#include <time.h>+#include "base64_simd.h"++typedef size_t (*prefix_fn)(const uint8_t *, size_t, uint8_t *, const uint8_t *);+struct variant { const char *name; prefix_fn encode; prefix_fn decode; };+static const struct variant variants[] = {+ {"native", hs_base64_simd_encode, hs_base64_simd_decode}+};+#define COUNT (sizeof(variants) / sizeof(variants[0]))+static const uint8_t alphabets[2][65] = {+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"+};+static volatile size_t sink;++static void require(int condition, const char *message)+{+ if (!condition) { fprintf(stderr, "%s\n", message); exit(1); }+}+static double now(void)+{+ struct timespec t;+ require(clock_gettime(CLOCK_MONOTONIC, &t) == 0, "clock_gettime failed");+ return (double)t.tv_sec + (double)t.tv_nsec * 1e-9;+}+static void fixture(uint8_t *p, size_t len, unsigned kind)+{+ uint64_t state = UINT64_C(0x243f6a8885a308d3);+ for (size_t i = 0; i < len; i++) {+ state ^= state << 13; state ^= state >> 7; state ^= state << 17;+ p[i] = kind == 0 ? (uint8_t)state : kind == 1 ? (uint8_t)i : 90;+ }+}+static void reference_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+ for (size_t i = 0; i + 2 < len; i += 3) {+ const uint32_t n = (uint32_t)src[i] << 16 | (uint32_t)src[i + 1] << 8 | src[i + 2];+ *dst++ = alphabet[n >> 18]; *dst++ = alphabet[(n >> 12) & 63];+ *dst++ = alphabet[(n >> 6) & 63]; *dst++ = alphabet[n & 63];+ }+}+static double timed(prefix_fn fn, const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table, size_t iterations)+{+ size_t total = 0;+ const double start = now();+ for (size_t i = 0; i < iterations; i++) total += fn(src, len, dst, table);+ const double elapsed = now() - start;+ sink += total + dst[0];+ return elapsed;+}+static size_t calibrate(prefix_fn fn, const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table, double target)+{+ size_t iterations = 1;+ double elapsed;+ do {+ elapsed = timed(fn, src, len, dst, table, iterations);+ if (elapsed < 0.003) iterations *= 2;+ } while (elapsed < 0.003);+ return (size_t)((double)iterations * target / elapsed) + 1;+}+int main(int argc, char **argv)+{+ double seconds = 0.02;+ unsigned rounds = 3, timed_alphabet = 0;+ const char *only_pattern = NULL;+ const size_t sizes[] = {128, 4096, 1048576};+ const char *patterns[] = {"random", "cycle256", "repeated"};+ for (int i = 1; i < argc; i++) {+ if (strcmp(argv[i], "--seconds") == 0 && i + 1 < argc) seconds = strtod(argv[++i], NULL);+ else if (strcmp(argv[i], "--rounds") == 0 && i + 1 < argc) rounds = (unsigned)strtoul(argv[++i], NULL, 10);+ else if (strcmp(argv[i], "--pattern") == 0 && i + 1 < argc) only_pattern = argv[++i];+ else if (strcmp(argv[i], "--url") == 0) timed_alphabet = 1;+ else { fprintf(stderr, "usage: %s [--seconds 0.02] [--rounds 3] [--url] [--pattern random|cycle256|repeated]\n", argv[0]); return 1; }+ }+ require(seconds >= 0.001 && seconds <= 10 && rounds > 0 && rounds <= 50, "invalid timing options");+ if (only_pattern) {+ int found = 0;+ for (size_t i = 0; i < 3; i++) found |= strcmp(only_pattern, patterns[i]) == 0;+ require(found, "unknown --pattern (use random, cycle256, or repeated)");+ }+ const int backend = hs_base64_simd_backend();+ fprintf(stderr, "native backend=%d (0 scalar, 1 SSSE3, 2 NEON, 3 AVX2)\n", backend);+ require(backend != 0, "no native SIMD backend is available to time");++ puts("variant,operation,alphabet,data,raw_bytes,input_bytes,consumed,offset,round,iterations,ns_per_call,input_gib_per_second");+ for (unsigned kind = 0; kind < 3; kind++) {+ if (only_pattern && strcmp(only_pattern, patterns[kind]) != 0) continue;+ for (size_t n = 0; n < sizeof(sizes) / sizeof(sizes[0]); n++) {+ const size_t len = sizes[n], encoded_len = len / 3 * 4 + 4;+ for (size_t offset = 0; offset <= 1; offset++) {+ uint8_t *raw_base = malloc(len + 2), *encoded_base = malloc(encoded_len + 2);+ uint8_t *output_base = malloc(encoded_len + 34), *reference = malloc(encoded_len);+ require(raw_base && encoded_base && output_base && reference, "allocation failed");+ uint8_t *raw = raw_base + offset, *encoded = encoded_base + offset;+ uint8_t *output = output_base + offset;+ fixture(raw, len, kind);+ /* Validate both alphabets even if timing only one. */+ for (unsigned alphabet = 0; alphabet < 2; alphabet++) {+ uint8_t table[256];+ memset(table, 255, sizeof(table));+ for (size_t j = 0; j < 64; j++) table[alphabets[alphabet][j]] = (uint8_t)j;+ table['='] = 64;+ reference_encode(raw, len, encoded, alphabets[alphabet]);+ memcpy(reference, encoded, encoded_len - 4);+ memcpy(encoded + encoded_len - 4, "AAAA", 4);+ for (unsigned operation = 0; operation < 2; operation++) {+ const uint8_t *src = operation ? encoded : raw;+ const uint8_t *lookup = operation ? table : alphabets[alphabet];+ const size_t input_len = operation ? encoded_len : len;+ const uint8_t *expected = operation ? raw : reference;+ size_t consumed[COUNT] = {0}, iterations[COUNT] = {0};+ for (size_t v = 0; v < COUNT; v++) {+ prefix_fn fn = operation ? variants[v].decode : variants[v].encode;+ memset(output_base, 0xa5, encoded_len + 34);+ consumed[v] = fn(src, input_len, output, lookup);+ const size_t written = operation ? consumed[v] / 4 * 3 : consumed[v] / 3 * 4;+ require(consumed[v] <= input_len && consumed[v] % (operation ? 4 : 3) == 0,+ "invalid prefix return");+ require(!operation || input_len - consumed[v] >= 4, "final quartet consumed");+ require(memcmp(output, expected, written) == 0, "incorrect prefix bytes");+ for (size_t j = 0; j < offset; j++) require(output_base[j] == 0xa5, "leading canary changed");+ for (size_t j = written; j < encoded_len + 32; j++) require(output[j] == 0xa5, "trailing canary changed");+ if (alphabet == timed_alphabet && consumed[v] != 0)+ iterations[v] = calibrate(fn, src, input_len, output, lookup, seconds);+ }+ if (alphabet != timed_alphabet) continue;+ for (unsigned round = 0; round < rounds; round++) {+ /* Rotate implementation order to limit drift bias. */+ for (size_t order = 0; order < COUNT; order++) {+ const size_t v = (order + round) % COUNT;+ if (consumed[v] == 0) continue;+ prefix_fn fn = operation ? variants[v].decode : variants[v].encode;+ const double elapsed = timed(fn, src, input_len, output, lookup, iterations[v]);+ const double ns = elapsed * 1e9 / iterations[v];+ printf("%s,%s,%s,%s,%zu,%zu,%zu,%zu,%u,%zu,%.6f,%.6f\n",+ variants[v].name, operation ? "decode" : "encode", alphabet ? "url" : "standard",+ patterns[kind], len, input_len, consumed[v], offset, round, iterations[v], ns,+ consumed[v] / ns * 1e9 / 1073741824.0);+ }+ }+ }+ }+ free(reference); free(output_base); free(encoded_base); free(raw_base);+ }+ }+ }+ fprintf(stderr, "validated both alphabets; checksum=%zu\n", sink);+ return 0;+}
+ cbits/base64_simd.c view
@@ -0,0 +1,443 @@+/* Copyright (c) 2026, Emily Pillmore. BSD-3-Clause; see LICENSE. */++#include <string.h>++#include "base64_simd.h"++/*+ * These are original prefix kernels. No external SIMD library is required.+ * Target-specific instructions are confined to their implementation below;+ * x86 builds need no global ISA flags, and dispatch has no mutable state.+ * HS_BASE64_SIMD_FORCE_SCALAR exercises the unsupported-target fallback.+ * HS_BASE64_SIMD_FORCE_SSSE3 caps x86 dispatch for testing older processors.+ */+#if !defined(HS_BASE64_SIMD_FORCE_SCALAR)+#if defined(__aarch64__) && defined(__ARM_NEON) && \+ (defined(__GNUC__) || defined(__clang__))+#define HS_BASE64_NEON 1+#include <arm_neon.h>+#elif defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__))+#define HS_BASE64_SSSE3 1+#include <immintrin.h>+#define HS_BASE64_TARGET_SSSE3 __attribute__((target("ssse3")))+#if !defined(HS_BASE64_SIMD_FORCE_SSSE3)+#define HS_BASE64_AVX2 1+#define HS_BASE64_TARGET_AVX2 __attribute__((target("avx2")))+#endif+#endif+#endif++#if defined(HS_BASE64_NEON) || defined(HS_BASE64_SSSE3)+/*+ * Nibble-table classification, following the lookup approach used by Klomp+ * and simdutf. These masks are derived directly from our two alphabets:+ * bit h in valid_lo[l] says that byte (h << 4) | l is a valid character.+ * valid_hi[h] selects that bit for h = 2..7, and is zero otherwise.+ * Their intersection is nonzero exactly for an alphabet character, including+ * neither padding nor bytes >= 128. Rows are standard Base64, then Base64url.+ */+static const uint8_t hs_base64_valid_lo[2][16] = {+ {0xa8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8,+ 0xf8, 0xf8, 0xf0, 0x54, 0x50, 0x50, 0x50, 0x54},+ {0xa8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8, 0xf8,+ 0xf8, 0xf8, 0xf0, 0x50, 0x50, 0x54, 0x50, 0x70}+};+static const uint8_t hs_base64_valid_hi[16] = {+ 0, 0, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0, 0, 0, 0, 0, 0, 0, 0+};++/*+ * Add these offsets modulo 256 to ASCII to obtain its sextet. The only+ * exception in each alphabet is its last character: '/' needs -3 after the+ * '+' offset; '_' needs +33 after the uppercase offset. Invalid lanes are+ * checked separately, so their translated values are never stored.+ */+static const uint8_t hs_base64_offsets[2][16] = {+ {0, 0, 62 - '+', 52 - '0', 256 - 'A', 256 - 'A',+ 256 + 26 - 'a', 256 + 26 - 'a', 0, 0, 0, 0, 0, 0, 0, 0},+ {0, 0, 62 - '-', 52 - '0', 256 - 'A', 256 - 'A',+ 256 + 26 - 'a', 256 + 26 - 'a', 0, 0, 0, 0, 0, 0, 0, 0}+};+#endif++#if defined(HS_BASE64_NEON)++static size_t hs_base64_neon_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+ size_t consumed = 0;+ uint8x16x4_t lookup;+ const uint8x16_t mask = vdupq_n_u8(63);++ if (len < 48) {+ return 0;+ }+ lookup.val[0] = vld1q_u8(alphabet);+ lookup.val[1] = vld1q_u8(alphabet + 16);+ lookup.val[2] = vld1q_u8(alphabet + 32);+ lookup.val[3] = vld1q_u8(alphabet + 48);++ while (len - consumed >= 48) {+ /* Deinterleave sixteen triples without reading beyond their 48 bytes a la Klomp */+ const uint8x16x3_t input = vld3q_u8(src + consumed);+ uint8x16x4_t output;+ const uint8x16_t a = vshrq_n_u8(input.val[0], 2);+ const uint8x16_t b = vandq_u8(mask, vorrq_u8(vshlq_n_u8(input.val[0], 4),+ vshrq_n_u8(input.val[1], 4)));+ const uint8x16_t c = vandq_u8(mask, vorrq_u8(vshlq_n_u8(input.val[1], 2),+ vshrq_n_u8(input.val[2], 6)));+ const uint8x16_t d = vandq_u8(mask, input.val[2]);+ output.val[0] = vqtbl4q_u8(lookup, a);+ output.val[1] = vqtbl4q_u8(lookup, b);+ output.val[2] = vqtbl4q_u8(lookup, c);+ output.val[3] = vqtbl4q_u8(lookup, d);+ vst4q_u8(dst, output);+ consumed += 48;+ dst += 64;+ }+ return consumed;+}++static inline uint8x16_t hs_base64_neon_sextets(uint8x16_t input,+ uint8x16_t valid_lo, uint8x16_t valid_hi, uint8x16_t offsets,+ uint8x16_t last_char, uint8x16_t last_adjustment, uint8x16_t *valid)+{+ const uint8x16_t lo = vandq_u8(input, vdupq_n_u8(15));+ const uint8x16_t hi = vshrq_n_u8(input, 4);+ const uint8x16_t adjustment = vandq_u8(vceqq_u8(input, last_char), last_adjustment);++ *valid = vandq_u8(vqtbl1q_u8(valid_lo, lo), vqtbl1q_u8(valid_hi, hi));+ return vaddq_u8(input, vaddq_u8(vqtbl1q_u8(offsets, hi), adjustment));+}++static size_t hs_base64_neon_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table)+{+ size_t consumed = 0;++ if (len < 68) {+ return 0;+ }+ const int is_url = table['-'] == 62;+ const uint8x16_t valid_lo = vld1q_u8(hs_base64_valid_lo[is_url]);+ const uint8x16_t valid_hi = vld1q_u8(hs_base64_valid_hi);+ const uint8x16_t offsets = vld1q_u8(hs_base64_offsets[is_url]);+ const uint8x16_t last_char = vdupq_n_u8(is_url ? '_' : '/');+ const uint8x16_t last_adjustment = vdupq_n_u8((uint8_t)(is_url ? 33 : -3));++ /* Always reserve the last quartet for Haskell's padding/canonical checks not ala Klomp */+ while (len - consumed >= 68) {+ const uint8x16x4_t input = vld4q_u8(src + consumed);+ uint8x16_t valid_a, valid_b, valid_c, valid_d;+ const uint8x16_t a = hs_base64_neon_sextets(input.val[0], valid_lo, valid_hi,+ offsets, last_char, last_adjustment, &valid_a);+ const uint8x16_t b = hs_base64_neon_sextets(input.val[1], valid_lo, valid_hi,+ offsets, last_char, last_adjustment, &valid_b);+ const uint8x16_t c = hs_base64_neon_sextets(input.val[2], valid_lo, valid_hi,+ offsets, last_char, last_adjustment, &valid_c);+ const uint8x16_t d = hs_base64_neon_sextets(input.val[3], valid_lo, valid_hi,+ offsets, last_char, last_adjustment, &valid_d);+ const uint8x16_t valid = vminq_u8(vminq_u8(valid_a, valid_b),+ vminq_u8(valid_c, valid_d));+ uint8x16x3_t output;++ if (vminvq_u8(valid) == 0) {+ break;+ }+ output.val[0] = vorrq_u8(vshlq_n_u8(a, 2), vshrq_n_u8(b, 4));+ output.val[1] = vorrq_u8(vshlq_n_u8(b, 4), vshrq_n_u8(c, 2));+ output.val[2] = vorrq_u8(vshlq_n_u8(c, 6), d);+ vst3q_u8(dst, output);+ consumed += 64;+ dst += 48;+ }+ return consumed;+}++#elif defined(HS_BASE64_SSSE3)++HS_BASE64_TARGET_SSSE3+static size_t hs_base64_ssse3_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+ size_t consumed = 0;+ __m128i lookup[4];+ const __m128i gather = _mm_setr_epi8(2, 1, 0, -1, 5, 4, 3, -1,+ 8, 7, 6, -1, 11, 10, 9, -1);++ if (len < 12) {+ return 0;+ }+ for (size_t i = 0; i < 4; i++) {+ lookup[i] = _mm_loadu_si128((const __m128i *)(alphabet + i * 16));+ }++ while (len - consumed >= 12) {+ __m128i input = _mm_setzero_si128();+ __m128i indices;+ __m128i high;+ __m128i output = _mm_setzero_si128();++ /* memcpy expresses a 12-byte load, including at the buffer boundary. */+ memcpy(&input, src + consumed, 12);+ input = _mm_shuffle_epi8(input, gather);+ /* Each 32-bit lane is now a big-endian 24-bit input triple. */+ indices = _mm_and_si128(_mm_srli_epi32(input, 18), _mm_set1_epi32(0x0000003f));+ indices = _mm_or_si128(indices,+ _mm_and_si128(_mm_srli_epi32(input, 4), _mm_set1_epi32(0x00003f00)));+ indices = _mm_or_si128(indices,+ _mm_and_si128(_mm_slli_epi32(input, 10), _mm_set1_epi32(0x003f0000)));+ indices = _mm_or_si128(indices,+ _mm_and_si128(_mm_slli_epi32(input, 24), _mm_set1_epi32(0x3f000000)));+ high = _mm_and_si128(indices, _mm_set1_epi8(0x30));+ for (size_t i = 0; i < 4; i++) {+ const __m128i matches = _mm_cmpeq_epi8(high, _mm_set1_epi8((char)(i * 16)));+ output = _mm_or_si128(output,+ _mm_and_si128(matches, _mm_shuffle_epi8(lookup[i], indices)));+ }+ _mm_storeu_si128((__m128i *)dst, output);+ consumed += 12;+ dst += 16;+ }+ return consumed;+}++HS_BASE64_TARGET_SSSE3+static size_t hs_base64_ssse3_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table)+{+ size_t consumed = 0;+ const __m128i scatter = _mm_setr_epi8(2, 1, 0, 6, 5, 4, 10, 9,+ 8, 14, 13, 12, -1, -1, -1, -1);++ if (len < 20) {+ return 0;+ }+ const int is_url = table['-'] == 62;+ const __m128i valid_lo = _mm_loadu_si128((const __m128i *)hs_base64_valid_lo[is_url]);+ const __m128i valid_hi = _mm_loadu_si128((const __m128i *)hs_base64_valid_hi);+ const __m128i offsets = _mm_loadu_si128((const __m128i *)hs_base64_offsets[is_url]);+ const __m128i last_char = _mm_set1_epi8(is_url ? '_' : '/');+ const __m128i last_adjustment = _mm_set1_epi8(is_url ? 33 : -3);+ const __m128i nibble_mask = _mm_set1_epi8(15);++ while (len - consumed >= 20) {+ const __m128i input = _mm_loadu_si128((const __m128i *)(src + consumed));+ const __m128i lo = _mm_and_si128(input, nibble_mask);+ const __m128i hi = _mm_and_si128(_mm_srli_epi16(input, 4), nibble_mask);+ const __m128i valid = _mm_and_si128(_mm_shuffle_epi8(valid_lo, lo),+ _mm_shuffle_epi8(valid_hi, hi));+ const __m128i adjustment = _mm_and_si128(_mm_cmpeq_epi8(input, last_char),+ last_adjustment);+ __m128i value;+ __m128i output;++ if (_mm_movemask_epi8(_mm_cmpeq_epi8(valid, _mm_setzero_si128())) != 0) {+ break;+ }+ value = _mm_add_epi8(input,+ _mm_add_epi8(_mm_shuffle_epi8(offsets, hi), adjustment));++ /* Adjacent sextets become 12-bit pairs, then 24-bit triples. */+ output = _mm_maddubs_epi16(value, _mm_set1_epi32(0x01400140));+ output = _mm_madd_epi16(output, _mm_set1_epi32(0x00011000));+ output = _mm_shuffle_epi8(output, scatter);+ /* A 16-byte store would overwrite Haskell's uncommitted suffix. */+ memcpy(dst, &output, 12);+ consumed += 16;+ dst += 12;+ }+ return consumed;+}++#if defined(HS_BASE64_AVX2)++HS_BASE64_TARGET_AVX2+static inline __m128i hs_base64_avx2_load12(const uint8_t *src)+{+ int64_t first;+ int32_t last;+ memcpy(&first, src, sizeof(first));+ memcpy(&last, src + 8, sizeof(last));+ return _mm_insert_epi32(_mm_cvtsi64_si128(first), last, 2);+}++HS_BASE64_TARGET_AVX2+static inline void hs_base64_avx2_store12(uint8_t *dst, __m128i value)+{+ const int64_t first = _mm_cvtsi128_si64(value);+ const int32_t last = _mm_extract_epi32(value, 2);+ memcpy(dst, &first, sizeof(first));+ memcpy(dst + 8, &last, sizeof(last));+}++HS_BASE64_TARGET_AVX2+static size_t hs_base64_avx2_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+ size_t consumed = 0;+ __m256i lookup[4];+ const __m256i gather = _mm256_broadcastsi128_si256(+ _mm_setr_epi8(2, 1, 0, -1, 5, 4, 3, -1, 8, 7, 6, -1, 11, 10, 9, -1));++ if (len < 24) {+ return 0;+ }+ for (size_t i = 0; i < 4; i++) {+ lookup[i] = _mm256_broadcastsi128_si256(+ _mm_loadu_si128((const __m128i *)(alphabet + i * 16)));+ }++ while (len - consumed >= 24) {+ const __m128i first = hs_base64_avx2_load12(src + consumed);+ const __m128i second = hs_base64_avx2_load12(src + consumed + 12);+ __m256i input;+ __m256i indices;+ __m256i high;+ __m256i output = _mm256_setzero_si256();++ /* Each 128-bit lane receives exactly four triples; neither load is wide. */+ input = _mm256_inserti128_si256(_mm256_castsi128_si256(first), second, 1);+ input = _mm256_shuffle_epi8(input, gather);+ indices = _mm256_and_si256(_mm256_srli_epi32(input, 18),+ _mm256_set1_epi32(0x0000003f));+ indices = _mm256_or_si256(indices,+ _mm256_and_si256(_mm256_srli_epi32(input, 4), _mm256_set1_epi32(0x00003f00)));+ indices = _mm256_or_si256(indices,+ _mm256_and_si256(_mm256_slli_epi32(input, 10), _mm256_set1_epi32(0x003f0000)));+ indices = _mm256_or_si256(indices,+ _mm256_and_si256(_mm256_slli_epi32(input, 24), _mm256_set1_epi32(0x3f000000)));+ high = _mm256_and_si256(indices, _mm256_set1_epi8(0x30));+ for (size_t i = 0; i < 4; i++) {+ const __m256i matches = _mm256_cmpeq_epi8(high, _mm256_set1_epi8((char)(i * 16)));+ output = _mm256_or_si256(output,+ _mm256_and_si256(matches, _mm256_shuffle_epi8(lookup[i], indices)));+ }+ _mm256_storeu_si256((__m256i *)dst, output);+ consumed += 24;+ dst += 32;+ }+ return consumed;+}++HS_BASE64_TARGET_AVX2+static size_t hs_base64_avx2_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table)+{+ size_t consumed = 0;+ const __m256i scatter = _mm256_broadcastsi128_si256(+ _mm_setr_epi8(2, 1, 0, 6, 5, 4, 10, 9, 8, 14, 13, 12, -1, -1, -1, -1));++ if (len < 36) {+ return 0;+ }+ const int is_url = table['-'] == 62;+ const __m256i valid_lo = _mm256_broadcastsi128_si256(+ _mm_loadu_si128((const __m128i *)hs_base64_valid_lo[is_url]));+ const __m256i valid_hi = _mm256_broadcastsi128_si256(+ _mm_loadu_si128((const __m128i *)hs_base64_valid_hi));+ const __m256i offsets = _mm256_broadcastsi128_si256(+ _mm_loadu_si128((const __m128i *)hs_base64_offsets[is_url]));+ const __m256i last_char = _mm256_set1_epi8(is_url ? '_' : '/');+ const __m256i last_adjustment = _mm256_set1_epi8(is_url ? 33 : -3);+ const __m256i nibble_mask = _mm256_set1_epi8(15);++ while (len - consumed >= 36) {+ const __m256i input = _mm256_loadu_si256((const __m256i *)(src + consumed));+ const __m256i lo = _mm256_and_si256(input, nibble_mask);+ const __m256i hi = _mm256_and_si256(_mm256_srli_epi16(input, 4), nibble_mask);+ const __m256i valid = _mm256_and_si256(_mm256_shuffle_epi8(valid_lo, lo),+ _mm256_shuffle_epi8(valid_hi, hi));+ const __m256i adjustment = _mm256_and_si256(_mm256_cmpeq_epi8(input, last_char),+ last_adjustment);+ __m256i value;+ __m256i output;+ __m128i first;+ __m128i second;++ /* Validate both lanes before committing any bytes from this block. */+ if (_mm256_movemask_epi8(_mm256_cmpeq_epi8(valid, _mm256_setzero_si256())) != 0) {+ break;+ }+ value = _mm256_add_epi8(input,+ _mm256_add_epi8(_mm256_shuffle_epi8(offsets, hi), adjustment));+ output = _mm256_maddubs_epi16(value, _mm256_set1_epi32(0x01400140));+ output = _mm256_madd_epi16(output, _mm256_set1_epi32(0x00011000));+ output = _mm256_shuffle_epi8(output, scatter);+ first = _mm256_castsi256_si128(output);+ second = _mm256_extracti128_si256(output, 1);+ /* Store exactly 24 bytes, with no gaps or overlapping speculative bytes. */+ hs_base64_avx2_store12(dst, first);+ hs_base64_avx2_store12(dst + 12, second);+ consumed += 32;+ dst += 24;+ }+ return consumed;+}++#endif /* HS_BASE64_AVX2 */++#endif++int hs_base64_simd_backend(void)+{+#if defined(HS_BASE64_NEON)+ return 2;+#elif defined(HS_BASE64_SSSE3)+#if defined(HS_BASE64_AVX2)+ /* The compiler's CPU query also checks OS support for the AVX register state. */+ if (__builtin_cpu_supports("avx2")) {+ return 3;+ }+#endif+ return __builtin_cpu_supports("ssse3") ? 1 : 0;+#else+ return 0;+#endif+}++size_t hs_base64_simd_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+#if defined(HS_BASE64_NEON)+ return hs_base64_neon_encode(src, len, dst, alphabet);+#elif defined(HS_BASE64_SSSE3)+ const int backend = hs_base64_simd_backend();+#if defined(HS_BASE64_AVX2)+ if (backend == 3) {+ return hs_base64_avx2_encode(src, len, dst, alphabet);+ }+#endif+ if (backend == 1) {+ return hs_base64_ssse3_encode(src, len, dst, alphabet);+ }+#endif+ (void)src;+ (void)len;+ (void)dst;+ (void)alphabet;+ return 0;+}++size_t hs_base64_simd_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table)+{+#if defined(HS_BASE64_NEON)+ return hs_base64_neon_decode(src, len, dst, table);+#elif defined(HS_BASE64_SSSE3)+ const int backend = hs_base64_simd_backend();+#if defined(HS_BASE64_AVX2)+ if (backend == 3) {+ return hs_base64_avx2_decode(src, len, dst, table);+ }+#endif+ if (backend == 1) {+ return hs_base64_ssse3_decode(src, len, dst, table);+ }+#endif+ (void)src;+ (void)len;+ (void)dst;+ (void)table;+ return 0;+}
+ cbits/base64_simd.h view
@@ -0,0 +1,44 @@+/* Copyright (c) 2026, Emily Pillmore. BSD-3-Clause; see LICENSE. */++#ifndef HS_BASE64_SIMD_H+#define HS_BASE64_SIMD_H++#include <stddef.h>+#include <stdint.h>++#ifdef __cplusplus+extern "C" {+#endif++/*+ * Process a prefix only; the Haskell implementation handles the remaining+ * bytes, padding, and error reporting. Source and destination must not overlap.+ * Source loads remain inside len; destination stores cover only the produced+ * prefix. Alphabet/table pointers must identify the complete tables below.+ *+ * Encode returns a multiple of 3 and writes exactly (result / 3) * 4 bytes.+ * Alphabet points to the existing 64-byte standard or URL alphabet.+ *+ * Decode returns a multiple of 4, leaves at least the final 4 source bytes,+ * and writes exactly (result / 4) * 3 bytes. A vector block containing an+ * invalid byte or padding is left completely untouched. Table points to the+ * existing 256-byte decoding table: table['-'] == 62 selects URL decoding.+ *+ * Unsupported targets return zero without reading or writing any buffers.+ */+size_t hs_base64_simd_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet);+size_t hs_base64_simd_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table);++/*+ * 0: scalar fallback; 1: x86-64 SSSE3; 2: ARM64 NEON; 3: x86-64 AVX2.+ * x86 features are detected at runtime; AVX2 is preferred when supported.+ */+int hs_base64_simd_backend(void);++#ifdef __cplusplus+}+#endif++#endif /* HS_BASE64_SIMD_H */
src/Data/Base64/Types.hs view
@@ -38,6 +38,9 @@ -- | Assert the provenance of a value encoded in a particular -- base64 alphabet. --+-- For untrusted standard Base64 input, use the @parseBase64@ function+-- in the corresponding ByteString or Text module to validate it first.+-- -- /Warning/: This is a blind assertion that a particular -- value is base64 encoded in some alphabet. If you are not -- sure of the provenance of the value, you may experience
src/Data/ByteString/Base64.hs view
@@ -23,18 +23,15 @@ , decodeBase64 , decodeBase64Untyped , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where import Data.Base64.Types import Data.ByteString.Internal (ByteString(..))-import Data.ByteString.Base64.Internal import Data.ByteString.Base64.Internal.Head import Data.ByteString.Base64.Internal.Tables-import Data.Either (isRight) import Data.Text (Text) import qualified Data.Text.Encoding as T @@ -133,49 +130,29 @@ decodeBase64Lenient = decodeBase64Lenient_ decodeB64Table {-# inline decodeBase64Lenient #-} --- | Tell whether a 'ByteString' value is base64 encoded.+-- | Parse a canonical, padded Base64-encoded 'ByteString' value. ----- This function will also detect non-canonical encodings such as @ZE==@, which are--- externally valid Base64-encoded values, but are internally inconsistent "impossible"--- values.+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. -- -- === __Examples__: ----- >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False------ >>> isBase64 "U3V="--- False----isBase64 :: ByteString -> Bool-isBase64 bs- = isValidBase64 bs- && isRight (decodeBase64Untyped bs)-{-# inline isBase64 #-}---- | Tell whether a 'ByteString' value is a valid Base64 format.------ This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'ByteString' value, use 'isBase64'.------ === __Examples__:+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3Vu"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V"--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ----- >>> isValidBase64 "%"--- False+-- @since 2.0 ---isValidBase64 :: ByteString -> Bool-isValidBase64 = validateBase64 "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"-{-# inline isValidBase64 #-}+parseBase64 :: ByteString -> Either Text (Base64 'StdPadded ByteString)+parseBase64 bs = assertBase64 bs <$ decodeBase64Untyped bs+{-# INLINE parseBase64 #-}
src/Data/ByteString/Base64/Internal.hs view
@@ -13,8 +13,7 @@ -- processes and tables. -- module Data.ByteString.Base64.Internal-( validateBase64-, validateBase64Url+( validateBase64Url , validateLastPad ) where @@ -28,27 +27,6 @@ import Foreign.Storable import System.IO.Unsafe---- | Given a bytestring, check to see that it conforms to a given alphabet----validateBase64 :: ByteString -> ByteString -> Bool-validateBase64 !alphabet (PS !fp !off !l) =- accursedUnutterablePerformIO $ withForeignPtr fp $ \p ->- go (plusPtr p off) (plusPtr p (l + off))- where- go !p !end- | p == end = return True- | otherwise = do- w <- peek p-- let f a- | a == 0x3d, plusPtr p 1 == end = True- | a == 0x3d, plusPtr p 2 == end = True- | a == 0x3d = False- | otherwise = BS.elem a alphabet-- if f w then go (plusPtr p 1) end else return False-{-# INLINE validateBase64 #-} validateBase64Url :: ByteString -> ByteString -> Bool validateBase64Url !alphabet bs@(PS _ _ l)
src/Data/ByteString/Base64/Internal/Head.hs view
@@ -24,6 +24,9 @@ import Data.ByteString.Base64.Internal.Tail import Data.ByteString.Base64.Internal.Utils import Data.ByteString.Base64.Internal.W64.Loop+#ifdef BASE64_SIMD+import Data.ByteString.Base64.Internal.SIMD+#endif import Data.ByteString.Internal import Data.Text (Text) @@ -44,10 +47,15 @@ withForeignPtr sfp $ \sptr -> withForeignPtr efp $ \eptr -> do let !end = plusPtr sptr (soff + slen)+#ifdef BASE64_SIMD+ consumed <- encodePrefix (plusPtr sptr soff) slen dptr aptr+#else+ let consumed = 0+#endif innerLoop eptr- (castPtr (plusPtr sptr soff))- (castPtr dptr)+ (castPtr (plusPtr sptr (soff + consumed)))+ (castPtr (plusPtr dptr ((consumed `quot` 3) * 4))) end (loopTail dfp dptr aptr (castPtr end)) where@@ -61,10 +69,15 @@ withForeignPtr efp $ \etable -> withForeignPtr sfp $ \sptr -> do let !end = plusPtr sptr (soff + slen)+#ifdef BASE64_SIMD+ consumed <- encodePrefix (plusPtr sptr soff) slen dptr aptr+#else+ let consumed = 0+#endif innerLoop etable- (castPtr (plusPtr sptr soff))- (castPtr dptr)+ (castPtr (plusPtr sptr (soff + consumed)))+ (castPtr (plusPtr dptr ((consumed `quot` 3) * 4))) end (loopTailNoPad dfp aptr dptr (castPtr end)) where@@ -90,9 +103,12 @@ dfp <- mallocPlainForeignPtrBytes dlen withForeignPtr dfp $ \dptr -> do let !end = plusPtr sptr (soff + slen)- decodeLoop dtable- (plusPtr sptr soff)- dptr end dfp+#ifdef BASE64_SIMD+ consumed <- decodePrefix (plusPtr sptr soff) slen dptr dtable+ decodeLoopFrom dtable (plusPtr sptr soff) dptr end dfp consumed+#else+ decodeLoop dtable (plusPtr sptr soff) dptr end dfp+#endif where !dlen = (slen `quot` 4) * 3 {-# inline decodeBase64_ #-}@@ -116,9 +132,12 @@ dfp <- mallocPlainForeignPtrBytes dlen withForeignPtr dfp $ \dptr -> do let !end = plusPtr sptr (soff + slen)- decodeLoopNoError dtable- (plusPtr sptr soff)- dptr end dfp+#ifdef BASE64_SIMD+ consumed <- decodePrefix (plusPtr sptr soff) slen dptr dtable+ decodeLoopNoErrorFrom dtable (plusPtr sptr soff) dptr end dfp consumed+#else+ decodeLoopNoError dtable (plusPtr sptr soff) dptr end dfp+#endif where !dlen = (slen `quot` 4) * 3 {-# inline decodeBase64Typed_ #-}
+ src/Data/ByteString/Base64/Internal/SIMD.hs view
@@ -0,0 +1,46 @@+{-# LANGUAGE ForeignFunctionInterface #-}+-- |+-- Module : Data.ByteString.Base64.Internal.SIMD+-- Copyright : (c) 2026 Emily Pillmore+-- License : BSD-style+--+-- Maintainer : Emily Pillmore <emilypi@cohomolo.gy>+-- Stability : Experimental+-- Portability : portable+--+-- Note: Buffers must remain alive for the duration+-- of each call, using @withForeignPtr@. The decoder leaves the+-- final quanta and stops before invalid blocks so scalar code retains all+-- padding checks and error reporting, with canonicity checks *preserved*. +--+-- Why not do something like Alfred Klomp's `fastbase64`? Because we can do better!+module Data.ByteString.Base64.Internal.SIMD+( encodePrefix+, decodePrefix+) where++import Foreign.C.Types (CSize(..))+import Foreign.Ptr (Ptr)+import Data.Word (Word8)++foreign import ccall unsafe "hs_base64_simd_encode"+ c_encodePrefix :: Ptr Word8 -> CSize -> Ptr Word8 -> Ptr Word8 -> IO CSize++foreign import ccall unsafe "hs_base64_simd_decode"+ c_decodePrefix :: Ptr Word8 -> CSize -> Ptr Word8 -> Ptr Word8 -> IO CSize++-- | Encode a prefix; return the number of input bytes consumed (multiple of 3).+-- Short inputs avoid the foreign-call and runtime-dispatch overhead.+encodePrefix :: Ptr Word8 -> Int -> Ptr Word8 -> Ptr Word8 -> IO Int+encodePrefix src len dst alphabet+ | len < 96 = pure 0+ | otherwise = fromIntegral <$> c_encodePrefix src (fromIntegral len) dst alphabet+{-# INLINE encodePrefix #-}++-- | Decode a prefix; return the number of input bytes consumed (multiple of 4).+-- The original source and destination pointers are retained by the caller.+decodePrefix :: Ptr Word8 -> Int -> Ptr Word8 -> Ptr Word8 -> IO Int+decodePrefix src len dst table+ | len < 128 = pure 0+ | otherwise = fromIntegral <$> c_decodePrefix src (fromIntegral len) dst table+{-# INLINE decodePrefix #-}
src/Data/ByteString/Base64/Internal/W16/Loop.hs view
@@ -15,7 +15,9 @@ module Data.ByteString.Base64.Internal.W16.Loop ( innerLoop , decodeLoop+, decodeLoopFrom , decodeLoopNoError+, decodeLoopNoErrorFrom , lenientLoop ) where @@ -74,7 +76,23 @@ -> Ptr Word8 -> ForeignPtr Word8 -> IO (Either Text ByteString)-decodeLoop !dtable !sptr !dptr !end !dfp = go dptr sptr+decodeLoop dtable sptr dptr end dfp =+ decodeLoopFrom dtable sptr dptr end dfp 0+{-# INLINE decodeLoop #-}++-- | Resume after a validated prefix while retaining the original pointers+-- for error offsets and the total output length. The consumed input length+-- must be a multiple of four and leave at least the final quartet.+decodeLoopFrom+ :: Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> ForeignPtr Word8+ -> Int+ -> IO (Either Text ByteString)+decodeLoopFrom !dtable !sptr !dptr !end !dfp !consumed =+ go (plusPtr dptr ((consumed `quot` 4) * 3)) (plusPtr sptr consumed) where err :: Ptr Word8 -> IO (Either Text ByteString) err p = return . Left . T.pack@@ -172,7 +190,7 @@ poke @Word8 (plusPtr dst 1) (fromIntegral (unsafeShiftR w 8)) poke @Word8 (plusPtr dst 2) (fromIntegral w) return $ Right $ PS dfp 0 (3 + (dst `minusPtr` dptr))-{-# inline decodeLoop #-}+{-# INLINE decodeLoopFrom #-} decodeLoopNoError :: Ptr Word8@@ -184,7 +202,21 @@ -> Ptr Word8 -> ForeignPtr Word8 -> IO ByteString-decodeLoopNoError !dtable !sptr !dptr !end !dfp = go dptr sptr+decodeLoopNoError dtable sptr dptr end dfp =+ decodeLoopNoErrorFrom dtable sptr dptr end dfp 0+{-# INLINE decodeLoopNoError #-}++-- | Typed counterpart of 'decodeLoopFrom', with the same prefix contract.+decodeLoopNoErrorFrom+ :: Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> ForeignPtr Word8+ -> Int+ -> IO ByteString+decodeLoopNoErrorFrom !dtable !sptr !dptr !end !dfp !consumed =+ go (plusPtr dptr ((consumed `quot` 4) * 3)) (plusPtr sptr consumed) where look :: Ptr Word8 -> IO Word32 look !p = do@@ -231,7 +263,7 @@ poke @Word8 (plusPtr dst 1) (fromIntegral (unsafeShiftR w 8)) poke @Word8 (plusPtr dst 2) (fromIntegral w) go (plusPtr dst 3) (plusPtr src 4)-{-# inline decodeLoopNoError #-}+{-# INLINE decodeLoopNoErrorFrom #-} lenientLoop
src/Data/ByteString/Base64/Internal/W64/Loop.hs view
@@ -13,7 +13,9 @@ module Data.ByteString.Base64.Internal.W64.Loop ( innerLoop , decodeLoop+, decodeLoopFrom , decodeLoopNoError+, decodeLoopNoErrorFrom , lenientLoop ) where @@ -85,6 +87,17 @@ decodeLoop = W16.decodeLoop {-# inline decodeLoop #-} +decodeLoopFrom+ :: Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> ForeignPtr Word8+ -> Int+ -> IO (Either Text ByteString)+decodeLoopFrom = W16.decodeLoopFrom+{-# INLINE decodeLoopFrom #-}+ decodeLoopNoError :: Ptr Word8 -- ^ decode lookup table@@ -99,6 +112,17 @@ -> IO ByteString decodeLoopNoError = W16.decodeLoopNoError {-# inline decodeLoopNoError #-}++decodeLoopNoErrorFrom+ :: Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> Ptr Word8+ -> ForeignPtr Word8+ -> Int+ -> IO ByteString+decodeLoopNoErrorFrom = W16.decodeLoopNoErrorFrom+{-# INLINE decodeLoopNoErrorFrom #-} lenientLoop :: Ptr Word8
src/Data/ByteString/Lazy/Base64.hs view
@@ -22,9 +22,8 @@ , decodeBase64 , decodeBase64Untyped , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where @@ -35,7 +34,6 @@ import Data.ByteString.Lazy (fromChunks, toChunks) import qualified Data.ByteString.Lazy as BL import Data.ByteString.Lazy.Internal (ByteString(..))-import Data.Either (isRight) import qualified Data.Text as T import qualified Data.Text.Lazy as TL import qualified Data.Text.Lazy.Encoding as TL@@ -141,55 +139,32 @@ . toChunks {-# INLINE decodeBase64Lenient #-} --- | Tell whether a 'ByteString' value is base64 encoded.------ This function will also detect non-canonical encodings such as @ZE==@, which are--- externally valid Base64-encoded values, but are internally inconsistent "impossible"--- values.------ === __Examples__:------ >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False------ >>> isBase64 "U3V="--- False+-- | Parse a canonical, padded Base64-encoded 'ByteString' value. ---isBase64 :: ByteString -> Bool-isBase64 bs- = isValidBase64 bs- && isRight (decodeBase64Untyped bs)-{-# INLINE isBase64 #-}---- | Tell whether a 'ByteString' value is a valid Base64 format.+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. ----- This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'ByteString' value, use 'isBase64'.+-- The entire input is validated before returning 'Right'; the original+-- chunk boundaries are preserved in the result. -- -- === __Examples__: ----- >>> isValidBase64 "U3Vu"--- True+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3V"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "%"--- False+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ---isValidBase64 :: ByteString -> Bool-isValidBase64 = go . toChunks- where- go [] = True- go [c] = B64.isValidBase64 c- go (c:cs) = -- note the lack of padding char- BS.all (`BL.elem` "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/") c- && go cs-{-# INLINE isValidBase64 #-}+-- @since 2.0+--+parseBase64 :: ByteString -> Either T.Text (Base64 'StdPadded ByteString)+parseBase64 bs = assertBase64 bs <$ B64.parseBase64 (BS.concat $ toChunks bs)+{-# INLINE parseBase64 #-}
src/Data/ByteString/Short/Base64.hs view
@@ -21,9 +21,8 @@ , decodeBase64 , decodeBase64Untyped , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where import Data.Base64.Types@@ -124,47 +123,29 @@ decodeBase64Lenient = toShort . B64.decodeBase64Lenient . fromShort {-# INLINE decodeBase64Lenient #-} --- | Tell whether a 'ShortByteString' value is base64 encoded.+-- | Parse a canonical, padded Base64-encoded 'ShortByteString' value. ----- This function will also detect non-canonical encodings such as @ZE==@, which are--- externally valid Base64-encoded values, but are internally inconsistent "impossible"--- values.+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. -- -- === __Examples__: ----- >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False------ >>> isBase64 "U3V="--- False----isBase64 :: ShortByteString -> Bool-isBase64 = B64.isBase64 . fromShort-{-# INLINE isBase64 #-}---- | Tell whether a 'ShortByteString' value is a valid Base64 format.------ This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'ShortByteString' value, use 'isBase64'.------ === __Examples__:+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3Vu"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V"--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ----- >>> isValidBase64 "%"--- False+-- @since 2.0 ---isValidBase64 :: ShortByteString -> Bool-isValidBase64 = B64.isValidBase64 . fromShort-{-# INLINE isValidBase64 #-}+parseBase64 :: ShortByteString -> Either Text (Base64 'StdPadded ShortByteString)+parseBase64 bs = assertBase64 bs <$ B64.parseBase64 (fromShort bs)+{-# INLINE parseBase64 #-}
src/Data/Text/Encoding/Base64.hs view
@@ -21,9 +21,8 @@ , decodeBase64Untyped , decodeBase64UntypedWith , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where @@ -132,43 +131,32 @@ . T.encodeUtf8 {-# INLINE decodeBase64Lenient #-} --- | Tell whether an untyped 'Text' value is Base64-encoded.------ === __Examples__:------ >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False------ >>> isBase64 "U3V="--- False+-- | Parse a canonical, padded Base64-encoded 'Text' value. ---isBase64 :: Text -> Bool-isBase64 = B64.isBase64 . T.encodeUtf8-{-# INLINE isBase64 #-}---- | Tell whether an untyped 'Text' value is a valid Base64 format.+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. ----- This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'Text' value, use 'isBase64'.+-- This validates the Base64 encoding, not the UTF-8 validity of the decoded+-- bytes. Use 'decodeBase64UntypedWith' to check the decoded text encoding. -- -- === __Examples__: ----- >>> isValidBase64 "U3Vu"--- True+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3V"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "%"--- False+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ---isValidBase64 :: Text -> Bool-isValidBase64 = B64.isValidBase64 . T.encodeUtf8-{-# INLINE isValidBase64 #-}+-- @since 2.0+--+parseBase64 :: Text -> Either Text (Base64 'StdPadded Text)+parseBase64 bs = assertBase64 bs <$ B64.parseBase64 (T.encodeUtf8 bs)+{-# INLINE parseBase64 #-}
src/Data/Text/Lazy/Encoding/Base64.hs view
@@ -21,9 +21,8 @@ , decodeBase64Untyped , decodeBase64UntypedWith , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where @@ -140,43 +139,35 @@ . TL.encodeUtf8 {-# INLINE decodeBase64Lenient #-} --- | Tell whether an untyped 'TL.Text' value is Base64-encoded.------ === __Examples__:------ >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False+-- | Parse a canonical, padded Base64-encoded 'TL.Text' value. ----- >>> isBase64 "U3V="--- False+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. ---isBase64 :: TL.Text -> Bool-isBase64 = BL64.isBase64 . TL.encodeUtf8-{-# INLINE isBase64 #-}---- | Tell whether an untyped 'TL.Text' value is a valid Base64 format.+-- This validates the Base64 encoding, not the UTF-8 validity of the decoded+-- bytes. Use 'decodeBase64UntypedWith' to check the decoded text encoding. ----- This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'TL.Text' value, use 'isBase64'.+-- The entire input is validated before returning 'Right'; the original+-- chunk boundaries are preserved in the result. -- -- === __Examples__: ----- >>> isValidBase64 "U3Vu"--- True+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3V"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "%"--- False+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ---isValidBase64 :: TL.Text -> Bool-isValidBase64 = BL64.isValidBase64 . TL.encodeUtf8-{-# INLINE isValidBase64 #-}+-- @since 2.0+--+parseBase64 :: TL.Text -> Either T.Text (Base64 'StdPadded TL.Text)+parseBase64 bs = assertBase64 bs <$ BL64.parseBase64 (TL.encodeUtf8 bs)+{-# INLINE parseBase64 #-}
src/Data/Text/Short/Encoding/Base64.hs view
@@ -21,9 +21,8 @@ , decodeBase64Untyped , decodeBase64UntypedWith , decodeBase64Lenient- -- * Validation-, isBase64-, isValidBase64+ -- * Parsing+, parseBase64 ) where import Data.Base64.Types@@ -139,43 +138,32 @@ decodeBase64Lenient = fromText . B64T.decodeBase64Lenient . toText {-# INLINE decodeBase64Lenient #-} --- | Tell whether an untyped 'ShortText' value is Base64-encoded.------ === __Examples__:------ >>> isBase64 "U3Vu"--- True------ >>> isBase64 "U3V"--- False------ >>> isBase64 "U3V="--- False+-- | Parse a canonical, padded Base64-encoded 'ShortText' value. ---isBase64 :: ShortText -> Bool-isBase64 = B64.isBase64 . toByteString-{-# INLINE isBase64 #-}---- | Tell whether an untyped 'ShortText' value is a valid Base64 format.+-- Checks the alphabet, length, padding, and unused padding bits. On success,+-- returns the original encoded value wrapped in 'Base64', ready for+-- 'decodeBase64'. On failure, returns the same error as 'decodeBase64Untyped'.+-- The input is neither decoded nor normalized in the returned value. ----- This will not tell you whether or not this is a correct Base64 representation,--- only that it conforms to the correct shape. To check whether it is a true--- Base64 encoded 'ShortText' value, use 'isBase64'.+-- This validates the Base64 encoding, not the UTF-8 validity of the decoded+-- bytes. Use 'decodeBase64UntypedWith' to check the decoded text encoding. -- -- === __Examples__: ----- >>> isValidBase64 "U3Vu"--- True+-- >>> parseBase64 "U3Vu"+-- Right "U3Vu" ----- >>> isValidBase64 "U3V"--- True+-- >>> fmap decodeBase64 (parseBase64 "U3Vu")+-- Right "Sun" ----- >>> isValidBase64 "U3V="--- True+-- >>> parseBase64 "U3V"+-- Left "Base64-encoded bytestring requires padding" ----- >>> isValidBase64 "%"--- False+-- >>> parseBase64 "U3V="+-- Left "non-canonical encoding detected at offset: 2" ---isValidBase64 :: ShortText -> Bool-isValidBase64 = B64.isValidBase64 . toByteString-{-# INLINE isValidBase64 #-}+-- @since 2.0+--+parseBase64 :: ShortText -> Either Text (Base64 'StdPadded ShortText)+parseBase64 bs = assertBase64 bs <$ B64.parseBase64 (toByteString bs)+{-# INLINE parseBase64 #-}
test/Internal.hs view
@@ -74,9 +74,8 @@ , decodeUrlTypedNopad :: Base64 'UrlUnpadded bs -> bs , lenientUrl :: bs -> bs , lenient :: bs -> bs- , correct :: bs -> Bool+ , parse :: bs -> Either Text (Base64 'StdPadded bs) , correctUrl :: bs -> Bool- , validate :: bs -> Bool , validateUrl :: bs -> Bool } @@ -88,8 +87,7 @@ , decode = B64.decodeBase64Untyped , decodeTyped = B64.decodeBase64 , lenient = B64.decodeBase64Lenient- , correct = B64.isBase64- , validate = B64.isValidBase64+ , parse = B64.parseBase64 , encodeUrl = B64U.encodeBase64' , encodeUrlNopad = B64U.encodeBase64Unpadded' , decodeUrl = B64U.decodeBase64Untyped@@ -110,8 +108,7 @@ , decode = LB64.decodeBase64Untyped , decodeTyped = LB64.decodeBase64 , lenient = LB64.decodeBase64Lenient- , correct = LB64.isBase64- , validate = LB64.isValidBase64+ , parse = LB64.parseBase64 , encodeUrl = LB64U.encodeBase64' , encodeUrlNopad = LB64U.encodeBase64Unpadded' , decodeUrl = LB64U.decodeBase64Untyped@@ -132,8 +129,7 @@ , decode = SB64.decodeBase64Untyped , decodeTyped = SB64.decodeBase64 , lenient = SB64.decodeBase64Lenient- , correct = SB64.isBase64- , validate = SB64.isValidBase64+ , parse = SB64.parseBase64 , encodeUrl = SB64U.encodeBase64' , encodeUrlNopad = SB64U.encodeBase64Unpadded' , decodeUrl = SB64U.decodeBase64Untyped@@ -154,8 +150,7 @@ , decode = T64.decodeBase64Untyped , decodeTyped = T64.decodeBase64 , lenient = T64.decodeBase64Lenient- , correct = T64.isBase64- , validate = T64.isValidBase64+ , parse = T64.parseBase64 , encodeUrl = T64U.encodeBase64 , encodeUrlNopad = T64U.encodeBase64Unpadded , decodeUrl = T64U.decodeBase64Untyped@@ -176,8 +171,7 @@ , decode = TL64.decodeBase64Untyped , decodeTyped = TL64.decodeBase64 , lenient = TL64.decodeBase64Lenient- , correct = TL64.isBase64- , validate = TL64.isValidBase64+ , parse = TL64.parseBase64 , encodeUrl = TL64U.encodeBase64 , encodeUrlNopad = TL64U.encodeBase64Unpadded , decodeUrl = TL64U.decodeBase64Untyped@@ -198,8 +192,7 @@ , decode = TS64.decodeBase64Untyped , decodeTyped = TS64.decodeBase64 , lenient = TS64.decodeBase64Lenient- , correct = TS64.isBase64- , validate = TS64.isValidBase64+ , parse = TS64.parseBase64 , encodeUrl = TS64U.encodeBase64 , encodeUrlNopad = TS64U.encodeBase64Unpadded , decodeUrl = TS64U.decodeBase64Untyped
test/Main.hs view
@@ -1,3 +1,4 @@+{-# LANGUAGE CPP #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PackageImports #-}@@ -33,6 +34,7 @@ import "base64" Data.ByteString.Base64.URL as B64U import qualified "base64-bytestring" Data.ByteString.Base64 as Bos import qualified "base64-bytestring" Data.ByteString.Base64.URL as BosU+import Data.Either (isLeft) import qualified Data.Text as T import qualified Data.Text.Encoding as T import Data.Text.Encoding.Base64.Error (Base64Error(..))@@ -42,6 +44,7 @@ import Data.Word import Internal+import qualified Simd import Test.Tasty import Test.Tasty.HUnit@@ -55,8 +58,13 @@ tests :: TestTree-tests = testGroup "Base64 Tests"- [ mkTree b64+#ifdef BASE64_SIMD+tests = testGroup "Base64 Tests (simd enabled)"+#else+tests = testGroup "Base64 Tests (scalar)"+#endif+ [ Simd.tests+ , mkTree b64 [ mkPropTree , mkUnitTree BS.last BS.length ]@@ -84,6 +92,7 @@ , mkDecodeTree (second TS.fromText . T.decodeUtf8' . SBS.fromShort) tts64 sb64 ]+ , parseRepresentationTests ] -- ---------------------------------------------------------------- --@@ -122,6 +131,7 @@ mkPropTree :: (Arbitrary a, IsString a, Eq a, Show a) => Harness a -> TestTree mkPropTree = mkTests "Property Tests" [ prop_roundtrip+ , prop_parse , prop_correctness , prop_url_padding , const prop_bos_coherence@@ -141,6 +151,7 @@ , offsetVectors , validityTests , canonicityTests+ , parseTests ] -- | Make unit tests for textual 'decode*With' functions@@ -182,14 +193,104 @@ prop_correctness :: (Arbitrary a, IsString a, Eq a, Show a) => Harness a -> TestTree prop_correctness Harness{..} = testGroup "prop_validity"- [ testProperty "prop_std_valid" $ \(bs :: a) ->- validate (extractBase64 $ encode bs)- , testProperty "prop_url_valid" $ \(bs :: a) ->+ [ testProperty "prop_url_valid" $ \(bs :: a) -> validateUrl (extractBase64 $ encodeUrl bs)- , testProperty "prop_std_correct" $ \(bs :: a) ->- correct (extractBase64 $ encode bs) , testProperty "prop_url_correct" $ \(bs :: a) -> correctUrl (extractBase64 $ encodeUrl bs)+ ]++prop_parse :: (Arbitrary a, IsString a, Eq a, Show a) => Harness a -> TestTree+prop_parse Harness{..} = testGroup "Parsing properties"+ [ testProperty "parse an encoded value" $ \(bs :: a) ->+ parse (extractBase64 $ encode bs) === Right (encode bs)+ , testProperty "decode a parsed value" $ \(bs :: a) ->+ fmap decodeTyped (parse $ extractBase64 $ encode bs) === Right bs+ , testProperty "preserve the encoded input" $ \(bs :: a) ->+ case parse bs of+ Left _ -> property True+ Right encoded -> extractBase64 encoded === bs+ , testProperty "agree with untyped decoding errors" $ \(bs :: a) ->+ -- Discard the decoded payload without forcing a possible UTF-8 conversion.+ (() <$ parse bs) === (() <$ decode bs)+ ]++parseTests :: (IsString a, Eq a, Show a) => Harness a -> TestTree+parseTests Harness{..} = testGroup "Parsing unit tests"+ [ testGroup "Preserve valid encoded values"+ [ testCase (show input) $+ parse input @?= Right (assertBase64 input)+ | input <- [ "", "Zg==", "Zm8=", "Zm9v", "Zm9vYg=="+ , "Zm9vYmE=", "Zm9vYmFy", "PDw/Pz4+"+ , "/w==" -- Valid Base64 whose decoded byte is not valid UTF-8.+ ]+ ]+ , testGroup "Reject malformed and non-canonical values"+ [ testCase (show input) $ parse input @?= Left err+ | (input, err) <-+ [ ("Z", "Base64-encoded bytestring has invalid size")+ , ("Zg", "Base64-encoded bytestring requires padding")+ , ("Zm8", "Base64-encoded bytestring requires padding")+ , ("Zg===", "Base64-encoded bytestring has invalid size")+ , ("====", "invalid padding near offset: 0")+ , ("A===", "invalid padding near offset: 1")+ , ("AA=A", "invalid base64 encoding near offset: 3")+ , ("Zg==AAAA", "invalid padding near offset: 2")+ , ("Zm8=AAAA", "invalid padding near offset: 3")+ , ("%AAA", "invalid base64 encoding near offset: 0")+ , ("A%AA", "invalid base64 encoding near offset: 1")+ , ("AA%A", "invalid base64 encoding near offset: 2")+ , ("AAA%", "invalid base64 encoding near offset: 3")+ , ("AAAA%AAA", "invalid base64 encoding near offset: 4")+ , ("PDw_", "invalid base64 encoding near offset: 3")+ , ("PDw-", "invalid base64 encoding near offset: 3")+ , ("AAA ", "invalid base64 encoding near offset: 3")+ , ("AAA\n", "invalid base64 encoding near offset: 3")+ , ("ZE==", "non-canonical encoding detected at offset: 1")+ , ("ZmC=", "non-canonical encoding detected at offset: 2")+ , ("Zm9vYE==", "non-canonical encoding detected at offset: 5")+ , ("Zm9vYmC=", "non-canonical encoding detected at offset: 6")+ ]+ ]+ , testCase "reject non-ASCII input" $+ assertBool "non-ASCII input was accepted" $+ isLeft (parse "\255\255\255\255")+ ]++parseRepresentationTests :: TestTree+parseRepresentationTests = testGroup "Parsing representations"+ [ testProperty "parse sliced ByteStrings" $ \bs ->+ let input = extractBase64 (B64.encodeBase64' bs)+ sliced = BS.take (BS.length input) $ BS.drop 2 ("!!" <> input <> "!!")+ in B64.parseBase64 sliced === Right (assertBase64 input)+ , testCase "error offsets are relative to the ByteString slice" $+ B64.parseBase64 (BS.take 4 $ BS.drop 2 "!!AA%A!!")+ @?= Left "invalid base64 encoding near offset: 2"+ , testGroup "Lazy ByteString chunk boundaries"+ [ testCase (show (input, n)) $+ let chunks = [BS.take n input, BS.drop n input]+ bytes = LBS.fromChunks chunks+ in fmap (fmap LBS.toStrict) (parse lb64 bytes)+ @?= B64.parseBase64 input+ | input <- ["Zg==", "Zm9vYmFy", "Zg==AAAA", "Zm9vYmC=", "Zg", "AAAA%AAA"]+ , n <- [0 .. BS.length input]+ ]+ , testGroup "Lazy Text chunk boundaries"+ [ testCase (show (input, n)) $+ let chunks = [T.take n input, T.drop n input]+ txt = TL.fromChunks chunks+ in fmap (fmap TL.toStrict) (parse tl64 txt)+ @?= parse t64 input+ | input <- ["Zg==", "Zm9vYmFy", "Zg==AAAA", "Zm9vYmC=", "Zg", "AAAA%AAA"]+ , n <- [0 .. T.length input]+ ]+ , testCase "preserve lazy ByteString chunks, including split padding" $ do+ let chunks = ["Z", "g=", "="]+ fmap (LBS.toChunks . extractBase64) (parse lb64 $ LBS.fromChunks chunks)+ @?= Right chunks+ , testCase "preserve lazy Text chunks, including split padding" $ do+ let chunks = ["Z", "g=", "="]+ fmap (TL.toChunks . extractBase64) (parse tl64 $ TL.fromChunks chunks)+ @?= Right chunks ] prop_url_padding :: (Arbitrary a, IsString a, Eq a, Show a) => Harness a -> TestTree
+ test/Simd.hs view
@@ -0,0 +1,235 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE PackageImports #-}+-- |+-- Module : Simd+-- Copyright : (c) 2026 Emily Pillmore+-- License : BSD-style+--+-- Maintainer : Emily Pillmore <emilypi@cohomolo.gy>+-- Stability : Experimental+-- Portability : portable+--+-- Public API regressions shared by the scalar and optional SIMD builds.+-- Expected encodings come from an independent implementation, including for+-- typed decoding: matching mistakes in our encoder and decoder cannot cancel.+module Simd (tests) where++import Control.Monad (forM_)+import "base64" Data.Base64.Types (assertBase64, extractBase64)+import qualified Data.ByteString as BS+import qualified Data.ByteString.Lazy as LBS+import qualified Data.ByteString.Short as SBS+import qualified "base64" Data.ByteString.Base64 as B64+import qualified "base64" Data.ByteString.Base64.URL as B64U+import qualified "base64-bytestring" Data.ByteString.Base64 as Bos+import qualified "base64-bytestring" Data.ByteString.Base64.URL as BosU+import qualified Data.Text as T+import qualified Data.Text.Encoding as T+import qualified Data.Text.Lazy as TL+import qualified Data.Text.Short as TS+import Data.Word (Word8)+import Internal (Harness(..), b64, lb64, sb64, t64, tl64, ts64)+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.HUnit (Assertion, assertEqual, testCase)+++tests :: TestTree+tests = testGroup "SIMD compatibility"+ [ testCase "vector boundaries and every slice alignment" $+ forM_ sizes $ \n ->+ forM_ [0 .. 31] $ \offset ->+ checkHarness (context n offset) b64 (sliceAt offset) (payload n)+ , testCase "every input byte across vector blocks" $+ forM_ [0 .. 255] $ \byte ->+ checkHarness ("repeated byte=" ++ show byte) b64+ (sliceAt (fromIntegral byte `mod` 32)) (BS.replicate 257 byte)+ , testCase "invalid characters retain absolute offsets" invalidCharacters+ , testCase "all high bytes are invalid in both alphabets" highBytes+ , testCase "padding and canonical tails after vector prefixes" paddingAndTails+ , testCase "padding errors precede invalid bytes in the same quartet" mixedErrors+ , testCase "lazy chunks split input and encoded quartets" lazyChunks+ , testCase "short bytestrings preserve large buffers and tails" $+ forM_ [0, 1, 2, 47, 48, 49, 127, 128, 129, 4096, 65536] $ \n ->+ checkHarness ("short length=" ++ show n) sb64 SBS.toShort (payload n)+ , testCase "Unicode text wrappers preserve long values and UTF-8" textWrappers+ ]+ where+ sizes = [0 .. 5]+ ++ concatMap (\n -> [n - 2 .. n + 2]) [12, 16, 24, 32, 48, 64, 96, 128, 192, 256]+ ++ [255, 256, 257, 4095, 4096, 4097, 65535, 65536, 65537]++-- The prefix and suffix are deliberately non-base64 bytes. Only the slice is+-- valid input, and its address varies independently of its logical contents.+sliceAt :: Int -> BS.ByteString -> BS.ByteString+sliceAt offset bytes = BS.take (BS.length bytes) . BS.drop offset $+ BS.replicate offset 0xe5 <> bytes <> BS.replicate 32 0xff++payload :: Int -> BS.ByteString+payload n = BS.pack (take n (cycle [0 .. 255]))++context :: Int -> Int -> String+context n offset = "length=" ++ show n ++ ", slice offset=" ++ show offset++checkHarness+ :: (Eq a, Show a)+ => String+ -> Harness a+ -> (BS.ByteString -> a)+ -> BS.ByteString+ -> Assertion+checkHarness note harness convert raw = do+ equal "standard encode" std (extractBase64 (encode harness input))+ equal "URL padded encode" url (extractBase64 (encodeUrl harness input))+ equal "URL unpadded encode" unpadded (extractBase64 (encodeUrlNopad harness input))+ equal "standard typed decode" input (decodeTyped harness (assertBase64 std))+ equal "standard untyped decode" (Right input) (decode harness std)+ equal "URL padded typed decode" input (decodeUrlTypedPad harness (assertBase64 url))+ equal "URL padded untyped decode" (Right input) (decodeUrlPad harness url)+ equal "URL unpadded typed decode" input+ (decodeUrlTypedNopad harness (assertBase64 unpadded))+ equal "URL unpadded untyped decode" (Right input) (decodeUrlNopad harness unpadded)+ equal "URL generic padded decode" (Right input) (decodeUrl harness url)+ equal "URL generic unpadded decode" (Right input) (decodeUrl harness unpadded)+ where+ input = convert raw+ std = convert (Bos.encode raw)+ url = convert (BosU.encode raw)+ unpadded = convert (BS.takeWhile (/= 61) (BosU.encode raw))+ equal name = assertEqual (note ++ ": " ++ name)++-- Exercise invalid bytes in every lane, across native block boundaries, and+-- well after a valid SIMD prefix. A valid suffix prevents tail-only coverage.+invalidCharacters :: Assertion+invalidCharacters =+ forM_ [0, 1, 15, 31] $ \offset ->+ forM_ positions $ \position -> do+ forM_ [35, 128, 255] $ \byte ->+ forM_ decoders $ \(name, decoder) ->+ checkInvalid name decoder offset position byte+ forM_ [45, 95] $ \byte ->+ checkInvalid "standard rejects URL alphabet" B64.decodeBase64Untyped+ offset position byte+ forM_ [43, 47] $ \byte ->+ forM_ urlDecoders $ \(name, decoder) ->+ checkInvalid name decoder offset position byte+ where+ positions = [0 .. 19] ++ [31 .. 33] ++ [47 .. 49] ++ [63 .. 65]+ ++ [95 .. 97] ++ [127 .. 129] ++ [191 .. 193] ++ [255 .. 257]+ ++ [511 .. 513] ++ [4095 .. 4097]++highBytes :: Assertion+highBytes =+ forM_ [128 .. 255] $ \byte ->+ forM_ [15, 16, 31, 32, 63, 64, 127, 255, 256, 4096] $ \position ->+ forM_ decoders $ \(name, decoder) ->+ checkInvalid name decoder (fromIntegral byte `mod` 32) position byte++type Decoder = BS.ByteString -> Either T.Text BS.ByteString++decoders :: [(String, Decoder)]+decoders = ("standard", B64.decodeBase64Untyped) : urlDecoders++urlDecoders :: [(String, Decoder)]+urlDecoders =+ [ ("URL padded", B64U.decodeBase64PaddedUntyped)+ , ("URL unpadded", B64U.decodeBase64UnpaddedUntyped)+ , ("URL generic", B64U.decodeBase64Untyped)+ ]++checkInvalid :: String -> Decoder -> Int -> Int -> Word8 -> Assertion+checkInvalid name decoder offset position byte =+ assertEqual note+ (Left (atOffset "invalid base64 encoding near offset: " position))+ (decoder input)+ where+ input = sliceAt offset $+ BS.replicate position 65 <> BS.singleton byte+ <> BS.replicate (position `div` 4 * 4 + 260 - position - 1) 65+ note = name ++ ": " ++ context (BS.length input) offset+ ++ ", invalid byte=" ++ show byte ++ " at=" ++ show position++atOffset :: T.Text -> Int -> T.Text+atOffset prefix offset = prefix <> T.pack (show offset)++mixedErrors :: Assertion+mixedErrors =+ forM_ [0, 32, 64, 128, 256, 4096] $ \prefixLength ->+ forM_ [0, 1, 15, 31] $ \offset ->+ forM_ [("#=AA", 1), ("A#=A", 2), ("#AA=", 3), ("=#AA", 0)] $+ \(quartet, paddingPosition) ->+ forM_ decoders $ \(name, decoder) ->+ assertEqual (name ++ ": " ++ context prefixLength offset+ ++ ", mixed errors=" ++ show quartet)+ (Left (atOffset "invalid padding near offset: "+ (prefixLength + paddingPosition)))+ (decoder (sliceAt offset+ (BS.replicate prefixLength 65 <> quartet <> BS.replicate 256 65)))++paddingAndTails :: Assertion+paddingAndTails =+ forM_ [0, 16, 32, 48, 64, 96, 128, 192, 256, 4096] $ \prefixLength ->+ forM_ [0, 1, 15, 31] $ \offset -> do+ let prefix = BS.replicate prefixLength 65+ input suffix = sliceAt offset (prefix <> suffix)+ note name = name ++ ": " ++ context prefixLength offset+ expected suffix = Right (BS.replicate (prefixLength `div` 4 * 3) 0 <> suffix)+ forM_ decoders $ \(name, decoder) ->+ forM_ [0 .. 3] $ \position ->+ assertEqual (note name ++ ", interior padding at=" ++ show position)+ (Left (atOffset "invalid padding near offset: " (prefixLength + position)))+ (decoder (input (BS.replicate position 65 <> "="+ <> BS.replicate (7 - position) 65)))+ forM_ (("standard", B64.decodeBase64Untyped) : take 1 urlDecoders+ ++ [("URL generic", B64U.decodeBase64Untyped)]) $ \(name, decoder) -> do+ assertEqual (note name ++ ", one-byte tail") (expected "d")+ (decoder (input "ZA=="))+ assertEqual (note name ++ ", two-byte tail") (expected "f`")+ (decoder (input "ZmA="))+ forM_ [("ZE==", 1), ("ZmC=", 2)] $ \(suffix, position) ->+ assertEqual (note name ++ ", noncanonical tail=" ++ show suffix)+ (Left (atOffset "non-canonical encoding detected at offset: "+ (prefixLength + position))) (decoder (input suffix))+ forM_ (drop 1 urlDecoders) $ \(name, decoder) -> do+ assertEqual (note name ++ ", unpadded one-byte tail") (expected "d")+ (decoder (input "ZA"))+ assertEqual (note name ++ ", unpadded two-byte tail") (expected "f`")+ (decoder (input "ZmA"))+ forM_ [("ZE", 1), ("ZmC", 2)] $ \(suffix, position) ->+ assertEqual (note name ++ ", noncanonical unpadded tail=" ++ show suffix)+ (Left (atOffset "non-canonical encoding detected at offset: "+ (prefixLength + position))) (decoder (input suffix))+ assertEqual (note "unpadded rejects padding")+ (Left "Base64-encoded bytestring has invalid padding")+ (B64U.decodeBase64UnpaddedUntyped (input "ZA=="))+ assertEqual (note "padded requires padding")+ (Left "Base64-encoded bytestring requires padding")+ (B64U.decodeBase64PaddedUntyped (input "ZA"))+ forM_ decoders $ \(name, decoder) ->+ assertEqual (note name ++ ", invalid length")+ (Left "Base64-encoded bytestring has invalid size") (decoder (input "A"))++lazyChunks :: Assertion+lazyChunks =+ forM_ [47, 48, 49, 95, 96, 97, 127, 128, 129, 4096] $ \n ->+ forM_ [1, 2, 3, 4, 11, 12, 15, 16, 47, 48, 63, 64, 127] $ \chunkSize ->+ checkHarness ("lazy length=" ++ show n ++ ", chunk=" ++ show chunkSize)+ lb64 (LBS.fromChunks . chunksOf chunkSize) (payload n)++chunksOf :: Int -> BS.ByteString -> [BS.ByteString]+chunksOf n bytes+ | BS.null bytes = []+ | otherwise = let (chunk, rest) = BS.splitAt n bytes+ in sliceAt 1 chunk : chunksOf n rest++textWrappers :: Assertion+textWrappers =+ forM_ [0, 1, 15, 16, 31, 32, 127, 1024] $ \repetitions -> do+ let value = T.replicate repetitions "a\x00e9\x03bb\x1f642"+ raw = T.encodeUtf8 value+ note = "text repetitions=" ++ show repetitions+ checkHarness note t64 T.decodeUtf8 raw+ checkHarness note ts64 (TS.fromText . T.decodeUtf8) raw+ forM_ [1, 15, 16, 63, 64] $ \chunkSize ->+ checkHarness (note ++ ", text chunk=" ++ show chunkSize) tl64+ (TL.fromChunks . T.chunksOf chunkSize . T.decodeUtf8) raw
+ test/simd-native.c view
@@ -0,0 +1,380 @@+/* Copyright (c) 2026, Emily Pillmore. BSD-3-Clause; see LICENSE. */++/*+ * Standalone differential/bounds checks; link with cbits/base64_simd.c.+ * Recommended: -std=c99 -O2 -Wall -Wextra -Werror+ * -fsanitize=address,undefined -fno-omit-frame-pointer+ * Repeat with -DHS_BASE64_SIMD_FORCE_SCALAR to test fallback compilation.+ */+#ifndef _DEFAULT_SOURCE+#define _DEFAULT_SOURCE 1+#endif++#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#if defined(__unix__) || defined(__APPLE__)+#include <sys/mman.h>+#include <unistd.h>+#define HS_BASE64_GUARD_PAGES 1+#endif++#include "base64_simd.h"++#define CHECK(condition) do { \+ if (!(condition)) { \+ fprintf(stderr, "%s:%d: check failed: %s\n", __FILE__, __LINE__, #condition); \+ exit(EXIT_FAILURE); \+ } \+} while (0)++static const uint8_t alphabets[2][65] = {+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"+};++static size_t encode_block;+static size_t decode_block;+static size_t checks;++static void make_table(const uint8_t *alphabet, uint8_t *table)+{+ memset(table, 255, 256);+ for (size_t i = 0; i < 64; i++) {+ table[alphabet[i]] = (uint8_t)i;+ }+ table['='] = 64;+}++static size_t encode_prefix(size_t len)+{+ return encode_block == 0 ? 0 : len / encode_block * encode_block;+}++static size_t decode_prefix(const uint8_t *src, size_t len, const uint8_t *table)+{+ size_t consumed = 0;++ if (decode_block == 0 || len < 4) {+ return 0;+ }+ while (len - consumed >= decode_block + 4) {+ for (size_t i = 0; i < decode_block; i++) {+ if (table[src[consumed + i]] >= 64) {+ return consumed;+ }+ }+ consumed += decode_block;+ }+ return consumed;+}++static void scalar_encode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *alphabet)+{+ for (size_t i = 0; i < len; i += 3) {+ const unsigned triple = ((unsigned)src[i] << 16) |+ ((unsigned)src[i + 1] << 8) | src[i + 2];+ *dst++ = alphabet[(triple >> 18) & 63];+ *dst++ = alphabet[(triple >> 12) & 63];+ *dst++ = alphabet[(triple >> 6) & 63];+ *dst++ = alphabet[triple & 63];+ }+}++static void scalar_decode(const uint8_t *src, size_t len, uint8_t *dst,+ const uint8_t *table)+{+ for (size_t i = 0; i < len; i += 4) {+ const unsigned quartet = ((unsigned)table[src[i]] << 18) |+ ((unsigned)table[src[i + 1]] << 12) |+ ((unsigned)table[src[i + 2]] << 6) |+ table[src[i + 3]];+ *dst++ = (uint8_t)(quartet >> 16);+ *dst++ = (uint8_t)(quartet >> 8);+ *dst++ = (uint8_t)quartet;+ }+}++static void check_canary(const uint8_t *data, size_t len)+{+ for (size_t i = 0; i < len; i++) {+ CHECK(data[i] == 0xa5);+ }+}++static void check_encode(const uint8_t *src, size_t len, size_t offset,+ const uint8_t *alphabet)+{+ const size_t expected = encode_prefix(len);+ const size_t output_len = expected / 3 * 4;+ uint8_t *output = malloc(offset + output_len + 16);+ uint8_t *reference = malloc(output_len + 1);+ size_t consumed;++ CHECK(output != NULL && reference != NULL);+ memset(output, 0xa5, offset + output_len + 16);+ scalar_encode(src, expected, reference, alphabet);+ consumed = hs_base64_simd_encode(src, len, output + offset, alphabet);+ CHECK(consumed == expected);+ CHECK(consumed <= len && consumed % 3 == 0);+ CHECK(memcmp(output + offset, reference, output_len) == 0);+ check_canary(output, offset);+ check_canary(output + offset + output_len, 16);+ free(reference);+ free(output);+ checks++;+}++static void check_decode(const uint8_t *src, size_t len, size_t offset,+ const uint8_t *table)+{+ const size_t expected = decode_prefix(src, len, table);+ const size_t output_len = expected / 4 * 3;+ uint8_t *output = malloc(offset + output_len + 16);+ uint8_t *reference = malloc(output_len + 1);+ size_t consumed;++ CHECK(output != NULL && reference != NULL);+ memset(output, 0xa5, offset + output_len + 16);+ scalar_decode(src, expected, reference, table);+ consumed = hs_base64_simd_decode(src, len, output + offset, table);+ CHECK(consumed == expected);+ CHECK(consumed % 4 == 0 && consumed <= len);+ CHECK(len < 4 ? consumed == 0 : len - consumed >= 4);+ CHECK(memcmp(output + offset, reference, output_len) == 0);+ check_canary(output, offset);+ check_canary(output + offset + output_len, 16);+ free(reference);+ free(output);+ checks++;+}++static void test_lengths_and_alignment(void)+{+ uint8_t table[256];+ uint8_t source[1056];+ uint8_t encoded[1408];++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (size_t offset = 0; offset < 32; offset++) {+ for (size_t i = 0; i < sizeof(source); i++) {+ source[i] = (uint8_t)(i * 73 + offset * 19);+ }+ scalar_encode(source, sizeof(source), encoded, alphabets[alphabet]);+ for (size_t len = 0; len <= 1024; len++) {+ check_encode(source + offset, len, 31 - offset, alphabets[alphabet]);+ check_decode(encoded + offset, len, 31 - offset, table);+ }+ }+ }+}++static void test_all_byte_values(void)+{+ uint8_t source[257];+ uint8_t encoded[344];+ uint8_t table[256];++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (unsigned value = 0; value <= 255; value++) {+ memset(source, (int)value, sizeof(source));+ check_encode(source, sizeof(source), 1, alphabets[alphabet]);+ scalar_encode(source, 255, encoded, alphabets[alphabet]);+ memcpy(encoded + 340, "AAAA", 4);+ check_decode(encoded, sizeof(encoded), 1, table);+ }+ }+}++static void test_invalid_and_padding_stops(void)+{+ uint8_t source[580];+ uint8_t table[256];++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (size_t position = 0; position < sizeof(source); position++) {+ for (unsigned value = 0; value <= 255; value++) {+ memset(source, 'A', sizeof(source));+ source[position] = (uint8_t)value;+ check_decode(source, sizeof(source), position % 32, table);+ }+ }+ /* In particular, every kind of final padding stays with the caller. */+ memset(source, '/', sizeof(source));+ for (size_t padding = 0; padding <= 4; padding++) {+ memset(source, alphabets[alphabet][63], sizeof(source));+ memset(source + sizeof(source) - padding, '=', padding);+ check_decode(source, sizeof(source), 0, table);+ }+ }+}++/* Fixed-seed xorshift exercises byte combinations outside repeated/cyclic data. */+static uint32_t next_random(uint32_t *state)+{+ uint32_t value = *state;+ value ^= value << 13;+ value ^= value >> 17;+ value ^= value << 5;+ *state = value;+ return value;+}++static void test_random_mixed_inputs(void)+{+ uint8_t source[6144];+ uint8_t encoded[8192];+ uint8_t table[256];+ uint32_t state = UINT32_C(0x6d253fb9);++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (size_t sample = 0; sample < 256; sample++) {+ const size_t len = next_random(&state) % (sizeof(source) + 1);+ const size_t encoded_len = len / 3 * 4;+ const size_t offset = next_random(&state) % 32;++ for (size_t i = 0; i < len; i++) {+ source[i] = (uint8_t)next_random(&state);+ }+ check_encode(source, len, offset, alphabets[alphabet]);+ scalar_encode(source, len / 3 * 3, encoded, alphabets[alphabet]);+ check_decode(encoded, encoded_len, offset, table);+ if (encoded_len > 0) {+ const size_t position = next_random(&state) % encoded_len;+ encoded[position] = (uint8_t)next_random(&state);+ check_decode(encoded, encoded_len, offset, table);+ }+ }+ }+}++static void test_tight_allocations(void)+{+ uint8_t table[256];++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (size_t len = 1; len <= 1024; len++) {+ uint8_t *source = malloc(len);+ const size_t encode_len = encode_prefix(len) / 3 * 4;+ uint8_t *encoded = malloc(encode_len == 0 ? 1 : encode_len);+ size_t decode_len;+ uint8_t *decoded;++ CHECK(source != NULL && encoded != NULL);+ memset(source, 0xfb, len);+ CHECK(hs_base64_simd_encode(source, len, encoded, alphabets[alphabet]) ==+ encode_prefix(len));+ memset(source, alphabets[alphabet][63], len);+ decode_len = decode_prefix(source, len, table) / 4 * 3;+ decoded = malloc(decode_len == 0 ? 1 : decode_len);+ CHECK(decoded != NULL);+ CHECK(hs_base64_simd_decode(source, len, decoded, table) ==+ decode_prefix(source, len, table));+ free(decoded);+ free(encoded);+ free(source);+ checks += 2;+ }+ }+}++#if defined(HS_BASE64_GUARD_PAGES)+/* Guard pages independently catch over-wide intrinsic loads/stores. */+static uint8_t *guarded_buffer(size_t page_size)+{+ uint8_t *mapping = mmap(NULL, page_size * 3, PROT_NONE,+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);+ CHECK(mapping != MAP_FAILED);+ CHECK(mprotect(mapping + page_size, page_size, PROT_READ | PROT_WRITE) == 0);+ return mapping;+}++static void test_guard_pages(void)+{+ const long raw_page_size = sysconf(_SC_PAGESIZE);+ size_t page_size;+ uint8_t *source_mapping;+ uint8_t *output_mapping;+ uint8_t table[256];++ CHECK(raw_page_size > 0);+ page_size = (size_t)raw_page_size;+ source_mapping = guarded_buffer(page_size);+ output_mapping = guarded_buffer(page_size);++ for (size_t alphabet = 0; alphabet < 2; alphabet++) {+ make_table(alphabets[alphabet], table);+ for (size_t len = 0; len <= 1024; len++) {+ for (size_t at_end = 0; at_end < 2; at_end++) {+ uint8_t *source = source_mapping + page_size ++ (at_end ? page_size - len : 0);+ const size_t encode_len = encode_prefix(len) / 3 * 4;+ uint8_t *output = output_mapping + page_size ++ (at_end ? page_size - encode_len : 0);+ size_t expected;+ size_t decode_len;++ memset(source, 0xfb, len);+ CHECK(hs_base64_simd_encode(source, len, output, alphabets[alphabet]) ==+ encode_prefix(len));+ memset(source, alphabets[alphabet][63], len);+ expected = decode_prefix(source, len, table);+ decode_len = expected / 4 * 3;+ output = output_mapping + page_size + (at_end ? page_size - decode_len : 0);+ CHECK(hs_base64_simd_decode(source, len, output, table) == expected);+ checks += 2;+ }+ }+ }+ CHECK(munmap(source_mapping, page_size * 3) == 0);+ CHECK(munmap(output_mapping, page_size * 3) == 0);+}+#endif++int main(int argc, char **argv)+{+ const int backend = hs_base64_simd_backend();++ if (argc != 1) {+ if (argc != 3 || strcmp(argv[1], "--require-backend") != 0 ||+ strlen(argv[2]) != 1 || argv[2][0] < '0' || argv[2][0] > '3') {+ fprintf(stderr, "usage: %s [--require-backend 0|1|2|3]\n", argv[0]);+ return EXIT_FAILURE;+ }+ if (backend != argv[2][0] - '0') {+ fprintf(stderr, "required SIMD backend %c, detected %d\n", argv[2][0], backend);+ return EXIT_FAILURE;+ }+ }+ CHECK(backend >= 0 && backend <= 3);+#if defined(HS_BASE64_SIMD_FORCE_SCALAR)+ CHECK(backend == 0);+#elif defined(HS_BASE64_SIMD_FORCE_SSSE3) && defined(__x86_64__)+ CHECK(backend == 0 || backend == 1);+#elif defined(__aarch64__) && defined(__ARM_NEON) && \+ (defined(__GNUC__) || defined(__clang__)) && !defined(HS_BASE64_SIMD_EMULATED_X86)+ CHECK(backend == 2);+#endif+ encode_block = backend == 3 ? 24 : backend == 2 ? 48 : backend == 1 ? 12 : 0;+ decode_block = backend == 3 ? 32 : backend == 2 ? 64 : backend == 1 ? 16 : 0;+ printf("SIMD backend: %d (%s)\n", backend,+ backend == 3 ? "AVX2" : backend == 2 ? "NEON" :+ backend == 1 ? "SSSE3" : "scalar fallback");+ test_lengths_and_alignment();+ test_all_byte_values();+ test_invalid_and_padding_stops();+ test_random_mixed_inputs();+ test_tight_allocations();+#if defined(HS_BASE64_GUARD_PAGES)+ test_guard_pages();+#endif+ printf("Passed %zu differential/bounds checks.\n", checks);+ return EXIT_SUCCESS;+}