packages feed

croaring-ffi (empty) → 1.0

raw patch · 91 files changed

+40941/−0 lines, 91 filesdep +basedep +binarydep +bitvec

Dependencies added: base, binary, bitvec, bytestring, containers, croaring-ffi, deepseq, ghc-compact, hedgehog, mmorph, resourcet, tasty, tasty-bench, tasty-hedgehog, vector

Files

+ CHANGELOG.md view
@@ -0,0 +1,15 @@+# Changelog for `croaring-ffi`++All notable changes to this project will be documented in this file.++The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),+and this project adheres to the+[Haskell Package Versioning Policy](https://pvp.haskell.org/).++## Unreleased++## 1.0 - 2026-11-06++- Raw bindings to CRoaring 5.5.5: 32- and 64-bit bitmaps, their iterators, and `bitset_t`.+- Scoped `with` helpers, validated loaders and views, exception-safe callbacks, `getStatistics`, and the memory hook.+- In-place operations and `overwrite` accept the same handle on both sides.
+ LICENSE view
@@ -0,0 +1,26 @@+Copyright 2026 IC Rainbow++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1.  Redistributions of source code must retain the above copyright notice, this+    list of conditions and the following disclaimer.++2.  Redistributions in binary form must reproduce the above copyright notice,+    this list of conditions and the following disclaimer in the documentation+    and/or other materials provided with the distribution.++3.  Neither the name of the copyright holder nor the names of its contributors+    may be used to endorse or promote products derived from this software+    without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ README.md view
@@ -0,0 +1,91 @@+# croaring-ffi++Raw Haskell bindings to a bundled copy of [CRoaring](https://github.com/RoaringBitmap/CRoaring), the C implementation of [Roaring bitmaps](https://roaringbitmap.org/).++## Usage++```haskell+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Foreign.Marshal.Array (withArrayLen)++main :: IO ()+main =+  Bitmap.with create $ \bitmap -> do+    Bitmap.addRange bitmap 10 20+    Bitmap.contains bitmap 15 >>= print+    Bitmap.getCardinality bitmap >>= print+    Bitmap.rank bitmap 100 >>= print+  where+    create = withArrayLen [1, 2, 3, 1000000] $ \count values ->+      Bitmap.ofPtr (fromIntegral count) values+```++Bindings are named after their C functions without the type prefixes.+For example, `roaring_bitmap_add_range` is `Bitmap.addRange`.+The [upstream documentation](https://github.com/RoaringBitmap/CRoaring/blob/master/include/roaring/roaring.h) describes what each function does.+The `_safe` loaders are named `*Bounded` because they only bound how far they read.++`Bitmap.with` frees the bitmap when the action ends.+If the constructor returns NULL, `Bitmap.with` throws `NullHandle`.+To let the garbage collector free a bitmap instead, wrap its pointer with `newForeignPtr Bitmap.finalizerFree`.++Use `portableDeserializeValidated` for bytes from outside the program.+It reads no more than the given size, and returns `Nothing` unless the bytes hold a valid bitmap:++```haskell+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.ByteString qualified as ByteString++main :: IO ()+main = do+  bytes <- ByteString.readFile "bitmap.roaring"+  loaded <- ByteString.useAsCStringLen bytes $ \(buffer, size) ->+    Bitmap.portableDeserializeValidated buffer $ fromIntegral size+  case loaded of+    Nothing -> putStrLn "not a valid bitmap"+    Just bitmap -> do+      Bitmap.getCardinality bitmap >>= print+      Bitmap.free bitmap+```++## Performance++At a million sparse elements, both Roaring bitmaps beat `Data.IntSet` at every operation by 1.5 to 7 times.+They count elements 110 to 140 times faster ([numbers](https://gitlab.com/dpwiz/croaring-ffi/-/blob/main/bench/results.csv)):++![Time of 32-bit and 64-bit Roaring bitmaps relative to Data.IntSet for six operations on 1M sparse elements](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/sparse.svg)++In smaller sparse sets, most elements are alone in their block of 65,536 values.+So `IntSet` combines and iterates faster until the set grows to between a hundred thousand and a million elements.+Roaring still counts faster at every size, and looks up elements faster from a few thousand elements:++![Time per operation against set size for Data.IntSet and 32-bit and 64-bit Roaring bitmaps on sparse elements](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/scaling.svg)++Much of that small-set cost is allocation.+CRoaring allocates every block separately.+Under GHC, glibc's `malloc` always runs in its slower multi-threaded mode, because the runtime keeps a timer thread.++Set `GLIBC_TUNABLES=glibc.malloc.tcache_count=65535` to make small-set unions about 3.4 times faster and construction about twice as fast.++In a small range, plain bit arrays (CRoaring's `bitset_t` and [bitvec](https://hackage.haskell.org/package/bitvec)) build, look up and combine scattered elements fastest.+Roaring bitmaps count 70 to 810 times faster than them, and combine a completely filled range fastest:++![Time of Roaring bitmaps, CRoaring bitset_t and bitvec relative to Data.IntSet for six operations on two dense datasets](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/dense.svg)++Some comparisons are unfair to `IntSet`.+It has no rank, range count, allocation-free intersection size or zero-copy loading, so it walks, rebuilds or decodes the set instead.+Here Roaring is 1.4 to 2,500 times faster on sparse elements, and 10 to 560,000 times faster on dense ones:++![Time of 32-bit and 64-bit Roaring bitmaps relative to Data.IntSet for ten operations IntSet has no direct primitive for](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/unfair.svg)++Roaring also takes far less space.+In memory, it needs 5 to 6.5 bytes per element on sparse data, where `IntSet` needs 64.+Within a small range it needs about 2, like the plain bit arrays.+A completely filled million-element range takes one to two kilobytes, where `IntSet` needs a megabyte.+Serialized, Roaring needs at most about 2.5 bytes per element, where `Data.Binary` needs 8 ([numbers](https://gitlab.com/dpwiz/croaring-ffi/-/blob/main/bench/sizes.csv)):++![Bytes per element in memory and serialized for Data.IntSet, Roaring bitmaps, CRoaring bitset_t and bitvec on three datasets](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/sizes.svg)++Reading elements back in chunks costs about a nanosecond each, while a Haskell callback per element costs about a hundred:++![Nanoseconds per element for four ways of reading 1M elements out of a 32-bit Roaring bitmap from Haskell](https://gitlab.com/dpwiz/croaring-ffi/-/raw/main/bench/traversal.svg)
+ bench/Bench/Baseline.hs view
@@ -0,0 +1,18 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedRecordDot #-}++module Bench.Baseline (versus) where++import Data.List (intercalate)+import GHC.Records (HasField)+import Test.Tasty.Bench (Benchmark, Benchmarkable, bcompare, bench, bgroup)++versus :: (HasField "name" contender String) => [String] -> contender -> [contender] -> String -> (contender -> Benchmarkable) -> Benchmark+versus path baseline others operation select =+  bgroup operation $+    bench baseline.name (select baseline)+      : [bcompare baselinePattern . bench other.name $ select other | other <- others]+  where+    baselinePattern = intercalate " && " . zipWith field [0 :: Int ..] $ baseline.name : operation : path+    field 0 component = "$NF == \"" <> component <> "\""+    field depth component = "$(NF-" <> show depth <> ") == \"" <> component <> "\""
+ bench/Bench/Bitmap.hs view
@@ -0,0 +1,253 @@+{-# LANGUAGE DuplicateRecordFields #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++module Bench.Bitmap (benchmarks) where++import Bench.Values (Values (..), countWhere, dense, forEach, prefix, scattered, withValues)+import Control.Monad (forM, void)+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap (..))+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap.BulkContext qualified as BulkContext+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator (Iterator (..))+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator qualified as Iterator+import Data.List (sort)+import Data.Word (Word32, Word64)+import Foreign.C.ConstPtr (ConstPtr (..))+import Foreign.C.Types (CChar, CSize)+import Foreign.Marshal.Alloc (allocaBytesAligned, free, mallocBytes)+import Foreign.Marshal.Array (advancePtr, allocaArray, mallocArray, peekArray, pokeArray, withArrayLen)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (Ptr, alignPtr, freeHaskellFunPtr, nullPtr)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark, bench, bgroup, whnfIO)++size :: Int+size = 1_000_000++benchmarks :: Benchmark+benchmarks =+  withValues (scattered 1 size) $ \getScattered ->+    withValues (scattered 2 size) $ \getOther ->+      withValues (dense size) $ \getDense ->+        withBitmap getScattered $ \getScatteredBitmap ->+          withBitmap getOther $ \getOtherBitmap ->+            withBitmap getDense $ \getDenseBitmap ->+              bgroup+                "Bitmap"+                [ bgroup+                    "construction"+                    [ bgroup "scattered" $ construction getScattered+                    , bgroup "dense" $ construction getDense <> [bench "fromRange" . whnfIO $ Bitmap.fromRange 0 (fromIntegral size) 1 >>= Bitmap.free]+                    ]+                , bgroup+                    "queries"+                    [ bgroup "hits" $ queries getScatteredBitmap getScattered+                    , bgroup "misses" $ queries getScatteredBitmap getOther+                    , rankMany getScatteredBitmap+                    ]+                , bgroup+                    "algebra"+                    [ bgroup "scattered" $ algebra getScatteredBitmap getOtherBitmap+                    , bgroup "mixed" $ algebra getScatteredBitmap getDenseBitmap+                    , many+                    ]+                , bgroup+                    "traversal"+                    [ bgroup "scattered" $ traversal getScatteredBitmap+                    , bgroup "dense" $ traversal getDenseBitmap+                    ]+                , bgroup+                    "serialization"+                    [ bgroup "scattered" $ serialization getScatteredBitmap+                    , bgroup "dense" $ serialization getDenseBitmap+                    ]+                ]++withBitmap :: IO (Values Word32) -> (IO Bitmap -> Benchmark) -> Benchmark+withBitmap getValues = withResource acquire Bitmap.free+  where+    acquire = do+      values <- getValues+      Bitmap.ofPtr (fromIntegral values.count) values.pointer++construction :: IO (Values Word32) -> [Benchmark]+construction getValues =+  [ bench "ofPtr" . whnfIO $ do+      values <- getValues+      Bitmap.ofPtr (fromIntegral values.count) values.pointer >>= Bitmap.free+  , filling "addMany" $ \bitmap values -> Bitmap.addMany bitmap (fromIntegral values.count) values.pointer+  , filling "add" $ \bitmap values -> forEach values $ Bitmap.add bitmap+  , filling "addChecked" $ \bitmap values -> forEach values $ void . Bitmap.addChecked bitmap+  , filling "addBulk" $ \bitmap values ->+      allocaBytesAligned BulkContext.size BulkContext.alignment $ \memory -> do+        fillBytes memory 0 BulkContext.size+        forEach values . Bitmap.addBulk bitmap $ BulkContext memory+  ]+  where+    filling :: String -> (Bitmap -> Values Word32 -> IO ()) -> Benchmark+    filling name fill =+      bench name . whnfIO $ do+        values <- getValues+        bitmap <- Bitmap.create+        fill bitmap values+        Bitmap.free bitmap++queries :: IO Bitmap -> IO (Values Word32) -> [Benchmark]+queries getBitmap getProbes =+  [ probing "contains" $ \bitmap probes -> countWhere probes $ Bitmap.contains bitmap+  , probing "containsBulk" $ \bitmap probes ->+      allocaBytesAligned BulkContext.size BulkContext.alignment $ \memory -> do+        fillBytes memory 0 BulkContext.size+        countWhere probes . Bitmap.containsBulk bitmap $ BulkContext memory+  , probing "rank of 1000" $ \bitmap probes -> countWhere (prefix 1000 probes) $ fmap (> 0) . Bitmap.rank bitmap+  , probing "getIndex of 1000" $ \bitmap probes -> countWhere (prefix 1000 probes) $ fmap (>= 0) . Bitmap.getIndex bitmap+  , bench "getCardinality" . whnfIO $ getBitmap >>= Bitmap.getCardinality+  ]+  where+    probing name query =+      bench name . whnfIO $ do+        bitmap <- getBitmap+        probes <- getProbes+        query bitmap probes++rankMany :: IO Bitmap -> Benchmark+rankMany getBitmap =+  withValues (sort $ scattered 1 size) $ \getSorted ->+    withResource (mallocArray size) free $ \getOutput ->+      bench "rankMany" . whnfIO $ do+        bitmap <- getBitmap+        sorted <- getSorted+        output <- getOutput+        Bitmap.rankMany bitmap sorted.pointer (advancePtr sorted.pointer sorted.count) output++algebra :: IO Bitmap -> IO Bitmap -> [Benchmark]+algebra getLeft getRight =+  [ fresh "and" Bitmap.and+  , fresh "or" Bitmap.or+  , fresh "xor" Bitmap.xor+  , fresh "andnot" Bitmap.andnot+  , measured "andCardinality" Bitmap.andCardinality+  , measured "orCardinality" Bitmap.orCardinality+  , measured "intersect" Bitmap.intersect+  , measured "equals" Bitmap.equals+  , bench "copy" . whnfIO $ getLeft >>= Bitmap.copy >>= Bitmap.free+  ]+  where+    fresh name operation = measured name $ \left right -> operation left right >>= Bitmap.free+    measured name operation =+      bench name . whnfIO $ do+        left <- getLeft+        right <- getRight+        operation left right++many :: Benchmark+many =+  withResource acquire release $ \getHandles ->+    bgroup+      "many"+      [ combined getHandles "orMany" $ Bitmap.orMany . fromIntegral+      , combined getHandles "orManyHeap" $ Bitmap.orManyHeap . fromIntegral+      , combined getHandles "xorMany" $ Bitmap.xorMany . fromIntegral+      ]+  where+    count = 16 :: Int+    combined getHandles name operation = bench name . whnfIO $ getHandles >>= operation count >>= Bitmap.free+    acquire = do+      bitmaps <- forM [1 .. fromIntegral count] $ \seed ->+        withArrayLen (scattered seed 100_000) $ \values ptr -> Bitmap.ofPtr (fromIntegral values) ptr+      handles <- mallocArray count+      pokeArray handles bitmaps+      pure handles+    release handles = do+      peekArray count handles >>= mapM_ Bitmap.free+      free handles++traversal :: IO Bitmap -> [Benchmark]+traversal getBitmap =+  [ withResource (mallocArray size) free $ \getOutput ->+      bench "toUint32Array" . whnfIO $ do+        bitmap <- getBitmap+        getOutput >>= Bitmap.toUint32Array bitmap+  , bench "Iterator.read" . whnfIO $ do+      bitmap <- getBitmap+      iterating bitmap $ \iterator ->+        allocaArray 256 $ \buffer -> do+          let loop !total = do+                count <- Iterator.read iterator buffer 256+                if count < 256 then pure $ total + count else loop $ total + count+          loop 0+  , bench "Iterator.advance" . whnfIO $ do+      bitmap <- getBitmap+      iterating bitmap $ \iterator -> do+        let loop !total = do+              present <- Iterator.hasValue iterator+              if present then do+                value <- Iterator.currentValue iterator+                _ <- Iterator.advance iterator+                loop $ total + fromIntegral value+              else+                pure (total :: Word64)+        loop 0+  , withResource (Bitmap.wrapCallback $ \_ _ -> pure True) freeHaskellFunPtr $ \getCallback ->+      bench "iterate" . whnfIO $ do+        bitmap <- getBitmap+        callback <- getCallback+        Bitmap.iterate bitmap callback nullPtr+  ]+  where+    iterating bitmap continue =+      allocaBytesAligned Iterator.size Iterator.alignment $ \memory -> do+        let iterator = Iterator memory+        Iterator.init bitmap iterator+        continue iterator++data Buffer = Buffer+  { allocation :: Ptr CChar+  , aligned :: Ptr CChar+  , size :: CSize+  }++withBuffer :: IO Bitmap -> (Bitmap -> IO CSize) -> (Bitmap -> Ptr CChar -> IO a) -> (IO Buffer -> Benchmark) -> Benchmark+withBuffer getBitmap measure write = withResource acquire release+  where+    acquire = do+      bitmap <- getBitmap+      measured <- measure bitmap+      allocation <- mallocBytes $ fromIntegral measured + 32+      let aligned = alignPtr allocation 32+      _ <- write bitmap aligned+      pure Buffer{allocation, aligned, size = measured}+    release buffer = free buffer.allocation++serialization :: IO Bitmap -> [Benchmark]+serialization getBitmap =+  [ bench "portableSizeInBytes" . whnfIO $ getBitmap >>= Bitmap.portableSizeInBytes+  , bench "internalValidate" . whnfIO $ getBitmap >>= (`Bitmap.internalValidate` nullPtr)+  , withBuffer getBitmap Bitmap.portableSizeInBytes Bitmap.portableSerialize $ \getBuffer ->+      bgroup+        "portable"+        [ bench "serialize" . whnfIO $ do+            bitmap <- getBitmap+            buffer <- getBuffer+            Bitmap.portableSerialize bitmap buffer.aligned+        , bench "deserializeBounded" . whnfIO $ do+            buffer <- getBuffer+            Bitmap.portableDeserializeBounded buffer.aligned buffer.size >>= Bitmap.free+        , bench "deserializeFrozen" . whnfIO $ do+            buffer <- getBuffer+            Bitmap.portableDeserializeFrozen buffer.aligned >>= Bitmap.free+        ]+  , withBuffer getBitmap Bitmap.frozenSizeInBytes Bitmap.frozenSerialize $ \getBuffer ->+      bgroup+        "frozen"+        [ bench "serialize" . whnfIO $ do+            bitmap <- getBitmap+            buffer <- getBuffer+            Bitmap.frozenSerialize bitmap buffer.aligned+        , bench "view" . whnfIO $ do+            buffer <- getBuffer+            Bitmap.frozenView buffer.aligned buffer.size >>= Bitmap.free . Bitmap . unConstPtr+        ]+  ]
+ bench/Bench/Bitmap64.hs view
@@ -0,0 +1,203 @@+{-# LANGUAGE DuplicateRecordFields #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++module Bench.Bitmap64 (benchmarks) where++import Bench.Values (Values (..), countWhere, dense, forEach, prefix, scattered64, withValues)+import Control.Monad (void)+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap64 (Bitmap64)+import Data.Bitmap.Roaring.FFI.Bitmap64 qualified as Bitmap64+import Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext qualified as BulkContext+import Data.Bitmap.Roaring.FFI.Bitmap64.Iterator qualified as Iterator+import Data.Word (Word64)+import Foreign.C.Types (CChar, CSize)+import Foreign.Marshal.Alloc (allocaBytesAligned, free, mallocBytes)+import Foreign.Marshal.Array (allocaArray, mallocArray)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (Ptr, freeHaskellFunPtr, nullPtr)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark, bench, bgroup, whnfIO)++size :: Int+size = 1_000_000++benchmarks :: Benchmark+benchmarks =+  withValues (scattered64 1 size) $ \getScattered ->+    withValues (scattered64 2 size) $ \getOther ->+      withValues (dense size) $ \getDense ->+        withBitmap getScattered $ \getScatteredBitmap ->+          withBitmap getOther $ \getOtherBitmap ->+            withBitmap getDense $ \getDenseBitmap ->+              bgroup+                "Bitmap64"+                [ bgroup+                    "construction"+                    [ bgroup "scattered" $ construction getScattered+                    , bgroup "dense" $ construction getDense+                    ]+                , bgroup+                    "queries"+                    [ bgroup "hits" $ queries getScatteredBitmap getScattered+                    , bgroup "misses" $ queries getScatteredBitmap getOther+                    ]+                , bgroup+                    "algebra"+                    [ bgroup "scattered" $ algebra getScatteredBitmap getOtherBitmap+                    , bgroup "mixed" $ algebra getScatteredBitmap getDenseBitmap+                    ]+                , bgroup+                    "traversal"+                    [ bgroup "scattered" $ traversal getScatteredBitmap+                    , bgroup "dense" $ traversal getDenseBitmap+                    ]+                , bgroup+                    "serialization"+                    [ bgroup "scattered" $ serialization getScatteredBitmap+                    , bgroup "dense" $ serialization getDenseBitmap+                    ]+                ]++withBitmap :: IO (Values Word64) -> (IO Bitmap64 -> Benchmark) -> Benchmark+withBitmap getValues = withResource acquire Bitmap64.free+  where+    acquire = do+      values <- getValues+      Bitmap64.ofPtr (fromIntegral values.count) values.pointer++construction :: IO (Values Word64) -> [Benchmark]+construction getValues =+  [ bench "ofPtr" . whnfIO $ do+      values <- getValues+      Bitmap64.ofPtr (fromIntegral values.count) values.pointer >>= Bitmap64.free+  , filling "addMany" $ \bitmap values -> Bitmap64.addMany bitmap (fromIntegral values.count) values.pointer+  , filling "add" $ \bitmap values -> forEach values $ Bitmap64.add bitmap+  , filling "addChecked" $ \bitmap values -> forEach values $ void . Bitmap64.addChecked bitmap+  , filling "addBulk" $ \bitmap values ->+      allocaBytesAligned BulkContext.size BulkContext.alignment $ \memory -> do+        fillBytes memory 0 BulkContext.size+        forEach values . Bitmap64.addBulk bitmap $ BulkContext memory+  ]+  where+    filling :: String -> (Bitmap64 -> Values Word64 -> IO ()) -> Benchmark+    filling name fill =+      bench name . whnfIO $ do+        values <- getValues+        bitmap <- Bitmap64.create+        fill bitmap values+        Bitmap64.free bitmap++queries :: IO Bitmap64 -> IO (Values Word64) -> [Benchmark]+queries getBitmap getProbes =+  [ probing "contains" $ \bitmap probes -> countWhere probes $ Bitmap64.contains bitmap+  , probing "containsBulk" $ \bitmap probes ->+      allocaBytesAligned BulkContext.size BulkContext.alignment $ \memory -> do+        fillBytes memory 0 BulkContext.size+        countWhere probes . Bitmap64.containsBulk bitmap $ BulkContext memory+  , probing "rank of 1000" $ \bitmap probes -> countWhere (prefix 1000 probes) $ fmap (> 0) . Bitmap64.rank bitmap+  , bench "getCardinality" . whnfIO $ getBitmap >>= Bitmap64.getCardinality+  ]+  where+    probing name query =+      bench name . whnfIO $ do+        bitmap <- getBitmap+        probes <- getProbes+        query bitmap probes++algebra :: IO Bitmap64 -> IO Bitmap64 -> [Benchmark]+algebra getLeft getRight =+  [ fresh "and" Bitmap64.and+  , fresh "or" Bitmap64.or+  , fresh "xor" Bitmap64.xor+  , fresh "andnot" Bitmap64.andnot+  , measured "andCardinality" Bitmap64.andCardinality+  , measured "orCardinality" Bitmap64.orCardinality+  , measured "intersect" Bitmap64.intersect+  , measured "equals" Bitmap64.equals+  , bench "copy" . whnfIO $ getLeft >>= Bitmap64.copy >>= Bitmap64.free+  ]+  where+    fresh name operation = measured name $ \left right -> operation left right >>= Bitmap64.free+    measured name operation =+      bench name . whnfIO $ do+        left <- getLeft+        right <- getRight+        operation left right++traversal :: IO Bitmap64 -> [Benchmark]+traversal getBitmap =+  [ withResource (mallocArray size) free $ \getOutput ->+      bench "toUint64Array" . whnfIO $ do+        bitmap <- getBitmap+        getOutput >>= Bitmap64.toUint64Array bitmap+  , bench "Iterator.read" . whnfIO $ do+      bitmap <- getBitmap+      iterating bitmap $ \iterator ->+        allocaArray 256 $ \buffer -> do+          let loop !total = do+                count <- Iterator.read iterator buffer 256+                if count < 256 then pure $ total + count else loop $ total + count+          loop 0+  , bench "Iterator.advance" . whnfIO $ do+      bitmap <- getBitmap+      iterating bitmap $ \iterator -> do+        let loop !total = do+              present <- Iterator.hasValue iterator+              if present then do+                value <- Iterator.value iterator+                _ <- Iterator.advance iterator+                loop $ total + value+              else+                pure total+        loop 0+  , withResource (Bitmap.wrapCallback64 $ \_ _ -> pure True) freeHaskellFunPtr $ \getCallback ->+      bench "iterate" . whnfIO $ do+        bitmap <- getBitmap+        callback <- getCallback+        Bitmap64.iterate bitmap callback nullPtr+  ]+  where+    iterating bitmap continue = do+      iterator <- Iterator.create bitmap+      result <- continue iterator+      Iterator.free iterator+      pure result++data Buffer = Buffer+  { pointer :: Ptr CChar+  , size :: CSize+  }++withPortable :: IO Bitmap64 -> (IO Buffer -> Benchmark) -> Benchmark+withPortable getBitmap = withResource acquire release+  where+    acquire = do+      bitmap <- getBitmap+      measured <- Bitmap64.portableSizeInBytes bitmap+      pointer <- mallocBytes $ fromIntegral measured+      _ <- Bitmap64.portableSerialize bitmap pointer+      pure Buffer{pointer, size = measured}+    release buffer = free buffer.pointer++serialization :: IO Bitmap64 -> [Benchmark]+serialization getBitmap =+  [ bench "portableSizeInBytes" . whnfIO $ getBitmap >>= Bitmap64.portableSizeInBytes+  , bench "internalValidate" . whnfIO $ getBitmap >>= (`Bitmap64.internalValidate` nullPtr)+  , withPortable getBitmap $ \getBuffer ->+      bgroup+        "portable"+        [ bench "serialize" . whnfIO $ do+            bitmap <- getBitmap+            buffer <- getBuffer+            Bitmap64.portableSerialize bitmap buffer.pointer+        , bench "deserializeBounded" . whnfIO $ do+            buffer <- getBuffer+            Bitmap64.portableDeserializeBounded buffer.pointer buffer.size >>= Bitmap64.free+        , bench "deserializeFrozen" . whnfIO $ do+            buffer <- getBuffer+            Bitmap64.portableDeserializeFrozen buffer.pointer buffer.size >>= Bitmap64.free+        ]+  ]
+ bench/Bench/Bitset.hs view
@@ -0,0 +1,111 @@+{-# LANGUAGE OverloadedRecordDot #-}++module Bench.Bitset (benchmarks) where++import Bench.Values (Values (..), countWhere, dense, forEach, scattered, withValues)+import Data.Bitmap.Roaring.FFI.Bitset (Bitset)+import Data.Bitmap.Roaring.FFI.Bitset qualified as Bitset+import Data.Bits ((.&.))+import Foreign.C.Types (CSize)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Marshal.Array (allocaArray)+import Foreign.Ptr (freeHaskellFunPtr, nullPtr)+import Foreign.Storable (peek, poke)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark, bench, bgroup, whnfIO)++size :: Int+size = 1_000_000++benchmarks :: Benchmark+benchmarks =+  withValues (map (fromIntegral . (.&. 0xFFFFFF)) $ scattered 1 size) $ \getScattered ->+    withValues (map (fromIntegral . (.&. 0xFFFFFF)) $ scattered 2 size) $ \getOther ->+      withValues (dense size) $ \getDense ->+        withBitset getScattered $ \getScatteredBitset ->+          withBitset getOther $ \getOtherBitset ->+            bgroup+              "Bitset"+              [ bgroup+                  "construction"+                  [ bgroup "scattered" $ construction getScattered+                  , bgroup "dense" $ construction getDense+                  ]+              , bgroup+                  "queries"+                  [ probing getScatteredBitset getScattered "get hits"+                  , probing getScatteredBitset getOther "get misses"+                  , bench "count" . whnfIO $ getScatteredBitset >>= Bitset.count+                  ]+              , bgroup+                  "algebra"+                  [ measured getScatteredBitset getOtherBitset "unionCount" Bitset.unionCount+                  , measured getScatteredBitset getOtherBitset "intersectionCount" Bitset.intersectionCount+                  , bench "copy and inplaceUnion" . whnfIO $ do+                      left <- getScatteredBitset+                      right <- getOtherBitset+                      copy <- Bitset.copy left+                      _ <- Bitset.inplaceUnion copy right+                      Bitset.free copy+                  ]+              , bgroup+                  "traversal"+                  [ bench "nextSetBits" . whnfIO $ getScatteredBitset >>= nextSetBits+                  , withResource (Bitset.wrapCallback $ \_ _ -> pure True) freeHaskellFunPtr $ \getCallback ->+                      bench "forEach" . whnfIO $ do+                        bitset <- getScatteredBitset+                        callback <- getCallback+                        Bitset.forEach bitset callback nullPtr+                  ]+              ]++withBitset :: IO (Values CSize) -> (IO Bitset -> Benchmark) -> Benchmark+withBitset getValues = withResource acquire Bitset.free+  where+    acquire = do+      values <- getValues+      bitset <- Bitset.create+      forEach values $ Bitset.set bitset+      pure bitset++construction :: IO (Values CSize) -> [Benchmark]+construction getValues =+  [ bench "set" . whnfIO $ do+      values <- getValues+      bitset <- Bitset.create+      forEach values $ Bitset.set bitset+      Bitset.free bitset+  , bench "set with capacity" . whnfIO $ do+      values <- getValues+      bitset <- Bitset.createWithCapacity 0x1000000+      forEach values $ Bitset.set bitset+      Bitset.free bitset+  ]++probing :: IO Bitset -> IO (Values CSize) -> String -> Benchmark+probing getBitset getProbes name =+  bench name . whnfIO $ do+    bitset <- getBitset+    probes <- getProbes+    countWhere probes $ Bitset.get bitset++measured :: IO Bitset -> IO Bitset -> String -> (Bitset -> Bitset -> IO CSize) -> Benchmark+measured getLeft getRight name operation =+  bench name . whnfIO $ do+    left <- getLeft+    right <- getRight+    operation left right++nextSetBits :: Bitset -> IO CSize+nextSetBits bitset =+  allocaArray 256 $ \buffer ->+    alloca $ \start -> do+      poke start 0+      let loop !total = do+            count <- Bitset.nextSetBits bitset buffer 256 start+            if count == 0 then+              pure total+            else do+              peek start >>= poke start . (+ 1)+              loop $ total + count+      loop 0
+ bench/Bench/Comparison.hs view
@@ -0,0 +1,352 @@+{-# LANGUAGE DuplicateRecordFields #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++module Bench.Comparison+  ( benchmarks+  , scaling+  ) where++import Bench.Baseline qualified as Baseline+import Bench.Values (dense, scattered)+import Control.DeepSeq (NFData)+import Control.Exception (bracket)+import Data.Bit (Bit (..), countBits, listBits, zipBits)+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap)+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator (Iterator (..))+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator qualified as Iterator+import Data.Bitmap.Roaring.FFI.Bitmap64 (Bitmap64)+import Data.Bitmap.Roaring.FFI.Bitmap64 qualified as Bitmap64+import Data.Bitmap.Roaring.FFI.Bitmap64.Iterator qualified as Iterator64+import Data.Bitmap.Roaring.FFI.Bitset (Bitset)+import Data.Bitmap.Roaring.FFI.Bitset qualified as Bitset+import Data.Bits ((.&.), (.|.))+import Data.IntSet (IntSet)+import Data.IntSet qualified as IntSet+import Data.Vector.Storable qualified as Storable+import Data.Vector.Unboxed qualified as Unboxed+import Data.Vector.Unboxed.Mutable qualified as Unboxed.Mutable+import Data.Word (Word32, Word64)+import Foreign.C.Types (CSize)+import Foreign.Marshal.Alloc (alloca, allocaBytesAligned)+import Foreign.Marshal.Array (allocaArray)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable, peek, peekElemOff, poke)+import GHC.Generics (Generic)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark, Benchmarkable, bgroup, env, nf, whnf, whnfIO)++data Dataset = Dataset+  { name :: String+  , universe :: Maybe Int+  , left :: [Word32]+  , right :: [Word32]+  }++data Inputs = Inputs+  { left :: Storable.Vector Word32+  , right :: Storable.Vector Word32+  , probes :: Storable.Vector Word32+  , left64 :: Storable.Vector Word64+  , right64 :: Storable.Vector Word64+  , probes64 :: Storable.Vector Word64+  }+  deriving (Generic)++instance NFData Inputs++data Contender = Contender+  { name :: String+  , construction :: Benchmarkable+  , member :: Benchmarkable+  , cardinality :: Benchmarkable+  , union :: Benchmarkable+  , intersection :: Benchmarkable+  , iteration :: Benchmarkable+  }++size :: Int+size = 1_000_000++benchmarks :: Benchmark+benchmarks =+  bgroup "comparison" $+    map+      (dataset "comparison")+      [ Dataset+          { name = "scattered"+          , universe = Nothing+          , left = scattered 1 size+          , right = scattered 2 size+          }+      , Dataset+          { name = "bounded"+          , universe = Just 0x1000000+          , left = map (.&. 0xFFFFFF) $ scattered 1 size+          , right = map (.&. 0xFFFFFF) $ scattered 2 size+          }+      , Dataset+          { name = "dense"+          , universe = Just 0x200000+          , left = dense size+          , right = map (+ fromIntegral (size `div` 2)) $ dense size+          }+      ]++scaling :: Benchmark+scaling =+  bgroup "scaling" $ map (dataset "scaling" . sparse) [1_000, 3_000, 10_000, 30_000, 100_000, 300_000, 1_000_000]+  where+    sparse count =+      Dataset+        { name = show count+        , universe = Nothing+        , left = scattered 1 count+        , right = scattered 2 count+        }++dataset :: String -> Dataset -> Benchmark+dataset group spec =+  env (pure prepared) $ \inputs ->+    withIntSet inputs $ \intSet ->+      withBitmap inputs $ \bitmap ->+        withBitmap64 inputs $ \bitmap64 ->+          withBitVectors spec.universe inputs $ \bitVectors ->+            bgroup spec.name . contests group spec.name intSet $ bitmap : bitmap64 : bitVectors+  where+    left = Storable.fromList spec.left+    right = Storable.fromList spec.right+    probes = Storable.fromList . concat $ zipWith (\hit miss -> [hit, miss]) spec.left spec.right+    prepared =+      Inputs+        { left+        , right+        , probes+        , left64 = Storable.map fromIntegral left+        , right64 = Storable.map fromIntegral right+        , probes64 = Storable.map fromIntegral probes+        }++contests :: String -> String -> Contender -> [Contender] -> [Benchmark]+contests group datasetName baseline others =+  [ versus "construction" (.construction)+  , versus "member" (.member)+  , versus "cardinality" (.cardinality)+  , versus "union" (.union)+  , versus "intersection" (.intersection)+  , versus "iteration" (.iteration)+  ]+  where+    versus = Baseline.versus [datasetName, group] baseline others++withBitmap :: Inputs -> (Contender -> Benchmark) -> Benchmark+withBitmap inputs continue =+  withResource (bitmapOf inputs.left) Bitmap.free $ \getLeft ->+    withResource (bitmapOf inputs.right) Bitmap.free $ \getRight ->+      continue+        Contender+          { name = "Bitmap"+          , construction = whnfIO $ bitmapOf inputs.left >>= Bitmap.free+          , member = whnfIO $ do+              bitmap <- getLeft+              tallyIO (Bitmap.contains bitmap) inputs.probes+          , cardinality = whnfIO $ getLeft >>= Bitmap.getCardinality+          , union = whnfIO $ combined getLeft getRight Bitmap.or+          , intersection = whnfIO $ combined getLeft getRight Bitmap.and+          , iteration = whnfIO $ getLeft >>= sumBitmap+          }+  where+    combined getLeft getRight operation = do+      left <- getLeft+      right <- getRight+      operation left right >>= Bitmap.free++bitmapOf :: Storable.Vector Word32 -> IO Bitmap+bitmapOf values = Storable.unsafeWith values . Bitmap.ofPtr . fromIntegral $ Storable.length values++sumBitmap :: Bitmap -> IO Word64+sumBitmap bitmap =+  allocaBytesAligned Iterator.size Iterator.alignment $ \memory ->+    allocaArray chunk $ \buffer -> do+      let+        iterator = Iterator memory+        loop !accumulated = do+          count <- fromIntegral <$> Iterator.read iterator buffer (fromIntegral chunk)+          summed <- sumBuffer buffer count accumulated+          if count < chunk then pure summed else loop summed+      Iterator.init bitmap iterator+      loop 0+  where+    chunk = 256++withBitmap64 :: Inputs -> (Contender -> Benchmark) -> Benchmark+withBitmap64 inputs continue =+  withResource (bitmap64Of inputs.left64) Bitmap64.free $ \getLeft ->+    withResource (bitmap64Of inputs.right64) Bitmap64.free $ \getRight ->+      continue+        Contender+          { name = "Bitmap64"+          , construction = whnfIO $ bitmap64Of inputs.left64 >>= Bitmap64.free+          , member = whnfIO $ do+              bitmap <- getLeft+              tallyIO (Bitmap64.contains bitmap) inputs.probes64+          , cardinality = whnfIO $ getLeft >>= Bitmap64.getCardinality+          , union = whnfIO $ combined getLeft getRight Bitmap64.or+          , intersection = whnfIO $ combined getLeft getRight Bitmap64.and+          , iteration = whnfIO $ getLeft >>= sumBitmap64+          }+  where+    combined getLeft getRight operation = do+      left <- getLeft+      right <- getRight+      operation left right >>= Bitmap64.free++bitmap64Of :: Storable.Vector Word64 -> IO Bitmap64+bitmap64Of values = Storable.unsafeWith values . Bitmap64.ofPtr . fromIntegral $ Storable.length values++sumBitmap64 :: Bitmap64 -> IO Word64+sumBitmap64 bitmap =+  bracket (Iterator64.create bitmap) Iterator64.free $ \iterator ->+    allocaArray chunk $ \buffer -> do+      let loop !accumulated = do+            count <- fromIntegral <$> Iterator64.read iterator buffer (fromIntegral chunk)+            summed <- sumBuffer buffer count accumulated+            if count < chunk then pure summed else loop summed+      loop 0+  where+    chunk = 256++withIntSet :: Inputs -> (Contender -> Benchmark) -> Benchmark+withIntSet inputs continue =+  env (pure (intSetOf inputs.left, intSetOf inputs.right)) $ \ ~(left, right) ->+    continue+      Contender+        { name = "IntSet"+        , construction = nf intSetOf inputs.left+        , member = whnf (countMembers inputs.probes) left+        , cardinality = whnf IntSet.size left+        , union = nf (IntSet.union left) right+        , intersection = nf (IntSet.intersection left) right+        , iteration = whnf (IntSet.foldl' (+) 0) left+        }+  where+    countMembers probes set = tally isMember probes+      where+        isMember value = IntSet.member (fromIntegral value) set++intSetOf :: Storable.Vector Word32 -> IntSet+intSetOf = Storable.foldl' (\set value -> IntSet.insert (fromIntegral value) set) IntSet.empty++withBitVectors :: Maybe Int -> Inputs -> ([Contender] -> Benchmark) -> Benchmark+withBitVectors universe inputs continue =+  case universe of+    Nothing -> continue []+    Just bits ->+      withBitset bits inputs $ \bitset ->+        withBitvec bits inputs $ \bitvec ->+          continue [bitset, bitvec]++withBitset :: Int -> Inputs -> (Contender -> Benchmark) -> Benchmark+withBitset universe inputs continue =+  withResource (bitsetOf universe inputs.left) Bitset.free $ \getLeft ->+    withResource (bitsetOf universe inputs.right) Bitset.free $ \getRight ->+      continue+        Contender+          { name = "Bitset"+          , construction = whnfIO $ bitsetOf universe inputs.left >>= Bitset.free+          , member = whnfIO $ do+              bitset <- getLeft+              tallyIO (Bitset.get bitset . fromIntegral) inputs.probes+          , cardinality = whnfIO $ getLeft >>= Bitset.count+          , union = whnfIO $ combined getLeft getRight Bitset.inplaceUnion+          , intersection = whnfIO $ combined getLeft getRight Bitset.inplaceIntersection+          , iteration = whnfIO $ getLeft >>= sumBitset+          }+  where+    combined getLeft getRight operation = do+      left <- getLeft+      right <- getRight+      copy <- Bitset.copy left+      _ <- operation copy right+      Bitset.free copy++bitsetOf :: Int -> Storable.Vector Word32 -> IO Bitset+bitsetOf universe values = do+  bitset <- Bitset.createWithCapacity $ fromIntegral universe+  Storable.mapM_ (Bitset.set bitset . fromIntegral) values+  pure bitset++sumBitset :: Bitset -> IO CSize+sumBitset bitset =+  allocaArray chunk $ \buffer ->+    alloca $ \start -> do+      poke start 0+      let loop !accumulated = do+            count <- fromIntegral <$> Bitset.nextSetBits bitset buffer (fromIntegral chunk) start+            if count == 0 then+              pure accumulated+            else do+              summed <- sumBuffer buffer count accumulated+              peek start >>= poke start . (+ 1)+              loop summed+      loop 0+  where+    chunk = 256++withBitvec :: Int -> Inputs -> (Contender -> Benchmark) -> Benchmark+withBitvec universe inputs continue =+  env (pure (bitvecOf universe inputs.left, bitvecOf universe inputs.right)) $ \ ~(left, right) ->+    continue+      Contender+        { name = "bitvec"+        , construction = whnf (bitvecOf universe) inputs.left+        , member = whnf (countMembers inputs.probes) left+        , cardinality = whnf countBits left+        , union = whnf (zipBits (.|.) left) right+        , intersection = whnf (zipBits (.&.) left) right+        , iteration = whnf (foldl' (+) 0 . listBits) left+        }+  where+    countMembers probes bits = tally isSet probes+      where+        isSet value = unBit . Unboxed.unsafeIndex bits $ fromIntegral value++bitvecOf :: Int -> Storable.Vector Word32 -> Unboxed.Vector Bit+bitvecOf universe values =+  Unboxed.create $ do+    bits <- Unboxed.Mutable.replicate universe $ Bit False+    Storable.forM_ values $ \value -> Unboxed.Mutable.unsafeWrite bits (fromIntegral value) $ Bit True+    pure bits++tallyIO :: (Storable a) => (a -> IO Bool) -> Storable.Vector a -> IO Int+tallyIO query values = go 0 0+  where+    go !hits index+      | index == Storable.length values = pure hits+      | otherwise = do+          hit <- query $ Storable.unsafeIndex values index+          go (counted hits hit) $ index + 1+{-# INLINE tallyIO #-}++tally :: (Storable a) => (a -> Bool) -> Storable.Vector a -> Int+tally query values = go 0 0+  where+    go !hits index+      | index == Storable.length values = hits+      | otherwise = go (counted hits . query $ Storable.unsafeIndex values index) $ index + 1+{-# INLINE tally #-}++counted :: Int -> Bool -> Int+counted hits hit = if hit then hits + 1 else hits+{-# INLINE counted #-}++sumBuffer :: (Storable a, Integral a, Num total) => Ptr a -> Int -> total -> IO total+sumBuffer buffer count = go 0+  where+    go index !accumulated+      | index == count = pure accumulated+      | otherwise = do+          value <- peekElemOff buffer index+          go (index + 1) $ accumulated + fromIntegral value+{-# INLINE sumBuffer #-}
+ bench/Bench/Unfair.hs view
@@ -0,0 +1,228 @@+{-# LANGUAGE DuplicateRecordFields #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++module Bench.Unfair (benchmarks) where++import Bench.Baseline qualified as Baseline+import Bench.Values (dense, scattered)+import Control.Exception (bracket)+import Data.Binary qualified as Binary+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap)+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap64 (Bitmap64)+import Data.Bitmap.Roaring.FFI.Bitmap64 qualified as Bitmap64+import Data.Bitmap.Roaring.FFI.Bitmap64.Iterator qualified as Iterator64+import Data.ByteString.Lazy qualified as Lazy+import Data.IntSet (IntSet)+import Data.IntSet qualified as IntSet+import Data.Word (Word32, Word64)+import Foreign.C.Types (CChar, CSize)+import Foreign.Marshal.Alloc (alloca, allocaBytes, free, mallocBytes)+import Foreign.Marshal.Array (allocaArray, peekArray, withArrayLen)+import Foreign.Ptr (Ptr)+import Foreign.Storable (peek)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark, Benchmarkable, bgroup, env, nf, nfIO, whnf, whnfIO)++data Dataset = Dataset+  { name :: String+  , left :: [Word32]+  , right :: [Word32]+  , low :: Word32+  , high :: Word32+  }++data Contender = Contender+  { name :: String+  , difference :: Benchmarkable+  , intersectionSize :: Benchmarkable+  , jaccard :: Benchmarkable+  , rangeCount :: Benchmarkable+  , rank :: Benchmarkable+  , page :: Benchmarkable+  , shift :: Benchmarkable+  , serialize :: Benchmarkable+  , deserialize :: Benchmarkable+  , view :: Benchmarkable+  }++size :: Int+size = 1_000_000++pageOffset :: Int+pageOffset = size `div` 2++pageSize :: Int+pageSize = 100++offset :: Int+offset = 1_000_000++benchmarks :: Benchmark+benchmarks =+  bgroup "unfair" $+    map+      dataset+      [ Dataset+          { name = "sparse"+          , left = scattered 1 size+          , right = scattered 2 size+          , low = 0x40000000+          , high = 0xC0000000+          }+      , Dataset+          { name = "dense"+          , left = dense size+          , right = map (+ fromIntegral (size `div` 2)) $ dense size+          , low = 250_000+          , high = 750_000+          }+      ]++dataset :: Dataset -> Benchmark+dataset spec =+  withIntSet spec $ \intSet ->+    withBitmap spec $ \bitmap ->+      withBitmap64 spec $ \bitmap64 ->+        bgroup spec.name $ contests spec.name intSet [bitmap, bitmap64]++contests :: String -> Contender -> [Contender] -> [Benchmark]+contests datasetName baseline others =+  [ versus "difference" (.difference)+  , versus "intersection-size" (.intersectionSize)+  , versus "jaccard" (.jaccard)+  , versus "range-count" (.rangeCount)+  , versus "rank" (.rank)+  , versus "page" (.page)+  , versus "shift" (.shift)+  , versus "serialize" (.serialize)+  , versus "deserialize" (.deserialize)+  , versus "view" (.view)+  ]+  where+    versus = Baseline.versus [datasetName, "unfair"] baseline others++middle :: Dataset -> Word32+middle spec = spec.low + (spec.high - spec.low) `div` 2++withIntSet :: Dataset -> (Contender -> Benchmark) -> Benchmark+withIntSet spec continue =+  env (pure (left, right, Binary.encode left)) $ \ ~(intSet, other, encoded) ->+    continue+      Contender+        { name = "IntSet"+        , difference = nf (IntSet.difference intSet) other+        , intersectionSize = whnf (IntSet.size . IntSet.intersection intSet) other+        , jaccard = whnf (jaccardOf intSet) other+        , rangeCount = whnf rangeCountOf intSet+        , rank = whnf rankOf intSet+        , page = nf pageOf intSet+        , shift = nf (IntSet.mapMonotonic (+ offset)) intSet+        , serialize = nf Binary.encode intSet+        , deserialize = nf decode encoded+        , view = nf decode encoded+        }+  where+    left = IntSet.fromList $ map fromIntegral spec.left+    right = IntSet.fromList $ map fromIntegral spec.right+    decode :: Lazy.ByteString -> IntSet+    decode = Binary.decode+    jaccardOf one two = fromIntegral common / fromIntegral (IntSet.size one + IntSet.size two - common) :: Double+      where+        common = IntSet.size $ IntSet.intersection one two+    rangeCountOf set = IntSet.size inside+      where+        (_, above) = IntSet.split (fromIntegral spec.low - 1) set+        (inside, _) = IntSet.split (fromIntegral spec.high) above+    rankOf set = IntSet.size below + fromEnum found+      where+        (below, found, _) = IntSet.splitMember (fromIntegral $ middle spec) set+    pageOf = take pageSize . drop pageOffset . IntSet.toAscList++withBitmap :: Dataset -> (Contender -> Benchmark) -> Benchmark+withBitmap spec continue =+  withResource (bitmapOf spec.left) Bitmap.free $ \getLeft ->+    withResource (bitmapOf spec.right) Bitmap.free $ \getRight ->+      withResource (getLeft >>= serialized) (free . fst) $ \getBytes ->+        let both operation = do+              left <- getLeft+              right <- getRight+              operation left right+        in continue+             Contender+               { name = "Bitmap"+               , difference = whnfIO $ both Bitmap.andnot >>= Bitmap.free+               , intersectionSize = whnfIO $ both Bitmap.andCardinality+               , jaccard = whnfIO $ both Bitmap.jaccardIndex+               , rangeCount = whnfIO $ getLeft >>= \bitmap -> Bitmap.rangeCardinality bitmap (fromIntegral spec.low) $ fromIntegral spec.high+               , rank = whnfIO $ getLeft >>= \bitmap -> Bitmap.rank bitmap $ middle spec+               , page = nfIO $ getLeft >>= pageOf+               , shift = whnfIO $ getLeft >>= \bitmap -> Bitmap.addOffset bitmap (fromIntegral offset) >>= Bitmap.free+               , serialize = whnfIO $ getLeft >>= serializeInto+               , deserialize = whnfIO $ getBytes >>= \(bytes, count) -> Bitmap.portableDeserializeBounded bytes count >>= Bitmap.free+               , view = whnfIO $ getBytes >>= \(bytes, _) -> Bitmap.portableDeserializeFrozen bytes >>= Bitmap.free+               }+  where+    serialized bitmap = do+      count <- Bitmap.portableSizeInBytes bitmap+      bytes <- mallocBytes $ fromIntegral count+      written <- Bitmap.portableSerialize bitmap bytes+      pure (bytes, written)+    serializeInto bitmap = do+      count <- Bitmap.portableSizeInBytes bitmap+      allocaBytes (fromIntegral count) $ Bitmap.portableSerialize bitmap+    pageOf bitmap =+      allocaArray pageSize $ \buffer -> do+        _ <- Bitmap.rangeUint32Array bitmap (fromIntegral pageOffset) (fromIntegral pageSize) buffer+        peekArray pageSize buffer++bitmapOf :: [Word32] -> IO Bitmap+bitmapOf values = withArrayLen values $ \count ptr -> Bitmap.ofPtr (fromIntegral count) ptr++withBitmap64 :: Dataset -> (Contender -> Benchmark) -> Benchmark+withBitmap64 spec continue =+  withResource (bitmap64Of spec.left) Bitmap64.free $ \getLeft ->+    withResource (bitmap64Of spec.right) Bitmap64.free $ \getRight ->+      withResource (getLeft >>= serialized) (free . fst) $ \getBytes ->+        let both operation = do+              left <- getLeft+              right <- getRight+              operation left right+        in continue+             Contender+               { name = "Bitmap64"+               , difference = whnfIO $ both Bitmap64.andnot >>= Bitmap64.free+               , intersectionSize = whnfIO $ both Bitmap64.andCardinality+               , jaccard = whnfIO $ both Bitmap64.jaccardIndex+               , rangeCount = whnfIO $ getLeft >>= \bitmap -> Bitmap64.rangeCardinality bitmap (fromIntegral spec.low) $ fromIntegral spec.high+               , rank = whnfIO $ getLeft >>= \bitmap -> Bitmap64.rank bitmap . fromIntegral $ middle spec+               , page = nfIO $ getLeft >>= pageOf+               , shift = whnfIO $ getLeft >>= \bitmap -> Bitmap64.addOffset bitmap (fromIntegral offset) >>= Bitmap64.free+               , serialize = whnfIO $ getLeft >>= serializeInto+               , deserialize = whnfIO $ getBytes >>= \(bytes, count) -> Bitmap64.portableDeserializeBounded bytes count >>= Bitmap64.free+               , view = whnfIO $ getBytes >>= \(bytes, count) -> Bitmap64.portableDeserializeFrozen bytes count >>= Bitmap64.free+               }+  where+    serialized :: Bitmap64 -> IO (Ptr CChar, CSize)+    serialized bitmap = do+      count <- Bitmap64.portableSizeInBytes bitmap+      bytes <- mallocBytes $ fromIntegral count+      written <- Bitmap64.portableSerialize bitmap bytes+      pure (bytes, written)+    serializeInto bitmap = do+      count <- Bitmap64.portableSizeInBytes bitmap+      allocaBytes (fromIntegral count) $ Bitmap64.portableSerialize bitmap+    pageOf :: Bitmap64 -> IO [Word64]+    pageOf bitmap =+      alloca $ \start -> do+        _ <- Bitmap64.select bitmap (fromIntegral pageOffset) start+        first <- peek start+        bracket (Iterator64.create bitmap) Iterator64.free $ \iterator ->+          allocaArray pageSize $ \buffer -> do+            _ <- Iterator64.moveEqualOrLarger iterator first+            count <- Iterator64.read iterator buffer $ fromIntegral pageSize+            peekArray (fromIntegral count) buffer++bitmap64Of :: [Word32] -> IO Bitmap64+bitmap64Of values = withArrayLen (map fromIntegral values) $ \count ptr -> Bitmap64.ofPtr (fromIntegral count) ptr
+ bench/Bench/Values.hs view
@@ -0,0 +1,73 @@+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++module Bench.Values+  ( Values (..)+  , dense+  , scattered+  , scattered64+  , withValues+  , prefix+  , forEach+  , countWhere+  ) where++import Data.Bits (shiftL, shiftR)+import Data.Word (Word32, Word64)+import Foreign.Marshal.Alloc (free)+import Foreign.Marshal.Array (mallocArray, pokeArray)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable, peekElemOff)+import Test.Tasty (withResource)+import Test.Tasty.Bench (Benchmark)++data Values a = Values+  { pointer :: Ptr a+  , count :: Int+  }++dense :: (Num a, Enum a) => Int -> [a]+dense size = take size $ enumFrom 0++scattered :: Word32 -> Int -> [Word32]+scattered seed size = take size . drop 1 $ iterate step seed+  where+    step value = value * 1664525 + 1013904223++scattered64 :: Word32 -> Int -> [Word64]+scattered64 seed = zipWith place (cycle [0 .. 3]) . scattered seed+  where+    place segment value = segment `shiftL` 40 + fromIntegral value `shiftR` 2++withValues :: (Storable a) => [a] -> (IO (Values a) -> Benchmark) -> Benchmark+withValues values = withResource acquire release+  where+    acquire = do+      let count = length values+      pointer <- mallocArray count+      pokeArray pointer values+      pure Values{pointer, count}+    release prepared = free prepared.pointer++prefix :: Int -> Values a -> Values a+prefix limit values = Values{pointer = values.pointer, count = min limit values.count}++forEach :: (Storable a) => Values a -> (a -> IO ()) -> IO ()+forEach values action = go 0+  where+    go index+      | index == values.count = pure ()+      | otherwise = do+          peekElemOff values.pointer index >>= action+          go $ index + 1+{-# INLINE forEach #-}++countWhere :: (Storable a) => Values a -> (a -> IO Bool) -> IO Int+countWhere values query = go 0 0+  where+    go !hits index+      | index == values.count = pure hits+      | otherwise = do+          hit <- peekElemOff values.pointer index >>= query+          go (if hit then hits + 1 else hits) $ index + 1+{-# INLINE countWhere #-}
+ bench/Main.hs view
@@ -0,0 +1,19 @@+module Main (main) where++import Bench.Bitmap qualified+import Bench.Bitmap64 qualified+import Bench.Bitset qualified+import Bench.Comparison qualified+import Bench.Unfair qualified+import Test.Tasty.Bench (defaultMain)++main :: IO ()+main =+  defaultMain+    [ Bench.Bitmap.benchmarks+    , Bench.Bitmap64.benchmarks+    , Bench.Bitset.benchmarks+    , Bench.Comparison.benchmarks+    , Bench.Comparison.scaling+    , Bench.Unfair.benchmarks+    ]
+ bench/sizes/Sizes.hs view
@@ -0,0 +1,125 @@+module Main (main) where++import Bench.Values (dense, scattered)+import Control.Exception (bracket, evaluate)+import Control.Monad (forM_)+import Data.Binary qualified as Binary+import Data.Bit (Bit (..))+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap64 qualified as Bitmap64+import Data.Bitmap.Roaring.FFI.Bitset qualified as Bitset+import Data.Bits ((.&.))+import Data.ByteString.Lazy qualified as Lazy+import Data.IntSet qualified as IntSet+import Data.List (intercalate)+import Data.Vector.Unboxed qualified as Unboxed+import Data.Vector.Unboxed.Mutable qualified as Unboxed.Mutable+import Data.Word (Word32)+import Foreign.C.Types (CSize (..))+import Foreign.Marshal.Array (withArrayLen)+import GHC.Compact (compact, compactSize)+import System.Environment (getArgs)+import System.Exit (die)++foreign import ccall unsafe "croaring_ffi_count_allocations"+  countAllocations :: IO ()++foreign import ccall unsafe "croaring_ffi_allocated_bytes"+  allocatedBytes :: IO CSize++size :: Int+size = 1_000_000++main :: IO ()+main = do+  arguments <- getArgs+  case arguments of+    [] -> table >>= putStr+    [output] -> table >>= writeFile output+    _ -> die "usage: croaring-ffi-sizes [OUTPUT.csv]"++table :: IO String+table = do+  countAllocations+  rows <- concat <$> mapM measure datasets+  pure . unlines $ "dataset,structure,bytes,values" : map (intercalate ",") rows++datasets :: [(String, [Word32], Bool)]+datasets =+  [ ("sparse", scattered 1 size, False)+  , ("bounded", map (.&. 0xFFFFFF) $ scattered 1 size, True)+  , ("dense", dense size, True)+  ]++measure :: (String, [Word32], Bool) -> IO [[String]]+measure (name, values, bitArrays) = do+  intSet <- evaluate . IntSet.fromList $ map fromIntegral values+  let+    count = IntSet.size intSet+    row structure bytes = [name, structure, show bytes, show count]+  intSetMemory <- compact intSet >>= compactSize+  let intSetBinary = Lazy.length $ Binary.encode intSet+  (roaring32Memory, roaring32Portable) <- roaring32 values+  (roaring64Memory, roaring64Portable) <- roaring64 values+  bitArrayRows <-+    if bitArrays then do+      bitsetMemory <- bitset values+      bitvecMemory <- bitvec values+      pure [row "bitset-memory" bitsetMemory, row "bitvec-memory" bitvecMemory]+    else+      pure []+  pure $+    [ row "intset-memory" $ toInteger intSetMemory+    , row "intset-binary" $ toInteger intSetBinary+    , row "roaring32-memory" roaring32Memory+    , row "roaring32-portable" roaring32Portable+    , row "roaring64-memory" roaring64Memory+    , row "roaring64-portable" roaring64Portable+    ]+      <> bitArrayRows++allocatedBy :: IO a -> (a -> IO ()) -> (a -> IO b) -> IO (Integer, b)+allocatedBy create release inspect = do+  before <- allocatedBytes+  bracket create release $ \structure -> do+    after <- allocatedBytes+    result <- inspect structure+    pure (toInteger after - toInteger before, result)++roaring32 :: [Word32] -> IO (Integer, Integer)+roaring32 values = allocatedBy create Bitmap.free $ fmap toInteger . Bitmap.portableSizeInBytes+  where+    create = do+      bitmap <- withArrayLen values $ \count ptr -> Bitmap.ofPtr (fromIntegral count) ptr+      _ <- Bitmap.runOptimize bitmap+      _ <- Bitmap.shrinkToFit bitmap+      pure bitmap++roaring64 :: [Word32] -> IO (Integer, Integer)+roaring64 values = allocatedBy create Bitmap64.free $ fmap toInteger . Bitmap64.portableSizeInBytes+  where+    create = do+      bitmap <- withArrayLen (map fromIntegral values) $ \count ptr -> Bitmap64.ofPtr (fromIntegral count) ptr+      _ <- Bitmap64.runOptimize bitmap+      _ <- Bitmap64.shrinkToFit bitmap+      pure bitmap++bitset :: [Word32] -> IO Integer+bitset values = fst <$> allocatedBy create Bitset.free (const $ pure ())+  where+    create = do+      structure <- Bitset.create+      forM_ values $ Bitset.set structure . fromIntegral+      _ <- Bitset.trim structure+      pure structure++bitvec :: [Word32] -> IO Integer+bitvec values = do+  bits <- evaluate built+  pure . toInteger $ 8 * ((Unboxed.length bits + 63) `div` 64)+  where+    built =+      Unboxed.create $ do+        vector <- Unboxed.Mutable.replicate (fromIntegral (maximum values) + 1) $ Bit False+        forM_ values $ \value -> Unboxed.Mutable.unsafeWrite vector (fromIntegral value) $ Bit True+        pure vector
+ bench/sizes/allocation-counter.c view
@@ -0,0 +1,56 @@+#include <stdlib.h>+#include <roaring/memory.h>++#if defined(__APPLE__)+#include <malloc/malloc.h>+static size_t usable_size(void *block) { return malloc_size(block); }+#else+#include <malloc.h>+static size_t usable_size(void *block) { return malloc_usable_size(block); }+#endif++static size_t allocated;++static void *counting_malloc(size_t size) {+    void *block = malloc(size);+    if (block) allocated += usable_size(block);+    return block;+}++static void *counting_realloc(void *block, size_t size) {+    size_t before = block ? usable_size(block) : 0;+    void *moved = realloc(block, size);+    if (moved) {+        allocated -= before;+        allocated += usable_size(moved);+    }+    return moved;+}++static void *counting_calloc(size_t count, size_t size) {+    void *block = calloc(count, size);+    if (block) allocated += usable_size(block);+    return block;+}++static void counting_free(void *block) {+    if (block) allocated -= usable_size(block);+    free(block);+}++static void *counting_aligned_malloc(size_t alignment, size_t size) {+    void *block = NULL;+    if (posix_memalign(&block, alignment, size) != 0) return NULL;+    allocated += usable_size(block);+    return block;+}++void croaring_ffi_count_allocations(void) {+    roaring_memory_t hooks = {+        counting_malloc, counting_realloc, counting_calloc,+        counting_free, counting_aligned_malloc, counting_free,+    };+    roaring_init_memory_hook(hooks);+}++size_t croaring_ffi_allocated_bytes(void) { return allocated; }
+ cbits/CRoaring/LICENSE view
@@ -0,0 +1,236 @@+The CRoaring project is under a dual license (Apache/MIT).+Users of the library may choose one or the other license.++------------------++                                 Apache License+                           Version 2.0, January 2004+                        http://www.apache.org/licenses/++   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++   1. Definitions.++      "License" shall mean the terms and conditions for use, reproduction,+      and distribution as defined by Sections 1 through 9 of this document.++      "Licensor" shall mean the copyright owner or entity authorized by+      the copyright owner that is granting the License.++      "Legal Entity" shall mean the union of the acting entity and all+      other entities that control, are controlled by, or are under common+      control with that entity. For the purposes of this definition,+      "control" means (i) the power, direct or indirect, to cause the+      direction or management of such entity, whether by contract or+      otherwise, or (ii) ownership of fifty percent (50%) or more of the+      outstanding shares, or (iii) beneficial ownership of such entity.++      "You" (or "Your") shall mean an individual or Legal Entity+      exercising permissions granted by this License.++      "Source" form shall mean the preferred form for making modifications,+      including but not limited to software source code, documentation+      source, and configuration files.++      "Object" form shall mean any form resulting from mechanical+      transformation or translation of a Source form, including but+      not limited to compiled object code, generated documentation,+      and conversions to other media types.++      "Work" shall mean the work of authorship, whether in Source or+      Object form, made available under the License, as indicated by a+      copyright notice that is included in or attached to the work+      (an example is provided in the Appendix below).++      "Derivative Works" shall mean any work, whether in Source or Object+      form, that is based on (or derived from) the Work and for which the+      editorial revisions, annotations, elaborations, or other modifications+      represent, as a whole, an original work of authorship. For the purposes+      of this License, Derivative Works shall not include works that remain+      separable from, or merely link (or bind by name) to the interfaces of,+      the Work and Derivative Works thereof.++      "Contribution" shall mean any work of authorship, including+      the original version of the Work and any modifications or additions+      to that Work or Derivative Works thereof, that is intentionally+      submitted to Licensor for inclusion in the Work by the copyright owner+      or by an individual or Legal Entity authorized to submit on behalf of+      the copyright owner. For the purposes of this definition, "submitted"+      means any form of electronic, verbal, or written communication sent+      to the Licensor or its representatives, including but not limited to+      communication on electronic mailing lists, source code control systems,+      and issue tracking systems that are managed by, or on behalf of, the+      Licensor for the purpose of discussing and improving the Work, but+      excluding communication that is conspicuously marked or otherwise+      designated in writing by the copyright owner as "Not a Contribution."++      "Contributor" shall mean Licensor and any individual or Legal Entity+      on behalf of whom a Contribution has been received by Licensor and+      subsequently incorporated within the Work.++   2. Grant of Copyright License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      copyright license to reproduce, prepare Derivative Works of,+      publicly display, publicly perform, sublicense, and distribute the+      Work and such Derivative Works in Source or Object form.++   3. Grant of Patent License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      (except as stated in this section) patent license to make, have made,+      use, offer to sell, sell, import, and otherwise transfer the Work,+      where such license applies only to those patent claims licensable+      by such Contributor that are necessarily infringed by their+      Contribution(s) alone or by combination of their Contribution(s)+      with the Work to which such Contribution(s) was submitted. If You+      institute patent litigation against any entity (including a+      cross-claim or counterclaim in a lawsuit) alleging that the Work+      or a Contribution incorporated within the Work constitutes direct+      or contributory patent infringement, then any patent licenses+      granted to You under this License for that Work shall terminate+      as of the date such litigation is filed.++   4. Redistribution. You may reproduce and distribute copies of the+      Work or Derivative Works thereof in any medium, with or without+      modifications, and in Source or Object form, provided that You+      meet the following conditions:++      (a) You must give any other recipients of the Work or+          Derivative Works a copy of this License; and++      (b) You must cause any modified files to carry prominent notices+          stating that You changed the files; and++      (c) You must retain, in the Source form of any Derivative Works+          that You distribute, all copyright, patent, trademark, and+          attribution notices from the Source form of the Work,+          excluding those notices that do not pertain to any part of+          the Derivative Works; and++      (d) If the Work includes a "NOTICE" text file as part of its+          distribution, then any Derivative Works that You distribute must+          include a readable copy of the attribution notices contained+          within such NOTICE file, excluding those notices that do not+          pertain to any part of the Derivative Works, in at least one+          of the following places: within a NOTICE text file distributed+          as part of the Derivative Works; within the Source form or+          documentation, if provided along with the Derivative Works; or,+          within a display generated by the Derivative Works, if and+          wherever such third-party notices normally appear. The contents+          of the NOTICE file are for informational purposes only and+          do not modify the License. You may add Your own attribution+          notices within Derivative Works that You distribute, alongside+          or as an addendum to the NOTICE text from the Work, provided+          that such additional attribution notices cannot be construed+          as modifying the License.++      You may add Your own copyright statement to Your modifications and+      may provide additional or different license terms and conditions+      for use, reproduction, or distribution of Your modifications, or+      for any such Derivative Works as a whole, provided Your use,+      reproduction, and distribution of the Work otherwise complies with+      the conditions stated in this License.++   5. Submission of Contributions. Unless You explicitly state otherwise,+      any Contribution intentionally submitted for inclusion in the Work+      by You to the Licensor shall be under the terms and conditions of+      this License, without any additional terms or conditions.+      Notwithstanding the above, nothing herein shall supersede or modify+      the terms of any separate license agreement you may have executed+      with Licensor regarding such Contributions.++   6. Trademarks. This License does not grant permission to use the trade+      names, trademarks, service marks, or product names of the Licensor,+      except as required for reasonable and customary use in describing the+      origin of the Work and reproducing the content of the NOTICE file.++   7. Disclaimer of Warranty. Unless required by applicable law or+      agreed to in writing, Licensor provides the Work (and each+      Contributor provides its Contributions) on an "AS IS" BASIS,+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+      implied, including, without limitation, any warranties or conditions+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+      PARTICULAR PURPOSE. You are solely responsible for determining the+      appropriateness of using or redistributing the Work and assume any+      risks associated with Your exercise of permissions under this License.++   8. Limitation of Liability. In no event and under no legal theory,+      whether in tort (including negligence), contract, or otherwise,+      unless required by applicable law (such as deliberate and grossly+      negligent acts) or agreed to in writing, shall any Contributor be+      liable to You for damages, including any direct, indirect, special,+      incidental, or consequential damages of any character arising as a+      result of this License or out of the use or inability to use the+      Work (including but not limited to damages for loss of goodwill,+      work stoppage, computer failure or malfunction, or any and all+      other commercial damages or losses), even if such Contributor+      has been advised of the possibility of such damages.++   9. Accepting Warranty or Additional Liability. While redistributing+      the Work or Derivative Works thereof, You may choose to offer,+      and charge a fee for, acceptance of support, warranty, indemnity,+      or other liability obligations and/or rights consistent with this+      License. However, in accepting such obligations, You may act only+      on Your own behalf and on Your sole responsibility, not on behalf+      of any other Contributor, and only if You agree to indemnify,+      defend, and hold each Contributor harmless for any liability+      incurred by, or claims asserted against, such Contributor by reason+      of your accepting any such warranty or additional liability.++   END OF TERMS AND CONDITIONS++   APPENDIX: How to apply the Apache License to your work.++      To apply the Apache License to your work, attach the following+      boilerplate notice, with the fields enclosed by brackets "{}"+      replaced with your own identifying information. (Don't include+      the brackets!)  The text should be enclosed in the appropriate+      comment syntax for the file format. We also recommend that a+      file or class name and description of purpose be included on the+      same "printed page" as the copyright notice for easier+      identification within third-party archives.++   Copyright 2016-2022 The CRoaring authors++   Licensed under the Apache License, Version 2.0 (the "License");+   you may not use this file except in compliance with the License.+   You may obtain a copy of the License at++       http://www.apache.org/licenses/LICENSE-2.0++   Unless required by applicable law or agreed to in writing, software+   distributed under the License is distributed on an "AS IS" BASIS,+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+   See the License for the specific language governing permissions and+   limitations under the License.++-----------------------------------++MIT License++Copyright 2016-2022 The CRoaring authors++Permission is hereby granted, free of charge, to any+person obtaining a copy of this software and associated+documentation files (the "Software"), to deal in the+Software without restriction, including without+limitation the rights to use, copy, modify, merge,+publish, distribute, sublicense, and/or sell copies of+the Software, and to permit persons to whom the Software+is furnished to do so, subject to the following+conditions:++The above copyright notice and this permission notice+shall be included in all copies or substantial portions+of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF+ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A+PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT+SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR+IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER+DEALINGS IN THE SOFTWARE.
+ cbits/CRoaring/include/roaring/array_util.h view
@@ -0,0 +1,384 @@+/*+ * array_util.h+ *+ * This header provides low-level utility routines for sorted arrays of+ * 16-bit integers, which are used heavily by CRoaring's array-based+ * containers and set-operation kernels. It includes search helpers, counting+ * helpers, and array intersection/difference primitives.+ *+ * Some of the routines also have SIMD-accelerated implementations on supported+ * platforms, allowing efficient operations on sorted integer arrays that form+ * the basis of sparse container processing.+ */+#ifndef CROARING_ARRAY_UTIL_H+#define CROARING_ARRAY_UTIL_H++#include <stddef.h>  // for size_t+#include <stdint.h>++#include <roaring/portability.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/*+ *  Sorted-array search.+ *  Assumes that array is sorted, has logarithmic complexity.+ *  if the result is x, then:+ *     if ( x>0 )  you have array[x] = ikey+ *     if ( x<0 ) then inserting ikey at position -x-1 in array (insuring that+ * array[-x-1]=ikey) keys the array sorted.+ *+ * Adapted from array_container_contains: a SIMD-quad block-narrowing+ * search at gap=16 (Daniel Lemire,+ * https://lemire.me/blog/2026/04/27/you-can-beat-the-binary-search/)+ * followed by a scalar in-block scan that recovers the exact insertion+ * point required by the binarySearch contract.+ */+CROARING_ALLOW_UNALIGNED+inline int32_t binarySearch(const uint16_t *array, int32_t lenarray,+                            uint16_t ikey) {+    const int32_t gap = 16;+    if (lenarray < gap) {+        for (int32_t j = 0; j < lenarray; j++) {+            if (array[j] >= ikey) {+                return (array[j] == ikey) ? j : -(j + 1);+            }+        }+        return -(lenarray + 1);+    }+    const int32_t num_blocks = lenarray / gap;+    int32_t base = 0;+    int32_t n = num_blocks;+    while (n > 3) {+        int32_t quarter = n >> 2;++        int32_t k1 = array[(base + quarter + 1) * gap - 1];+        int32_t k2 = array[(base + 2 * quarter + 1) * gap - 1];+        int32_t k3 = array[(base + 3 * quarter + 1) * gap - 1];++        int32_t c1 = (k1 < ikey);+        int32_t c2 = (k2 < ikey);+        int32_t c3 = (k3 < ikey);++        base += (c1 + c2 + c3) * quarter;+        n -= 3 * quarter;+    }+    while (n > 1) {+        int32_t half = n >> 1;+        base = (array[(base + half + 1) * gap - 1] < ikey) ? base + half : base;+        n -= half;+    }+    int32_t lo = (array[(base + 1) * gap - 1] < ikey) ? base + 1 : base;++    if (lo < num_blocks) {+        const int32_t start = lo * gap;+#if defined(CROARING_IS_X64)+        // SSE2: subs_epu16 yields zero where lane >= ikey. movemask of an+        // epi16 compare gives 2 bits per lane; ctz>>1 = lane index. Scan+        // the first 8 lanes first and exit early when they contain the+        // answer; otherwise the block-narrowing invariant guarantees the+        // second-half mask is non-zero.+        __m128i needle = _mm_set1_epi16((short)ikey);+        __m128i zero = _mm_setzero_si128();+        __m128i v0 = _mm_loadu_si128((const __m128i *)(array + start));+        __m128i ge0 = _mm_cmpeq_epi16(_mm_subs_epu16(needle, v0), zero);+        unsigned m0 = (unsigned)_mm_movemask_epi8(ge0);+        if (m0 != 0) {+            int32_t j = start + (int32_t)(roaring_trailing_zeroes(m0) >> 1);+            return (array[j] == ikey) ? j : -(j + 1);+        }+        __m128i v1 = _mm_loadu_si128((const __m128i *)(array + start + 8));+        __m128i ge1 = _mm_cmpeq_epi16(_mm_subs_epu16(needle, v1), zero);+        unsigned m1 = (unsigned)_mm_movemask_epi8(ge1);+        int32_t j = start + 8 + (int32_t)(roaring_trailing_zeroes(m1) >> 1);+        return (array[j] == ikey) ? j : -(j + 1);+#else+        const int32_t end = start + gap;+        for (int32_t j = start; j < end; j++) {+            if (array[j] >= ikey) {+                return (array[j] == ikey) ? j : -(j + 1);+            }+        }+        // Unreachable: the narrowing guarantees the last element of the+        // selected block is >= ikey.+        return -(end + 1);+#endif+    }++    for (int32_t j = num_blocks * gap; j < lenarray; j++) {+        if (array[j] >= ikey) {+            return (array[j] == ikey) ? j : -(j + 1);+        }+    }+    return -(lenarray + 1);+}++/**+ * Galloping search+ * Assumes that array is sorted, has logarithmic complexity.+ * if the result is x, then if x = length, you have that all values in array+ * between pos and length are smaller than min. otherwise returns the first+ * index x such that array[x] >= min.+ */+static inline int32_t advanceUntil(const uint16_t *array, int32_t pos,+                                   int32_t length, uint16_t min) {+    int32_t lower = pos + 1;++    if ((lower >= length) || (array[lower] >= min)) {+        return lower;+    }++    int32_t spansize = 1;++    while ((lower + spansize < length) && (array[lower + spansize] < min)) {+        spansize <<= 1;+    }+    int32_t upper = (lower + spansize < length) ? lower + spansize : length - 1;++    if (array[upper] == min) {+        return upper;+    }+    if (array[upper] < min) {+        // means+        // array+        // has no+        // item+        // >= min+        // pos = array.length;+        return length;+    }++    // we know that the next-smallest span was too small+    lower += (spansize >> 1);++    int32_t mid = 0;+    while (lower + 1 != upper) {+        mid = (lower + upper) >> 1;+        if (array[mid] == min) {+            return mid;+        } else if (array[mid] < min) {+            lower = mid;+        } else {+            upper = mid;+        }+    }+    return upper;+}++/**+ * Returns number of elements which are less than ikey.+ * Array elements must be unique and sorted.+ */+static inline int32_t count_less(const uint16_t *array, int32_t lenarray,+                                 uint16_t ikey) {+    if (lenarray == 0) return 0;+    int32_t pos = binarySearch(array, lenarray, ikey);+    return pos >= 0 ? pos : -(pos + 1);+}++/**+ * Returns number of elements which are greater than ikey.+ * Array elements must be unique and sorted.+ */+static inline int32_t count_greater(const uint16_t *array, int32_t lenarray,+                                    uint16_t ikey) {+    if (lenarray == 0) return 0;+    int32_t pos = binarySearch(array, lenarray, ikey);+    if (pos >= 0) {+        return lenarray - (pos + 1);+    } else {+        return lenarray - (-pos - 1);+    }+}++/**+ * From Schlegel et al., Fast Sorted-Set Intersection using SIMD Instructions+ * Optimized by D. Lemire on May 3rd 2013+ *+ * C should have capacity greater than the minimum of s_1 and s_b + 8+ * where 8 is sizeof(__m128i)/sizeof(uint16_t).+ */+int32_t intersect_vector16(const uint16_t *A, size_t s_a, const uint16_t *B,+                           size_t s_b, uint16_t *C);++int32_t intersect_vector16_inplace(uint16_t *A, size_t s_a, const uint16_t *B,+                                   size_t s_b);++/**+ * Take an array container and write it out to a 32-bit array, using base+ * as the offset.+ */+int array_container_to_uint32_array_vector16(void *vout, const uint16_t *array,+                                             size_t cardinality, uint32_t base);+#if CROARING_COMPILER_SUPPORTS_AVX512+int avx512_array_container_to_uint32_array(void *vout, const uint16_t *array,+                                           size_t cardinality, uint32_t base);+#endif+/**+ * Compute the cardinality of the intersection using SSE4 instructions+ */+int32_t intersect_vector16_cardinality(const uint16_t *A, size_t s_a,+                                       const uint16_t *B, size_t s_b);++/* Computes the intersection between one small and one large set of uint16_t.+ * Stores the result into buffer and return the number of elements. */+int32_t intersect_skewed_uint16(const uint16_t *smallarray, size_t size_s,+                                const uint16_t *largearray, size_t size_l,+                                uint16_t *buffer);++/* Computes the size of the intersection between one small and one large set of+ * uint16_t. */+int32_t intersect_skewed_uint16_cardinality(const uint16_t *smallarray,+                                            size_t size_s,+                                            const uint16_t *largearray,+                                            size_t size_l);++/* Check whether the size of the intersection between one small and one large+ * set of uint16_t is non-zero. */+bool intersect_skewed_uint16_nonempty(const uint16_t *smallarray, size_t size_s,+                                      const uint16_t *largearray,+                                      size_t size_l);+/**+ * Generic intersection function.+ */+int32_t intersect_uint16(const uint16_t *A, const size_t lenA,+                         const uint16_t *B, const size_t lenB, uint16_t *out);+/**+ * Compute the size of the intersection (generic).+ */+int32_t intersect_uint16_cardinality(const uint16_t *A, const size_t lenA,+                                     const uint16_t *B, const size_t lenB);++/**+ * Checking whether the size of the intersection  is non-zero.+ */+bool intersect_uint16_nonempty(const uint16_t *A, const size_t lenA,+                               const uint16_t *B, const size_t lenB);++#ifndef CROARING_SKEWED_MERGE_RATIO+#define CROARING_SKEWED_MERGE_RATIO 8+#endif++/**+ * True when an array of size_small values is short enough, next to one of+ * size_large values, that the scalar merges (union_uint16, xor_uint16,+ * difference_uint16 and the end of the intersections) gallop through the long+ * array instead of merging.+ */+static inline bool croaring_skewed_sizes(size_t size_small, size_t size_large) {+    return size_small * CROARING_SKEWED_MERGE_RATIO < size_large;+}++/**+ * The vectorized union and xor run at several values per cycle, so they only+ * lose to the galloping merges when the sizes differ by more.+ */+#ifndef CROARING_SKEWED_VECTOR_MERGE_RATIO+#define CROARING_SKEWED_VECTOR_MERGE_RATIO 32+#endif++/**+ * Generic union function.+ */+size_t union_uint16(const uint16_t *set_1, size_t size_1, const uint16_t *set_2,+                    size_t size_2, uint16_t *buffer);++/**+ * Generic XOR function.+ */+int32_t xor_uint16(const uint16_t *array_1, int32_t card_1,+                   const uint16_t *array_2, int32_t card_2, uint16_t *out);++/**+ * Generic difference function (ANDNOT).+ */+int difference_uint16(const uint16_t *a1, int length1, const uint16_t *a2,+                      int length2, uint16_t *a_out);++/**+ * Generic intersection function.+ */+size_t intersection_uint32(const uint32_t *A, const size_t lenA,+                           const uint32_t *B, const size_t lenB, uint32_t *out);++/**+ * Generic intersection function, returns just the cardinality.+ */+size_t intersection_uint32_card(const uint32_t *A, const size_t lenA,+                                const uint32_t *B, const size_t lenB);++/**+ * Generic union function.+ */+size_t union_uint32(const uint32_t *set_1, size_t size_1, const uint32_t *set_2,+                    size_t size_2, uint32_t *buffer);++/**+ * A fast SSE-based union function.+ */+uint32_t union_vector16(const uint16_t *set_1, uint32_t size_1,+                        const uint16_t *set_2, uint32_t size_2,+                        uint16_t *buffer);++#if CROARING_COMPILER_SUPPORTS_AVX512+/**+ * AVX-512 union of two sorted uint16 arrays, using a Batcher bitonic merge+ * network over 32 lanes. Same contract as union_vector16: `buffer` must have+ * room for size_1 + size_2 values.+ */+uint32_t avx512_union_uint16(const uint16_t *set_1, uint32_t size_1,+                             const uint16_t *set_2, uint32_t size_2,+                             uint16_t *buffer);+#endif  // CROARING_COMPILER_SUPPORTS_AVX512++/**+ * A fast SSE-based XOR function.+ */+uint32_t xor_vector16(const uint16_t *array1, uint32_t length1,+                      const uint16_t *array2, uint32_t length2,+                      uint16_t *output);++/**+ * A fast SSE-based difference function.+ */+int32_t difference_vector16(const uint16_t *A, size_t s_a, const uint16_t *B,+                            size_t s_b, uint16_t *C);++/**+ * Generic union function, returns just the cardinality.+ */+size_t union_uint32_card(const uint32_t *set_1, size_t size_1,+                         const uint32_t *set_2, size_t size_2);++/**+ * combines union_uint16 and  union_vector16 optimally+ */+size_t fast_union_uint16(const uint16_t *set_1, size_t size_1,+                         const uint16_t *set_2, size_t size_2,+                         uint16_t *buffer);++bool memequals(const void *s1, const void *s2, size_t n);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif+#endif
+ cbits/CRoaring/include/roaring/art/art.h view
@@ -0,0 +1,256 @@+#ifndef ART_ART_H+#define ART_ART_H++#include <stdbool.h>+#include <stddef.h>+#include <stdint.h>++/*+ * This file contains an implementation of an Adaptive Radix Tree as described+ * in https://db.in.tum.de/~leis/papers/ART.pdf.+ *+ * The ART contains the keys in _byte lexographical_ order.+ *+ * Other features:+ *  * Fixed 48 bit key length: all keys are assumed to be be 48 bits in size.+ *    This allows us to put the key and key prefixes directly in nodes, reducing+ *    indirection at no additional memory overhead.+ *  * Key compression: the only inner nodes created are at points where key+ *    chunks _differ_. This means that if there are two entries with different+ *    high 48 bits, then there is only one inner node containing the common key+ *    prefix, and two leaves.+ *  * Mostly pointer-free: nodes are referred to by index rather than pointer,+ *    so that the structure can be deserialized with a backing buffer.+ */++// Fixed length of keys in the ART. All keys are assumed to be of this length.+#define ART_KEY_BYTES 6++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++typedef uint8_t art_key_chunk_t;++// Internal node reference type. Contains the node typecode in the low 8 bits,+// and the index in the relevant node array in the high 48 bits. Has a value of+// CROARING_ART_NULL_REF when pointing to a non-existent node.+typedef uint64_t art_ref_t;++typedef void art_node_t;++/**+ * The ART is empty when root is a null ref.+ *+ * Each node type has its own dynamic array of node structs, indexed by+ * art_ref_t. The arrays are expanded as needed, and shrink only when+ * `shrink_to_fit` is called.+ */+typedef struct art_s {+    art_ref_t root;++    // Indexed by node typecode, thus 1 larger than they need to be for+    // convenience. `first_free` indicates the index where the first free node+    // lives, which may be equal to the capacity.+    uint64_t first_free[6];+    uint64_t capacities[6];+    art_node_t *nodes[6];+} art_t;++typedef uint64_t art_val_t;++/**+ * Compares two keys, returns their relative order:+ *  * Key 1 <  key 2: returns a negative value+ *  * Key 1 == key 2: returns 0+ *  * Key 1 >  key 2: returns a positive value+ */+int art_compare_keys(const art_key_chunk_t key1[],+                     const art_key_chunk_t key2[]);++/**+ * Initializes the ART.+ */+void art_init_cleared(art_t *art);++/**+ * Inserts the given key and value. Returns a pointer to the value inserted,+ * valid as long as the ART is not modified.+ */+art_val_t *art_insert(art_t *art, const art_key_chunk_t *key, art_val_t val);++/**+ * Returns true if a value was erased. Sets `*erased_val` to the value erased,+ * if any.+ */+bool art_erase(art_t *art, const art_key_chunk_t *key, art_val_t *erased_val);++/**+ * Returns the value associated with the given key, NULL if not found.+ */+art_val_t *art_find(const art_t *art, const art_key_chunk_t *key);++/**+ * Returns true if the ART is empty.+ */+bool art_is_empty(const art_t *art);++/**+ * Frees the contents of the ART. Should not be called when using+ * `art_deserialize_frozen_safe`.+ */+void art_free(art_t *art);++/**+ * Prints the ART using printf, useful for debugging.+ */+void art_printf(const art_t *art);++/**+ * Callback for validating the value stored in a leaf. `context` is a+ * user-provided value passed to the callback without modification.+ *+ * Should return true if the value is valid, false otherwise+ * If false is returned, `*reason` should be set to a static string describing+ * the reason for the failure.+ */+typedef bool (*art_validate_cb_t)(const art_val_t val, const char **reason,+                                  void *context);++/**+ * Validate the ART tree, ensuring it is internally consistent. `context` is a+ * user-provided value passed to the callback without modification.+ */+bool art_internal_validate(const art_t *art, const char **reason,+                           art_validate_cb_t validate_cb, void *context);++/**+ * ART-internal iterator bookkeeping. Users should treat this as an opaque type.+ */+typedef struct art_iterator_frame_s {+    art_ref_t ref;+    uint8_t index_in_node;+} art_iterator_frame_t;++/**+ * Users should only access `key` and `value` in iterators. The iterator is+ * valid when `value != NULL`.+ */+typedef struct art_iterator_s {+    art_key_chunk_t key[ART_KEY_BYTES];+    art_val_t *value;++    art_t *art;++    uint8_t depth;  // Key depth+    uint8_t frame;  // Node depth++    // State for each node in the ART the iterator has travelled from the root.+    // This is `ART_KEY_BYTES + 1` because it includes state for the leaf too.+    art_iterator_frame_t frames[ART_KEY_BYTES + 1];+} art_iterator_t;++/**+ * Creates an iterator initialzed to the first or last entry in the ART,+ * depending on `first`. The iterator is not valid if there are no entries in+ * the ART.+ */+art_iterator_t art_init_iterator(art_t *art, bool first);++/**+ * Returns an initialized iterator positioned at a key equal to or greater than+ * the given key, if it exists.+ */+art_iterator_t art_lower_bound(art_t *art, const art_key_chunk_t *key);++/**+ * Returns an initialized iterator positioned at a key greater than the given+ * key, if it exists.+ */+art_iterator_t art_upper_bound(art_t *art, const art_key_chunk_t *key);++/**+ * The following iterator movement functions return true if a new entry was+ * encountered.+ */+bool art_iterator_move(art_iterator_t *iterator, bool forward);+bool art_iterator_next(art_iterator_t *iterator);+bool art_iterator_prev(art_iterator_t *iterator);++/**+ * Moves the iterator forward to a key equal to or greater than the given key.+ */+bool art_iterator_lower_bound(art_iterator_t *iterator,+                              const art_key_chunk_t *key);++/**+ * Insert the value and positions the iterator at the key.+ */+void art_iterator_insert(art_iterator_t *iterator, const art_key_chunk_t *key,+                         art_val_t val);++/**+ * Erase the value pointed at by the iterator. Moves the iterator to the next+ * leaf.+ * Returns true if a value was erased. Sets `*erased_val` to the value erased,+ * if any.+ */+bool art_iterator_erase(art_iterator_t *iterator, art_val_t *erased_val);++/**+ * Ensures the ART has room for at least `num_leaves` leaves in total before+ * further leaf allocations are needed. Only leaf storage is reserved; inner+ * nodes keep growing on demand. Invalidates pointers to values previously+ * returned by `art_insert` and `art_find`.+ */+void art_reserve(art_t *art, uint64_t num_leaves);++/**+ * Returns the number of leaves (key/value pairs) in the ART. This walks the+ * free list of leaves, so it is O(1) for an ART that has had no erasures or+ * that has been shrunken, and O(number of free leaf slots) otherwise.+ */+uint64_t art_num_leaves(const art_t *art);++/**+ * Shrinks the internal arrays in the ART to remove any unused elements. Returns+ * the number of bytes freed.+ */+size_t art_shrink_to_fit(art_t *art);++/**+ * Returns true if the ART has no unused elements.+ */+bool art_is_shrunken(const art_t *art);++/**+ * Returns the serialized size in bytes.+ * Requires `art_shrink_to_fit` to be called first.+ */+size_t art_size_in_bytes(const art_t *art);++/**+ * Serializes the ART and returns the number of bytes written. Returns 0 on+ * error. Requires `art_shrink_to_fit` to be called first.+ */+size_t art_serialize(const art_t *art, char *buf);++/**+ * Deserializes the ART from a serialized buffer, reading up to `maxbytes`+ * bytes. Returns 0 on error. Requires `buf` to be 8 byte aligned.+ *+ * An ART deserialized in this way should only be used in a readonly context.The+ * underlying buffer must not be freed before the ART. `art_free` should not be+ * called on the ART deserialized in this way.+ */+size_t art_frozen_view(const char *buf, size_t maxbytes, art_t *art);++#ifdef __cplusplus+}  // extern "C"+}  // namespace roaring+}  // namespace internal+#endif++#endif
+ cbits/CRoaring/include/roaring/bitset/bitset.h view
@@ -0,0 +1,313 @@+/*+ * bitset.h+ *+ * This bitset is a general-purpose dynamic bitmap storing bits in a contiguous+ * array of 64-bit words. The array field points to the word buffer, arraysize+ * records how many words are currently in use, and capacity records how many+ * words are allocated.+ *+ * Unlike the fixed 16-bit-domain container bitset, this structure can grow to+ * cover an arbitrary number of bit positions. It is useful when callers need a+ * resizable bitmap with efficient bitwise operations, scans, and shifts over a+ * larger or runtime-defined domain.+ */+#ifndef CROARING_CBITSET_BITSET_H+#define CROARING_CBITSET_BITSET_H++// For compatibility with MSVC with the use of `restrict`+#ifdef __cplusplus+#define CROARING_CBITSET_RESTRICT+#elif (__STDC_VERSION__ >= 199901L) || \+    (defined(__GNUC__) && defined(__STDC_VERSION__))+#define CROARING_CBITSET_RESTRICT restrict+#else+#define CROARING_CBITSET_RESTRICT+#endif++#include <stdbool.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace api {+#endif++struct bitset_s {+    uint64_t *CROARING_CBITSET_RESTRICT array;+    /* For simplicity and performance, we prefer to have a size and a capacity+     * that is a multiple of 64 bits. Thus we only track the size and the+     * capacity in terms of 64-bit words allocated */+    size_t arraysize;+    size_t capacity;+};++typedef struct bitset_s bitset_t;++/* Create a new bitset. Return NULL in case of failure. */+bitset_t *bitset_create(void);++/* Create a new bitset able to contain size bits. Return NULL in case of+ * failure. */+bitset_t *bitset_create_with_capacity(size_t size);++/* Free memory. */+void bitset_free(bitset_t *bitset);++/* Set all bits to zero. */+void bitset_clear(bitset_t *bitset);++/* Set all bits to one. */+void bitset_fill(bitset_t *bitset);++/* Create a copy */+bitset_t *bitset_copy(const bitset_t *bitset);++/* For advanced users: Resize the bitset so that it can support newarraysize *+ * 64 bits. Return true in case of success, false for failure. Pad with zeroes+ * new buffer areas if requested. */+bool bitset_resize(bitset_t *bitset, size_t newarraysize, bool padwithzeroes);++/* returns how many bytes of memory the backend buffer uses */+inline size_t bitset_size_in_bytes(const bitset_t *bitset) {+    return bitset->arraysize * sizeof(uint64_t);+}++/* returns how many bits can be accessed */+inline size_t bitset_size_in_bits(const bitset_t *bitset) {+    return bitset->arraysize * 64;+}++/* returns how many words (64-bit) of memory the backend buffer uses */+inline size_t bitset_size_in_words(const bitset_t *bitset) {+    return bitset->arraysize;+}++/* For advanced users: Grow the bitset so that it can support newarraysize * 64+ * bits with padding. Return true in case of success, false for failure. */+bool bitset_grow(bitset_t *bitset, size_t newarraysize);++/* attempts to recover unused memory, return false in case of+ * roaring_reallocation failure */+bool bitset_trim(bitset_t *bitset);++/* shifts all bits by 's' positions so that the bitset representing values+ * 1,2,10 would represent values 1+s, 2+s, 10+s. Attempts to resize the bitset+ * (may silently fail, leaving the bitset unchanged) */+void bitset_shift_left(bitset_t *bitset, size_t s);++/* shifts all bits by 's' positions so that the bitset representing values+ * 1,2,10 would represent values 1-s, 2-s, 10-s, negative values are deleted */+void bitset_shift_right(bitset_t *bitset, size_t s);++/* Set the ith bit. Attempts to resize the bitset if needed (may silently fail)+ */+inline void bitset_set(bitset_t *bitset, size_t i) {+    size_t shiftedi = i / 64;+    if (shiftedi >= bitset->arraysize) {+        if (!bitset_grow(bitset, shiftedi + 1)) {+            return;+        }+    }+    bitset->array[shiftedi] |= ((uint64_t)1) << (i % 64);+}++/* Set the ith bit to the specified value. Attempts to resize the bitset if+ * needed (may silently fail) */+inline void bitset_set_to_value(bitset_t *bitset, size_t i, bool flag) {+    size_t shiftedi = i / 64;+    uint64_t mask = ((uint64_t)1) << (i % 64);+    uint64_t dynmask = ((uint64_t)flag) << (i % 64);+    if (shiftedi >= bitset->arraysize) {+        if (!bitset_grow(bitset, shiftedi + 1)) {+            return;+        }+    }+    uint64_t w = bitset->array[shiftedi];+    w &= ~mask;+    w |= dynmask;+    bitset->array[shiftedi] = w;+}++/* Get the value of the ith bit.  */+inline bool bitset_get(const bitset_t *bitset, size_t i) {+    size_t shiftedi = i / 64;+    if (shiftedi >= bitset->arraysize) {+        return false;+    }+    return (bitset->array[shiftedi] & (((uint64_t)1) << (i % 64))) != 0;+}++/* Count number of bits set.  */+size_t bitset_count(const bitset_t *bitset);++/* Returns true if no bit is set.  */+bool bitset_empty(const bitset_t *bitset);++/* Find the index of the first bit set. Or SIZE_MAX if the bitset is empty.  */+size_t bitset_minimum(const bitset_t *bitset);++/* Find the index of the last bit set. Or zero if the bitset is empty.  */+size_t bitset_maximum(const bitset_t *bitset);++/* compute the union in-place (to b1), returns true if successful, to generate a+ * new bitset first call bitset_copy */+bool bitset_inplace_union(bitset_t *CROARING_CBITSET_RESTRICT b1,+                          const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* report the size of the union (without materializing it) */+size_t bitset_union_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                          const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* compute the intersection in-place (to b1), to generate a new bitset first+ * call bitset_copy */+void bitset_inplace_intersection(bitset_t *CROARING_CBITSET_RESTRICT b1,+                                 const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* report the size of the intersection (without materializing it) */+size_t bitset_intersection_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                                 const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* returns true if the bitsets contain no common elements */+bool bitsets_disjoint(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                      const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* returns true if the bitsets contain any common elements */+bool bitsets_intersect(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                       const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* returns true if b1 contains all of the set bits of b2 */+bool bitset_contains_all(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                         const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* compute the difference in-place (to b1), to generate a new bitset first call+ * bitset_copy */+void bitset_inplace_difference(bitset_t *CROARING_CBITSET_RESTRICT b1,+                               const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* compute the size of the difference */+size_t bitset_difference_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                               const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* compute the symmetric difference in-place (to b1), return true if successful,+ * to generate a new bitset first call bitset_copy */+bool bitset_inplace_symmetric_difference(+    bitset_t *CROARING_CBITSET_RESTRICT b1,+    const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* compute the size of the symmetric difference  */+size_t bitset_symmetric_difference_count(+    const bitset_t *CROARING_CBITSET_RESTRICT b1,+    const bitset_t *CROARING_CBITSET_RESTRICT b2);++/* iterate over the set bits+ like so :+  for(size_t i = 0; bitset_next_set_bit(b,&i) ; i++) {+    //.....+  }+  */+inline bool bitset_next_set_bit(const bitset_t *bitset, size_t *i) {+    size_t x = *i / 64;+    if (x >= bitset->arraysize) {+        return false;+    }+    uint64_t w = bitset->array[x];+    w >>= (*i & 63);+    if (w != 0) {+        *i += roaring_trailing_zeroes(w);+        return true;+    }+    x++;+    while (x < bitset->arraysize) {+        w = bitset->array[x];+        if (w != 0) {+            *i = x * 64 + roaring_trailing_zeroes(w);+            return true;+        }+        x++;+    }+    return false;+}++/* iterate over the set bits+ like so :+   size_t buffer[256];+   size_t howmany = 0;+  for(size_t startfrom = 0; (howmany = bitset_next_set_bits(b,buffer,256,+ &startfrom)) > 0 ; startfrom++) {+    //.....+  }+  */+inline size_t bitset_next_set_bits(const bitset_t *bitset, size_t *buffer,+                                   size_t capacity, size_t *startfrom) {+    if (capacity == 0) return 0;  // sanity check+    size_t x = *startfrom / 64;+    if (x >= bitset->arraysize) {+        return 0;  // nothing more to iterate over+    }+    uint64_t w = bitset->array[x];+    // unset low bits inside the word less than *startfrom+    w &= ~((UINT64_C(1) << (*startfrom & 63)) - 1);+    size_t howmany = 0;+    size_t base = x << 6;+    while (howmany < capacity) {+        while (w != 0) {+            uint64_t t = w & (~w + 1);+            int r = roaring_trailing_zeroes(w);+            buffer[howmany++] = r + base;+            if (howmany == capacity) goto end;+            w ^= t;+        }+        x += 1;+        if (x == bitset->arraysize) {+            break;+        }+        base += 64;+        w = bitset->array[x];+    }+end:+    if (howmany > 0) {+        *startfrom = buffer[howmany - 1];+    }+    return howmany;+}++typedef bool (*bitset_iterator)(size_t value, void *param);++// return true if uninterrupted+inline bool bitset_for_each(const bitset_t *b, bitset_iterator iterator,+                            void *ptr) {+    size_t base = 0;+    for (size_t i = 0; i < b->arraysize; ++i) {+        uint64_t w = b->array[i];+        while (w != 0) {+            uint64_t t = w & (~w + 1);+            int r = roaring_trailing_zeroes(w);+            if (!iterator(r + base, ptr)) return false;+            w ^= t;+        }+        base += 64;+    }+    return true;+}++inline void bitset_print(const bitset_t *b) {+    printf("{");+    for (size_t i = 0; bitset_next_set_bit(b, &i); i++) {+        printf("%zu, ", i);+    }+    printf("}");+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace api {+#endif++#endif
+ cbits/CRoaring/include/roaring/bitset_util.h view
@@ -0,0 +1,768 @@+/*+ * bitset_util.h+ *+ * This header collects low-level utility functions for operating on raw+ * bitsets represented as arrays of 64-bit words. It includes helpers for+ * setting, clearing, flipping, counting, extracting, and combining bit ranges,+ * along with architecture-specific SIMD implementations for performance-+ * critical routines.+ *+ * In particular, this file contains accelerated extraction and population-+ * count code paths for x64 targets using AVX2 and, when supported by the+ * compiler and hardware, AVX-512.+ */+#ifndef CROARING_BITSET_UTIL_H+#define CROARING_BITSET_UTIL_H++#include <stdint.h>++#include <roaring/portability.h>+#include <roaring/utilasm.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/*+ * Set all bits in indexes [begin,end) to true.+ */+static inline void bitset_set_range(uint64_t *words, uint32_t start,+                                    uint32_t end) {+    if (start == end) return;+    uint32_t firstword = start / 64;+    uint32_t endword = (end - 1) / 64;+    if (firstword == endword) {+        words[firstword] |= ((~UINT64_C(0)) << (start % 64)) &+                            ((~UINT64_C(0)) >> ((~end + 1) % 64));+        return;+    }+    words[firstword] |= (~UINT64_C(0)) << (start % 64);+    for (uint32_t i = firstword + 1; i < endword; i++) {+        words[i] = ~UINT64_C(0);+    }+    words[endword] |= (~UINT64_C(0)) >> ((~end + 1) % 64);+}++/*+ * Find the cardinality of the bitset in [begin,begin+lenminusone]+ */+static inline int bitset_lenrange_cardinality(const uint64_t *words,+                                              uint32_t start,+                                              uint32_t lenminusone) {+    uint32_t firstword = start / 64;+    uint32_t endword = (start + lenminusone) / 64;+    if (firstword == endword) {+        return roaring_hamming(words[firstword] &+                               ((~UINT64_C(0)) >> ((63 - lenminusone) % 64))+                                   << (start % 64));+    }+    int answer =+        roaring_hamming(words[firstword] & ((~UINT64_C(0)) << (start % 64)));+    for (uint32_t i = firstword + 1; i < endword; i++) {+        answer += roaring_hamming(words[i]);+    }+    answer += roaring_hamming(words[endword] &+                              (~UINT64_C(0)) >>+                                  (((~start + 1) - lenminusone - 1) % 64));+    return answer;+}++/*+ * Check whether the cardinality of the bitset in [begin,begin+lenminusone] is 0+ */+static inline bool bitset_lenrange_empty(const uint64_t *words, uint32_t start,+                                         uint32_t lenminusone) {+    uint32_t firstword = start / 64;+    uint32_t endword = (start + lenminusone) / 64;+    if (firstword == endword) {+        return (words[firstword] & ((~UINT64_C(0)) >> ((63 - lenminusone) % 64))+                                       << (start % 64)) == 0;+    }+    if (((words[firstword] & ((~UINT64_C(0)) << (start % 64)))) != 0) {+        return false;+    }+    for (uint32_t i = firstword + 1; i < endword; i++) {+        if (words[i] != 0) {+            return false;+        }+    }+    if ((words[endword] &+         (~UINT64_C(0)) >> (((~start + 1) - lenminusone - 1) % 64)) != 0) {+        return false;+    }+    return true;+}++/*+ * Set all bits in indexes [begin,begin+lenminusone] to true.+ */+static inline void bitset_set_lenrange(uint64_t *words, uint32_t start,+                                       uint32_t lenminusone) {+    uint32_t firstword = start / 64;+    uint32_t endword = (start + lenminusone) / 64;+    if (firstword == endword) {+        words[firstword] |= ((~UINT64_C(0)) >> ((63 - lenminusone) % 64))+                            << (start % 64);+        return;+    }+    uint64_t temp = words[endword];+    words[firstword] |= (~UINT64_C(0)) << (start % 64);+    for (uint32_t i = firstword + 1; i < endword; i += 2)+        words[i] = words[i + 1] = ~UINT64_C(0);+    words[endword] =+        temp | (~UINT64_C(0)) >> (((~start + 1) - lenminusone - 1) % 64);+}++/*+ * Flip all the bits in indexes [begin,end).+ */+static inline void bitset_flip_range(uint64_t *words, uint32_t start,+                                     uint32_t end) {+    if (start == end) return;+    uint32_t firstword = start / 64;+    uint32_t endword = (end - 1) / 64;+    words[firstword] ^= ~((~UINT64_C(0)) << (start % 64));+    for (uint32_t i = firstword; i < endword; i++) {+        words[i] = ~words[i];+    }+    words[endword] ^= ((~UINT64_C(0)) >> ((~end + 1) % 64));+}++/*+ * Set all bits in indexes [begin,end) to false.+ */+static inline void bitset_reset_range(uint64_t *words, uint32_t start,+                                      uint32_t end) {+    if (start == end) return;+    uint32_t firstword = start / 64;+    uint32_t endword = (end - 1) / 64;+    if (firstword == endword) {+        words[firstword] &= ~(((~UINT64_C(0)) << (start % 64)) &+                              ((~UINT64_C(0)) >> ((~end + 1) % 64)));+        return;+    }+    words[firstword] &= ~((~UINT64_C(0)) << (start % 64));+    for (uint32_t i = firstword + 1; i < endword; i++) {+        words[i] = UINT64_C(0);+    }+    words[endword] &= ~((~UINT64_C(0)) >> ((~end + 1) % 64));+}++/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out", values start at "base".+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ * set.+ *+ * Returns how many values were actually decoded.+ *+ * This function should only be expected to be faster than+ * bitset_extract_setbits+ * when the density of the bitset is high.+ *+ * This function uses AVX2 decoding.+ */+size_t bitset_extract_setbits_avx2(const uint64_t *words, size_t length,+                                   uint32_t *out, size_t outcapacity,+                                   uint32_t base);++/*+ * Same as bitset_extract_setbits_avx2, using AVX-512 (VBMI2) decoding.+ *+ * It writes at most "outcapacity" values and requires no padding beyond them:+ * "out" only needs room for the values that are actually produced.+ *+ * Requires AVX-512 VBMI2 (VPCOMPRESSB): only call it when+ * croaring_hardware_support() reports ROARING_SUPPORTS_AVX512.+ */+size_t bitset_extract_setbits_avx512(const uint64_t *words, size_t length,+                                     uint32_t *out, size_t outcapacity,+                                     uint32_t base);+/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out", values start at "base".+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ *set.+ *+ * Returns how many values were actually decoded.+ */+size_t bitset_extract_setbits(const uint64_t *words, size_t length,+                              uint32_t *out, uint32_t base);++/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out" as 16-bit integers, values start at "base" (can+ *be set to zero)+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ *set.+ *+ * Returns how many values were actually decoded.+ *+ * This function should only be expected to be faster than+ *bitset_extract_setbits_uint16+ * when the density of the bitset is high.+ *+ * This function uses SSE decoding.+ */+size_t bitset_extract_setbits_sse_uint16(const uint64_t *words, size_t length,+                                         uint16_t *out, size_t outcapacity,+                                         uint16_t base);++/*+ * Same as bitset_extract_setbits_avx512, writing 16-bit values.+ *+ * Requires AVX-512 VBMI2 (VPCOMPRESSB): only call it when+ * croaring_hardware_support() reports ROARING_SUPPORTS_AVX512.+ */+size_t bitset_extract_setbits_avx512_uint16(const uint64_t *words,+                                            size_t length, uint16_t *out,+                                            size_t outcapacity, uint16_t base);++/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out",  values start at "base"+ * (can be set to zero)+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ *set.+ *+ * Returns how many values were actually decoded.+ */+size_t bitset_extract_setbits_uint16(const uint64_t *words, size_t length,+                                     uint16_t *out, uint16_t base);++/*+ * Given two bitsets containing "length" 64-bit words, write out the position+ * of all the common set bits to "out", values start at "base"+ * (can be set to zero)+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ * set.+ *+ * Returns how many values were actually decoded.+ */+size_t bitset_extract_intersection_setbits_uint16(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, uint16_t base);++/*+ * Same as bitset_extract_intersection_setbits_uint16, but the caller passes+ * the number of common set bits ("card", e.g., from a "justcard" function)+ * and "out" must hold exactly that many values. Knowing the cardinality lets+ * the function avoid a data-dependent branch per word, which is mispredicted+ * about once per value on sparse intersections.+ */+size_t bitset_extract_intersection_setbits_uint16_with_card(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, size_t card, uint16_t base);++/*+ * Same as bitset_extract_intersection_setbits_uint16_with_card, using AVX-512+ * VBMI2 (VPCOMPRESSB) and no data-dependent branch per word.+ *+ * Only call it when croaring_hardware_support() reports+ * ROARING_SUPPORTS_AVX512.+ */+size_t bitset_extract_intersection_setbits_avx512_uint16(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, size_t card, uint16_t base);++/*+ * Given a bitset having cardinality card, set all bit values in the list (there+ * are length of them)+ * and return the updated cardinality. This evidently assumes that the bitset+ * already contained data.+ */+uint64_t bitset_set_list_withcard(uint64_t *words, uint64_t card,+                                  const uint16_t *list, uint64_t length);+/*+ * Given a bitset, set all bit values in the list (there+ * are length of them).+ */+void bitset_set_list(uint64_t *words, const uint16_t *list, uint64_t length);++/*+ * Given a bitset having cardinality card, unset all bit values in the list+ * (there are length of them)+ * and return the updated cardinality. This evidently assumes that the bitset+ * already contained data.+ */+uint64_t bitset_clear_list(uint64_t *words, uint64_t card, const uint16_t *list,+                           uint64_t length);++/*+ * Given a bitset having cardinality card, toggle all bit values in the list+ * (there are length of them)+ * and return the updated cardinality. This evidently assumes that the bitset+ * already contained data.+ */++uint64_t bitset_flip_list_withcard(uint64_t *words, uint64_t card,+                                   const uint16_t *list, uint64_t length);++void bitset_flip_list(uint64_t *words, const uint16_t *list, uint64_t length);++#if CROARING_IS_X64+/***+ * BEGIN Harley-Seal popcount functions.+ */+CROARING_TARGET_AVX2+/**+ * Compute the population count of a 256-bit word+ * This is not especially fast, but it is convenient as part of other functions.+ */+static inline __m256i popcount256(__m256i v) {+    const __m256i lookuppos = _mm256_setr_epi8(+        /* 0 */ 4 + 0, /* 1 */ 4 + 1, /* 2 */ 4 + 1, /* 3 */ 4 + 2,+        /* 4 */ 4 + 1, /* 5 */ 4 + 2, /* 6 */ 4 + 2, /* 7 */ 4 + 3,+        /* 8 */ 4 + 1, /* 9 */ 4 + 2, /* a */ 4 + 2, /* b */ 4 + 3,+        /* c */ 4 + 2, /* d */ 4 + 3, /* e */ 4 + 3, /* f */ 4 + 4,++        /* 0 */ 4 + 0, /* 1 */ 4 + 1, /* 2 */ 4 + 1, /* 3 */ 4 + 2,+        /* 4 */ 4 + 1, /* 5 */ 4 + 2, /* 6 */ 4 + 2, /* 7 */ 4 + 3,+        /* 8 */ 4 + 1, /* 9 */ 4 + 2, /* a */ 4 + 2, /* b */ 4 + 3,+        /* c */ 4 + 2, /* d */ 4 + 3, /* e */ 4 + 3, /* f */ 4 + 4);+    const __m256i lookupneg = _mm256_setr_epi8(+        /* 0 */ 4 - 0, /* 1 */ 4 - 1, /* 2 */ 4 - 1, /* 3 */ 4 - 2,+        /* 4 */ 4 - 1, /* 5 */ 4 - 2, /* 6 */ 4 - 2, /* 7 */ 4 - 3,+        /* 8 */ 4 - 1, /* 9 */ 4 - 2, /* a */ 4 - 2, /* b */ 4 - 3,+        /* c */ 4 - 2, /* d */ 4 - 3, /* e */ 4 - 3, /* f */ 4 - 4,++        /* 0 */ 4 - 0, /* 1 */ 4 - 1, /* 2 */ 4 - 1, /* 3 */ 4 - 2,+        /* 4 */ 4 - 1, /* 5 */ 4 - 2, /* 6 */ 4 - 2, /* 7 */ 4 - 3,+        /* 8 */ 4 - 1, /* 9 */ 4 - 2, /* a */ 4 - 2, /* b */ 4 - 3,+        /* c */ 4 - 2, /* d */ 4 - 3, /* e */ 4 - 3, /* f */ 4 - 4);+    const __m256i low_mask = _mm256_set1_epi8(0x0f);++    const __m256i lo = _mm256_and_si256(v, low_mask);+    const __m256i hi = _mm256_and_si256(_mm256_srli_epi16(v, 4), low_mask);+    const __m256i popcnt1 = _mm256_shuffle_epi8(lookuppos, lo);+    const __m256i popcnt2 = _mm256_shuffle_epi8(lookupneg, hi);+    return _mm256_sad_epu8(popcnt1, popcnt2);+}+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+/**+ * Simple CSA over 256 bits+ */+static inline void CSA(__m256i *h, __m256i *l, __m256i a, __m256i b,+                       __m256i c) {+    const __m256i u = _mm256_xor_si256(a, b);+    *h = _mm256_or_si256(_mm256_and_si256(a, b), _mm256_and_si256(u, c));+    *l = _mm256_xor_si256(u, c);+}+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+/**+ * Fast Harley-Seal AVX population count function+ */+inline static uint64_t avx2_harley_seal_popcount256(const __m256i *data,+                                                    const uint64_t size) {+    __m256i total = _mm256_setzero_si256();+    __m256i ones = _mm256_setzero_si256();+    __m256i twos = _mm256_setzero_si256();+    __m256i fours = _mm256_setzero_si256();+    __m256i eights = _mm256_setzero_si256();+    __m256i sixteens = _mm256_setzero_si256();+    __m256i twosA, twosB, foursA, foursB, eightsA, eightsB;++    const uint64_t limit = size - size % 16;+    uint64_t i = 0;++    for (; i < limit; i += 16) {+        CSA(&twosA, &ones, ones, _mm256_loadu_si256(data + i),+            _mm256_loadu_si256(data + i + 1));+        CSA(&twosB, &ones, ones, _mm256_loadu_si256(data + i + 2),+            _mm256_loadu_si256(data + i + 3));+        CSA(&foursA, &twos, twos, twosA, twosB);+        CSA(&twosA, &ones, ones, _mm256_loadu_si256(data + i + 4),+            _mm256_loadu_si256(data + i + 5));+        CSA(&twosB, &ones, ones, _mm256_loadu_si256(data + i + 6),+            _mm256_loadu_si256(data + i + 7));+        CSA(&foursB, &twos, twos, twosA, twosB);+        CSA(&eightsA, &fours, fours, foursA, foursB);+        CSA(&twosA, &ones, ones, _mm256_loadu_si256(data + i + 8),+            _mm256_loadu_si256(data + i + 9));+        CSA(&twosB, &ones, ones, _mm256_loadu_si256(data + i + 10),+            _mm256_loadu_si256(data + i + 11));+        CSA(&foursA, &twos, twos, twosA, twosB);+        CSA(&twosA, &ones, ones, _mm256_loadu_si256(data + i + 12),+            _mm256_loadu_si256(data + i + 13));+        CSA(&twosB, &ones, ones, _mm256_loadu_si256(data + i + 14),+            _mm256_loadu_si256(data + i + 15));+        CSA(&foursB, &twos, twos, twosA, twosB);+        CSA(&eightsB, &fours, fours, foursA, foursB);+        CSA(&sixteens, &eights, eights, eightsA, eightsB);++        total = _mm256_add_epi64(total, popcount256(sixteens));+    }++    total = _mm256_slli_epi64(total, 4);  // * 16+    total = _mm256_add_epi64(+        total, _mm256_slli_epi64(popcount256(eights), 3));  // += 8 * ...+    total = _mm256_add_epi64(+        total, _mm256_slli_epi64(popcount256(fours), 2));  // += 4 * ...+    total = _mm256_add_epi64(+        total, _mm256_slli_epi64(popcount256(twos), 1));  // += 2 * ...+    total = _mm256_add_epi64(total, popcount256(ones));+    for (; i < size; i++)+        total =+            _mm256_add_epi64(total, popcount256(_mm256_loadu_si256(data + i)));++    return (uint64_t)(_mm256_extract_epi64(total, 0)) ++           (uint64_t)(_mm256_extract_epi64(total, 1)) ++           (uint64_t)(_mm256_extract_epi64(total, 2)) ++           (uint64_t)(_mm256_extract_epi64(total, 3));+}+CROARING_UNTARGET_AVX2++#define CROARING_AVXPOPCNTFNC(opname, avx_intrinsic)                           \+    static inline uint64_t avx2_harley_seal_popcount256_##opname(              \+        const __m256i *data1, const __m256i *data2, const uint64_t size) {     \+        __m256i total = _mm256_setzero_si256();                                \+        __m256i ones = _mm256_setzero_si256();                                 \+        __m256i twos = _mm256_setzero_si256();                                 \+        __m256i fours = _mm256_setzero_si256();                                \+        __m256i eights = _mm256_setzero_si256();                               \+        __m256i sixteens = _mm256_setzero_si256();                             \+        __m256i twosA, twosB, foursA, foursB, eightsA, eightsB;                \+        __m256i A1, A2;                                                        \+        const uint64_t limit = size - size % 16;                               \+        uint64_t i = 0;                                                        \+        for (; i < limit; i += 16) {                                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i),                  \+                               _mm256_loadu_si256(data2 + i));                 \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 1),              \+                               _mm256_loadu_si256(data2 + i + 1));             \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 2),              \+                               _mm256_loadu_si256(data2 + i + 2));             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 3),              \+                               _mm256_loadu_si256(data2 + i + 3));             \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursA, &twos, twos, twosA, twosB);                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 4),              \+                               _mm256_loadu_si256(data2 + i + 4));             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 5),              \+                               _mm256_loadu_si256(data2 + i + 5));             \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 6),              \+                               _mm256_loadu_si256(data2 + i + 6));             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 7),              \+                               _mm256_loadu_si256(data2 + i + 7));             \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursB, &twos, twos, twosA, twosB);                           \+            CSA(&eightsA, &fours, fours, foursA, foursB);                      \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 8),              \+                               _mm256_loadu_si256(data2 + i + 8));             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 9),              \+                               _mm256_loadu_si256(data2 + i + 9));             \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 10),             \+                               _mm256_loadu_si256(data2 + i + 10));            \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 11),             \+                               _mm256_loadu_si256(data2 + i + 11));            \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursA, &twos, twos, twosA, twosB);                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 12),             \+                               _mm256_loadu_si256(data2 + i + 12));            \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 13),             \+                               _mm256_loadu_si256(data2 + i + 13));            \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 14),             \+                               _mm256_loadu_si256(data2 + i + 14));            \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 15),             \+                               _mm256_loadu_si256(data2 + i + 15));            \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursB, &twos, twos, twosA, twosB);                           \+            CSA(&eightsB, &fours, fours, foursA, foursB);                      \+            CSA(&sixteens, &eights, eights, eightsA, eightsB);                 \+            total = _mm256_add_epi64(total, popcount256(sixteens));            \+        }                                                                      \+        total = _mm256_slli_epi64(total, 4);                                   \+        total = _mm256_add_epi64(total,                                        \+                                 _mm256_slli_epi64(popcount256(eights), 3));   \+        total =                                                                \+            _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(fours), 2)); \+        total =                                                                \+            _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(twos), 1));  \+        total = _mm256_add_epi64(total, popcount256(ones));                    \+        for (; i < size; i++) {                                                \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i),                  \+                               _mm256_loadu_si256(data2 + i));                 \+            total = _mm256_add_epi64(total, popcount256(A1));                  \+        }                                                                      \+        return (uint64_t)(_mm256_extract_epi64(total, 0)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 1)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 2)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 3));                     \+    }                                                                          \+    static inline uint64_t avx2_harley_seal_popcount256andstore_##opname(      \+        const __m256i *__restrict__ data1, const __m256i *__restrict__ data2,  \+        __m256i *__restrict__ out, const uint64_t size) {                      \+        __m256i total = _mm256_setzero_si256();                                \+        __m256i ones = _mm256_setzero_si256();                                 \+        __m256i twos = _mm256_setzero_si256();                                 \+        __m256i fours = _mm256_setzero_si256();                                \+        __m256i eights = _mm256_setzero_si256();                               \+        __m256i sixteens = _mm256_setzero_si256();                             \+        __m256i twosA, twosB, foursA, foursB, eightsA, eightsB;                \+        __m256i A1, A2;                                                        \+        const uint64_t limit = size - size % 16;                               \+        uint64_t i = 0;                                                        \+        for (; i < limit; i += 16) {                                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i),                  \+                               _mm256_loadu_si256(data2 + i));                 \+            _mm256_storeu_si256(out + i, A1);                                  \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 1),              \+                               _mm256_loadu_si256(data2 + i + 1));             \+            _mm256_storeu_si256(out + i + 1, A2);                              \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 2),              \+                               _mm256_loadu_si256(data2 + i + 2));             \+            _mm256_storeu_si256(out + i + 2, A1);                              \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 3),              \+                               _mm256_loadu_si256(data2 + i + 3));             \+            _mm256_storeu_si256(out + i + 3, A2);                              \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursA, &twos, twos, twosA, twosB);                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 4),              \+                               _mm256_loadu_si256(data2 + i + 4));             \+            _mm256_storeu_si256(out + i + 4, A1);                              \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 5),              \+                               _mm256_loadu_si256(data2 + i + 5));             \+            _mm256_storeu_si256(out + i + 5, A2);                              \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 6),              \+                               _mm256_loadu_si256(data2 + i + 6));             \+            _mm256_storeu_si256(out + i + 6, A1);                              \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 7),              \+                               _mm256_loadu_si256(data2 + i + 7));             \+            _mm256_storeu_si256(out + i + 7, A2);                              \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursB, &twos, twos, twosA, twosB);                           \+            CSA(&eightsA, &fours, fours, foursA, foursB);                      \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 8),              \+                               _mm256_loadu_si256(data2 + i + 8));             \+            _mm256_storeu_si256(out + i + 8, A1);                              \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 9),              \+                               _mm256_loadu_si256(data2 + i + 9));             \+            _mm256_storeu_si256(out + i + 9, A2);                              \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 10),             \+                               _mm256_loadu_si256(data2 + i + 10));            \+            _mm256_storeu_si256(out + i + 10, A1);                             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 11),             \+                               _mm256_loadu_si256(data2 + i + 11));            \+            _mm256_storeu_si256(out + i + 11, A2);                             \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursA, &twos, twos, twosA, twosB);                           \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 12),             \+                               _mm256_loadu_si256(data2 + i + 12));            \+            _mm256_storeu_si256(out + i + 12, A1);                             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 13),             \+                               _mm256_loadu_si256(data2 + i + 13));            \+            _mm256_storeu_si256(out + i + 13, A2);                             \+            CSA(&twosA, &ones, ones, A1, A2);                                  \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 14),             \+                               _mm256_loadu_si256(data2 + i + 14));            \+            _mm256_storeu_si256(out + i + 14, A1);                             \+            A2 = avx_intrinsic(_mm256_loadu_si256(data1 + i + 15),             \+                               _mm256_loadu_si256(data2 + i + 15));            \+            _mm256_storeu_si256(out + i + 15, A2);                             \+            CSA(&twosB, &ones, ones, A1, A2);                                  \+            CSA(&foursB, &twos, twos, twosA, twosB);                           \+            CSA(&eightsB, &fours, fours, foursA, foursB);                      \+            CSA(&sixteens, &eights, eights, eightsA, eightsB);                 \+            total = _mm256_add_epi64(total, popcount256(sixteens));            \+        }                                                                      \+        total = _mm256_slli_epi64(total, 4);                                   \+        total = _mm256_add_epi64(total,                                        \+                                 _mm256_slli_epi64(popcount256(eights), 3));   \+        total =                                                                \+            _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(fours), 2)); \+        total =                                                                \+            _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(twos), 1));  \+        total = _mm256_add_epi64(total, popcount256(ones));                    \+        for (; i < size; i++) {                                                \+            A1 = avx_intrinsic(_mm256_loadu_si256(data1 + i),                  \+                               _mm256_loadu_si256(data2 + i));                 \+            _mm256_storeu_si256(out + i, A1);                                  \+            total = _mm256_add_epi64(total, popcount256(A1));                  \+        }                                                                      \+        return (uint64_t)(_mm256_extract_epi64(total, 0)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 1)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 2)) +                    \+               (uint64_t)(_mm256_extract_epi64(total, 3));                     \+    }++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(or, _mm256_or_si256)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(union, _mm256_or_si256)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(and, _mm256_and_si256)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(intersection, _mm256_and_si256)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(xor, _mm256_xor_si256)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVXPOPCNTFNC(andnot, _mm256_andnot_si256)+CROARING_UNTARGET_AVX2++#define VPOPCNT_AND_ADD(ptr, i, accu)                                  \+    const __m512i v##i = _mm512_loadu_si512((const __m512i *)ptr + i); \+    const __m512i p##i = _mm512_popcnt_epi64(v##i);                    \+    accu = _mm512_add_epi64(accu, p##i);++#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+static inline uint64_t sum_epu64_256(const __m256i v) {+    return (uint64_t)(_mm256_extract_epi64(v, 0)) ++           (uint64_t)(_mm256_extract_epi64(v, 1)) ++           (uint64_t)(_mm256_extract_epi64(v, 2)) ++           (uint64_t)(_mm256_extract_epi64(v, 3));+}++static inline uint64_t simd_sum_epu64(const __m512i v) {+    __m256i lo = _mm512_extracti64x4_epi64(v, 0);+    __m256i hi = _mm512_extracti64x4_epi64(v, 1);++    return sum_epu64_256(lo) + sum_epu64_256(hi);+}++static inline uint64_t avx512_vpopcount(const __m512i *data,+                                        const uint64_t size) {+    const uint64_t limit = size - size % 4;+    __m512i total = _mm512_setzero_si512();+    uint64_t i = 0;++    for (; i < limit; i += 4) {+        VPOPCNT_AND_ADD(data + i, 0, total);+        VPOPCNT_AND_ADD(data + i, 1, total);+        VPOPCNT_AND_ADD(data + i, 2, total);+        VPOPCNT_AND_ADD(data + i, 3, total);+    }++    for (; i < size; i++) {+        total = _mm512_add_epi64(+            total, _mm512_popcnt_epi64(_mm512_loadu_si512(data + i)));+    }++    return simd_sum_epu64(total);+}+CROARING_UNTARGET_AVX512+#endif++#define CROARING_AVXPOPCNTFNC512(opname, avx_intrinsic)                       \+    static inline uint64_t avx512_harley_seal_popcount512_##opname(           \+        const __m512i *data1, const __m512i *data2, const uint64_t size) {    \+        __m512i total = _mm512_setzero_si512();                               \+        const uint64_t limit = size - size % 4;                               \+        uint64_t i = 0;                                                       \+        for (; i < limit; i += 4) {                                           \+            __m512i a1 = avx_intrinsic(_mm512_loadu_si512(data1 + i),         \+                                       _mm512_loadu_si512(data2 + i));        \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a1));         \+            __m512i a2 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 1),     \+                                       _mm512_loadu_si512(data2 + i + 1));    \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a2));         \+            __m512i a3 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 2),     \+                                       _mm512_loadu_si512(data2 + i + 2));    \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a3));         \+            __m512i a4 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 3),     \+                                       _mm512_loadu_si512(data2 + i + 3));    \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a4));         \+        }                                                                     \+        for (; i < size; i++) {                                               \+            __m512i a = avx_intrinsic(_mm512_loadu_si512(data1 + i),          \+                                      _mm512_loadu_si512(data2 + i));         \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a));          \+        }                                                                     \+        return simd_sum_epu64(total);                                         \+    }                                                                         \+    static inline uint64_t avx512_harley_seal_popcount512andstore_##opname(   \+        const __m512i *__restrict__ data1, const __m512i *__restrict__ data2, \+        __m512i *__restrict__ out, const uint64_t size) {                     \+        __m512i total = _mm512_setzero_si512();                               \+        const uint64_t limit = size - size % 4;                               \+        uint64_t i = 0;                                                       \+        for (; i < limit; i += 4) {                                           \+            __m512i a1 = avx_intrinsic(_mm512_loadu_si512(data1 + i),         \+                                       _mm512_loadu_si512(data2 + i));        \+            _mm512_storeu_si512(out + i, a1);                                 \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a1));         \+            __m512i a2 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 1),     \+                                       _mm512_loadu_si512(data2 + i + 1));    \+            _mm512_storeu_si512(out + i + 1, a2);                             \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a2));         \+            __m512i a3 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 2),     \+                                       _mm512_loadu_si512(data2 + i + 2));    \+            _mm512_storeu_si512(out + i + 2, a3);                             \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a3));         \+            __m512i a4 = avx_intrinsic(_mm512_loadu_si512(data1 + i + 3),     \+                                       _mm512_loadu_si512(data2 + i + 3));    \+            _mm512_storeu_si512(out + i + 3, a4);                             \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a4));         \+        }                                                                     \+        for (; i < size; i++) {                                               \+            __m512i a = avx_intrinsic(_mm512_loadu_si512(data1 + i),          \+                                      _mm512_loadu_si512(data2 + i));         \+            _mm512_storeu_si512(out + i, a);                                  \+            total = _mm512_add_epi64(total, _mm512_popcnt_epi64(a));          \+        }                                                                     \+        return simd_sum_epu64(total);                                         \+    }++#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+CROARING_AVXPOPCNTFNC512(or, _mm512_or_si512)+CROARING_AVXPOPCNTFNC512(union, _mm512_or_si512)+CROARING_AVXPOPCNTFNC512(and, _mm512_and_si512)+CROARING_AVXPOPCNTFNC512(intersection, _mm512_and_si512)+CROARING_AVXPOPCNTFNC512(xor, _mm512_xor_si512)+CROARING_AVXPOPCNTFNC512(andnot, _mm512_andnot_si512)+CROARING_UNTARGET_AVX512+#endif+/***+ * END Harley-Seal popcount functions.+ */++#endif  // CROARING_IS_X64++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif+#endif
+ cbits/CRoaring/include/roaring/containers/array.h view
@@ -0,0 +1,581 @@+/*+ * array.h+ *+ * Array containers store a sparse set of 16-bit integers as a sorted dynamic+ * array. The cardinality field tracks how many values are present, capacity+ * tracks the allocated length, and array points to the sorted values.+ *+ * This representation is used for low-cardinality containers because it is+ * compact and supports fast iteration and binary-search-based membership+ * tests. When the number of stored values grows beyond DEFAULT_MAX_SIZE,+ * CRoaring will typically switch to a denser container representation.+ */++#ifndef INCLUDE_CONTAINERS_ARRAY_H_+#define INCLUDE_CONTAINERS_ARRAY_H_++#include <string.h>++#include <roaring/roaring_types.h>  // roaring_iterator++// Include other headers after roaring_types.h+#include <roaring/array_util.h>  // binarySearch()/memequals() for inlining+#include <roaring/containers/container_defs.h>  // container_t, perfparameters+#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {++// Note: in pure C++ code, you should avoid putting `using` in header files+using api::roaring_iterator;+using api::roaring_iterator64;++namespace internal {+#endif++/* Containers with DEFAULT_MAX_SIZE or less integers should be arrays */+enum { DEFAULT_MAX_SIZE = 4096 };++/* struct array_container - sparse representation of a bitmap+ *+ * @cardinality: number of indices in `array` (and the bitmap)+ * @capacity:    allocated size of `array`+ * @array:       sorted list of integers+ */+STRUCT_CONTAINER(array_container_s) {+    int32_t cardinality;+    int32_t capacity;+    uint16_t *array;+};++typedef struct array_container_s array_container_t;++#define CAST_array(c) CAST(array_container_t *, c)  // safer downcast+#define const_CAST_array(c) CAST(const array_container_t *, c)+#define movable_CAST_array(c) movable_CAST(array_container_t **, c)++/* Create a new array with default. Return NULL in case of failure. See also+ * array_container_create_given_capacity. */+array_container_t *array_container_create(void);++/* Create a new array with a specified capacity size. Return NULL in case of+ * failure. */+array_container_t *array_container_create_given_capacity(int32_t size);++/* Create a new array containing all values in [min,max). */+array_container_t *array_container_create_range(uint32_t min, uint32_t max);++/*+ * Shrink the capacity to the actual size, return the number of bytes saved.+ */+int array_container_shrink_to_fit(array_container_t *src);++/* Free memory owned by `array'. */+void array_container_free(array_container_t *array);++/* Duplicate container */+array_container_t *array_container_clone(const array_container_t *src);++/* Get the cardinality of `array'. */+CROARING_ALLOW_UNALIGNED+static inline int array_container_cardinality(const array_container_t *array) {+    return array->cardinality;+}++static inline bool array_container_nonzero_cardinality(+    const array_container_t *array) {+    return array->cardinality > 0;+}++/* Copy one container into another. We assume that they are distinct. */+void array_container_copy(const array_container_t *src, array_container_t *dst);++/*  Add all the values in [min,max) (included) at a distance k*step from min.+    The container must have a size less or equal to DEFAULT_MAX_SIZE after this+   addition. */+void array_container_add_from_range(array_container_t *arr, uint32_t min,+                                    uint32_t max, uint16_t step);++static inline bool array_container_empty(const array_container_t *array) {+    return array->cardinality == 0;+}++/* check whether the cardinality is equal to the capacity (this does not mean+ * that it contains 1<<16 elements) */+static inline bool array_container_full(const array_container_t *array) {+    return array->cardinality == array->capacity;+}++/* Compute the union of `src_1' and `src_2' and write the result to `dst'+ * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */+void array_container_union(const array_container_t *src_1,+                           const array_container_t *src_2,+                           array_container_t *dst);++/* symmetric difference, see array_container_union */+void array_container_xor(const array_container_t *array_1,+                         const array_container_t *array_2,+                         array_container_t *out);++/* Computes the intersection of src_1 and src_2 and write the result to+ * dst. It is assumed that dst is distinct from both src_1 and src_2. */+void array_container_intersection(const array_container_t *src_1,+                                  const array_container_t *src_2,+                                  array_container_t *dst);++/* Check whether src_1 and src_2 intersect. */+bool array_container_intersect(const array_container_t *src_1,+                               const array_container_t *src_2);++/* computers the size of the intersection between two arrays.+ */+int array_container_intersection_cardinality(const array_container_t *src_1,+                                             const array_container_t *src_2);++/* computes the intersection of array1 and array2 and write the result to+ * array1.+ * */+void array_container_intersection_inplace(array_container_t *src_1,+                                          const array_container_t *src_2);++/*+ * Write out the 16-bit integers contained in this container as a list of 32-bit+ * integers using base+ * as the starting value (it might be expected that base has zeros in its 16+ * least significant bits).+ * The function returns the number of values written.+ * The caller is responsible for allocating enough memory in out.+ */+int array_container_to_uint32_array(void *vout, const array_container_t *cont,+                                    uint32_t base);++/* Compute the number of runs */+int32_t array_container_number_of_runs(const array_container_t *ac);++/*+ * Print this container using printf (useful for debugging).+ */+void array_container_printf(const array_container_t *v);++/*+ * Print this container using printf as a comma-separated list of 32-bit+ * integers starting at base.+ */+void array_container_printf_as_uint32_array(const array_container_t *v,+                                            uint32_t base);++bool array_container_validate(const array_container_t *v, const char **reason);++/**+ * Return the serialized size in bytes of a container having cardinality "card".+ */+static inline int32_t array_container_serialized_size_in_bytes(int32_t card) {+    return card * sizeof(uint16_t);+}++/**+ * Increase capacity to at least min.+ * Whether the existing data needs to be copied over depends on the "preserve"+ * parameter. If preserve is false, then the new content will be uninitialized,+ * otherwise the old content is copied.+ */+void array_container_grow(array_container_t *container, int32_t min,+                          bool preserve);++bool array_container_iterate(const array_container_t *cont, uint32_t base,+                             roaring_iterator iterator, void *ptr);+bool array_container_iterate64(const array_container_t *cont, uint32_t base,+                               roaring_iterator64 iterator, uint64_t high_bits,+                               void *ptr);++/**+ * Writes the underlying array to buf, outputs how many bytes were written.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes written should be+ * array_container_size_in_bytes(container).+ *+ */+int32_t array_container_write(const array_container_t *container, char *buf);+/**+ * Reads the instance from buf, outputs how many bytes were read.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes read should be array_container_size_in_bytes(container).+ * You need to provide the (known) cardinality.+ */+int32_t array_container_read(int32_t cardinality, array_container_t *container,+                             const char *buf);++/**+ * Return the serialized size in bytes of a container (see+ * bitset_container_write)+ * This is meant to be compatible with the Java and Go versions of Roaring and+ * assumes+ * that the cardinality of the container is already known.+ *+ */+CROARING_ALLOW_UNALIGNED+static inline int32_t array_container_size_in_bytes(+    const array_container_t *container) {+    return container->cardinality * sizeof(uint16_t);+}++/**+ * Return true if the two arrays have the same content.+ */+CROARING_ALLOW_UNALIGNED+static inline bool array_container_equals(const array_container_t *container1,+                                          const array_container_t *container2) {+    if (container1->cardinality != container2->cardinality) {+        return false;+    }+    return memequals(container1->array, container2->array,+                     container1->cardinality * 2);+}++/**+ * Return true if container1 is a subset of container2.+ */+bool array_container_is_subset(const array_container_t *container1,+                               const array_container_t *container2);++/**+ * If the element of given rank is in this container, supposing that the first+ * element has rank start_rank, then the function returns true and sets element+ * accordingly.+ * Otherwise, it returns false and update start_rank.+ */+CROARING_ALLOW_UNALIGNED+static inline bool array_container_select(const array_container_t *container,+                                          uint32_t *start_rank, uint32_t rank,+                                          uint32_t *element) {+    int card = array_container_cardinality(container);+    if (*start_rank + card <= rank) {+        *start_rank += card;+        return false;+    } else {+        *element = container->array[rank - *start_rank];+        return true;+    }+}++/* Computes the  difference of array1 and array2 and write the result+ * to array out.+ * Array out does not need to be distinct from array_1+ */+void array_container_andnot(const array_container_t *array_1,+                            const array_container_t *array_2,+                            array_container_t *out);++/* Append x to the set. Assumes that the value is larger than any preceding+ * values.  */+static inline void array_container_append(array_container_t *arr,+                                          uint16_t pos) {+    const int32_t capacity = arr->capacity;++    if (array_container_full(arr)) {+        array_container_grow(arr, capacity + 1, true);+    }++    arr->array[arr->cardinality++] = pos;+}++/**+ * Add value to the set if final cardinality doesn't exceed max_cardinality.+ * Return code:+ * 1  -- value was added+ * 0  -- value was already present+ * -1 -- value was not added because cardinality would exceed max_cardinality+ */+static inline int array_container_try_add(array_container_t *arr,+                                          uint16_t value,+                                          int32_t max_cardinality) {+    const int32_t cardinality = arr->cardinality;++    // best case, we can append.+    if ((array_container_empty(arr) || arr->array[cardinality - 1] < value) &&+        cardinality < max_cardinality) {+        array_container_append(arr, value);+        return 1;+    }++    const int32_t loc = binarySearch(arr->array, cardinality, value);++    if (loc >= 0) {+        return 0;+    } else if (cardinality < max_cardinality) {+        if (array_container_full(arr)) {+            array_container_grow(arr, arr->capacity + 1, true);+        }+        const int32_t insert_idx = -loc - 1;+        memmove(arr->array + insert_idx + 1, arr->array + insert_idx,+                (cardinality - insert_idx) * sizeof(uint16_t));+        arr->array[insert_idx] = value;+        arr->cardinality++;+        return 1;+    } else {+        return -1;+    }+}++/* Add value to the set. Returns true if x was not already present.  */+static inline bool array_container_add(array_container_t *arr, uint16_t value) {+    return array_container_try_add(arr, value, INT32_MAX) == 1;+}++/* Remove x from the set. Returns true if x was present.  */+static inline bool array_container_remove(array_container_t *arr,+                                          uint16_t pos) {+    const int32_t idx = binarySearch(arr->array, arr->cardinality, pos);+    const bool is_present = idx >= 0;+    if (is_present) {+        memmove(arr->array + idx, arr->array + idx + 1,+                (arr->cardinality - idx - 1) * sizeof(uint16_t));+        arr->cardinality--;+    }++    return is_present;+}++/* Check whether x is present.  */+CROARING_ALLOW_UNALIGNED+inline bool array_container_contains(const array_container_t *arr,+                                     uint16_t pos) {+    /**+     * SIMD Quad algorithm+     * Daniel Lemire, "You can beat the binary search," in Daniel Lemire's blog,+     *  April 27, 2026,+     * https://lemire.me/blog/2026/04/27/you-can-beat-the-binary-search/.+     */+    const int32_t gap = 16;+    const uint16_t *carr = arr->array;+    int32_t cardinality = arr->cardinality;+    if (cardinality < gap) {+        for (int32_t j = 0; j < cardinality; j++) {+            if (carr[j] >= pos) return carr[j] == pos;+        }+        return false;+    }+    int32_t num_blocks = cardinality / gap;+    int32_t base = 0;+    int32_t n = num_blocks;+    while (n > 3) {+        int32_t quarter = n >> 2;++        int32_t k1 = carr[(base + quarter + 1) * gap - 1];+        int32_t k2 = carr[(base + 2 * quarter + 1) * gap - 1];+        int32_t k3 = carr[(base + 3 * quarter + 1) * gap - 1];++        int32_t c1 = (k1 < pos);+        int32_t c2 = (k2 < pos);+        int32_t c3 = (k3 < pos);++        base += (c1 + c2 + c3) * quarter;+        n -= 3 * quarter;+    }+    while (n > 1) {+        int32_t half = n >> 1;+        base = (carr[(base + half + 1) * gap - 1] < pos) ? base + half : base;+        n -= half;+    }+    int32_t lo = (carr[(base + 1) * gap - 1] < pos) ? base + 1 : base;++    if (lo < num_blocks) {+        const uint16_t *blk = carr + lo * gap;+#ifdef CROARING_USENEON+        uint16x8_t needle = vdupq_n_u16(pos);+        uint16x8_t v0 = vld1q_u16(blk);+        uint16x8_t v1 = vld1q_u16(blk + 8);+        uint16x8_t hit =+            vorrq_u16(vceqq_u16(v0, needle), vceqq_u16(v1, needle));+#ifdef __arm__+        uint16x4_t hit8 = vmax_u16(vget_low_u16(hit), vget_high_u16(hit));+        uint16x4_t hit4 = vpmax_u16(hit8, hit8);+        return vget_lane_u32(hit4, 0) != 0;+#else /* __aarch64__ */+        return vmaxvq_u16(hit) != 0;+#endif+#elif defined(CROARING_IS_X64)+        __m128i needle = _mm_set1_epi16((short)pos);+        __m128i v0 = _mm_loadu_si128((const __m128i *)blk);+        __m128i v1 = _mm_loadu_si128((const __m128i *)(blk + 8));+        __m128i hit = _mm_or_si128(_mm_cmpeq_epi16(v0, needle),+                                   _mm_cmpeq_epi16(v1, needle));+        return _mm_movemask_epi8(hit) != 0;+#else+        for (int32_t j = 0; j < gap; j++) {+            if (blk[j] >= pos) return blk[j] == pos;+        }+        return false;+#endif+    }++    for (int32_t j = num_blocks * gap; j < cardinality; j++) {+        uint16_t v = carr[j];+        if (v >= pos) return (v == pos);+    }+    return false;+}++void array_container_offset(const array_container_t *c, container_t **loc,+                            container_t **hic, uint16_t offset);++//* Check whether a range of values from range_start (included) to range_end+//(excluded) is present. */+static inline bool array_container_contains_range(const array_container_t *arr,+                                                  uint32_t range_start,+                                                  uint32_t range_end) {+    const int32_t range_count = range_end - range_start;+    const uint16_t rs_included = (uint16_t)range_start;+    const uint16_t re_included = (uint16_t)(range_end - 1);++    // Empty range is always included+    if (range_count <= 0) {+        return true;+    }+    if (range_count > arr->cardinality) {+        return false;+    }++    const int32_t start =+        binarySearch(arr->array, arr->cardinality, rs_included);+    // If this sorted array contains all items in the range:+    // * the start item must be found+    // * the last item in range range_count must exist, and be the expected end+    // value+    return (start >= 0) && (arr->cardinality >= start + range_count) &&+           (arr->array[start + range_count - 1] == re_included);+}++/* Returns the smallest value (assumes not empty) */+CROARING_ALLOW_UNALIGNED+inline uint16_t array_container_minimum(const array_container_t *arr) {+    if (arr->cardinality == 0) return 0;+    return arr->array[0];+}++/* Returns the largest value (assumes not empty) */+CROARING_ALLOW_UNALIGNED+inline uint16_t array_container_maximum(const array_container_t *arr) {+    if (arr->cardinality == 0) return 0;+    return arr->array[arr->cardinality - 1];+}++/* Returns the number of values equal or smaller than x */+inline int array_container_rank(const array_container_t *arr, uint16_t x) {+    const int32_t idx = binarySearch(arr->array, arr->cardinality, x);+    const bool is_present = idx >= 0;+    if (is_present) {+        return idx + 1;+    } else {+        return -idx - 1;+    }+}++/*  bulk version of array_container_rank(); return number of consumed elements+ */+inline uint32_t array_container_rank_many(const array_container_t *arr,+                                          uint64_t start_rank,+                                          const uint32_t *begin,+                                          const uint32_t *end, uint64_t *ans) {+    const uint16_t high = (uint16_t)((*begin) >> 16);+    uint32_t pos = 0;+    const uint32_t *iter = begin;+    for (; iter != end; iter++) {+        uint32_t x = *iter;+        uint16_t xhigh = (uint16_t)(x >> 16);+        if (xhigh != high) return iter - begin;  // stop at next container++        const int32_t idx =+            binarySearch(arr->array + pos, arr->cardinality - pos, (uint16_t)x);+        const bool is_present = idx >= 0;+        if (is_present) {+            *(ans++) = start_rank + pos + (idx + 1);+            pos = idx + 1;+        } else {+            *(ans++) = start_rank + pos + (-idx - 1);+        }+    }+    return iter - begin;+}++/* Returns the index of x , if not exsist return -1 */+inline int array_container_get_index(const array_container_t *arr, uint16_t x) {+    const int32_t idx = binarySearch(arr->array, arr->cardinality, x);+    const bool is_present = idx >= 0;+    if (is_present) {+        return idx;+    } else {+        return -1;+    }+}++/* Returns the index of the first value equal or larger than x, or -1 */+inline int array_container_index_equalorlarger(const array_container_t *arr,+                                               uint16_t x) {+    const int32_t idx = binarySearch(arr->array, arr->cardinality, x);+    const bool is_present = idx >= 0;+    if (is_present) {+        return idx;+    } else {+        int32_t candidate = -idx - 1;+        if (candidate < arr->cardinality) return candidate;+        return -1;+    }+}++/*+ * Adds all values in range [min,max] using hint:+ *   nvals_less is the number of array values less than $min+ *   nvals_greater is the number of array values greater than $max+ */+static inline void array_container_add_range_nvals(array_container_t *array,+                                                   uint32_t min, uint32_t max,+                                                   int32_t nvals_less,+                                                   int32_t nvals_greater) {+    int32_t union_cardinality = nvals_less + (max - min + 1) + nvals_greater;+    if (union_cardinality > array->capacity) {+        array_container_grow(array, union_cardinality, true);+    }+    memmove(&(array->array[union_cardinality - nvals_greater]),+            &(array->array[array->cardinality - nvals_greater]),+            nvals_greater * sizeof(uint16_t));+    for (uint32_t i = 0; i <= max - min; i++) {+        array->array[nvals_less + i] = (uint16_t)(min + i);+    }+    array->cardinality = union_cardinality;+}++/**+ * Adds all values in range [min,max]. This function is currently unused+ * and left as a documentation.+ */+/*static inline void array_container_add_range(array_container_t *array,+                                             uint32_t min, uint32_t max) {+    int32_t nvals_greater = count_greater(array->array, array->cardinality,+max); int32_t nvals_less = count_less(array->array, array->cardinality -+nvals_greater, min); array_container_add_range_nvals(array, min, max,+nvals_less, nvals_greater);+}*/++/*+ * Removes all elements array[pos] .. array[pos+count-1]+ */+static inline void array_container_remove_range(array_container_t *array,+                                                uint32_t pos, uint32_t count) {+    if (count != 0) {+        memmove(&(array->array[pos]), &(array->array[pos + count]),+                (array->cardinality - pos - count) * sizeof(uint16_t));+        array->cardinality -= count;+    }+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_ARRAY_H_ */
+ cbits/CRoaring/include/roaring/containers/bitset.h view
@@ -0,0 +1,530 @@+/*+ * bitset.h+ *+ * Bitset containers store a set of 16-bit integers as a fixed-size bitmap.+ * The words pointer references an array of 64-bit words covering the full+ * 16-bit domain, with one bit per possible value. The cardinality field tracks+ * the number of set bits; when it is BITSET_UNKNOWN_CARDINALITY, the count must+ * be recomputed from the bitmap contents.+ *+ * This representation is used for denser containers because membership tests,+ * set operations, and sequential scans can be implemented efficiently with+ * word-level bitwise operations.+ */++#ifndef INCLUDE_CONTAINERS_BITSET_H_+#define INCLUDE_CONTAINERS_BITSET_H_++#include <stdbool.h>+#include <stdint.h>++#include <roaring/roaring_types.h>  // roaring_iterator++// Include other headers after roaring_types.h+#include <roaring/containers/container_defs.h>  // container_t, perfparameters+#include <roaring/portability.h>+#include <roaring/roaring_types.h>  // roaring_iterator+#include <roaring/utilasm.h>        // CROARING_ASM_XXX macros++#ifdef __cplusplus+extern "C" {+namespace roaring {++// Note: in pure C++ code, you should avoid putting `using` in header files+using api::roaring_iterator;+using api::roaring_iterator64;++namespace internal {+#endif++enum {+    BITSET_CONTAINER_SIZE_IN_WORDS = (1 << 16) / 64,+    BITSET_UNKNOWN_CARDINALITY = -1+};++STRUCT_CONTAINER(bitset_container_s) {+    int32_t cardinality;+    uint64_t *words;+};++typedef struct bitset_container_s bitset_container_t;++#define CAST_bitset(c) CAST(bitset_container_t *, c)  // safer downcast+#define const_CAST_bitset(c) CAST(const bitset_container_t *, c)+#define movable_CAST_bitset(c) movable_CAST(bitset_container_t **, c)++/* Create a new bitset. Return NULL in case of failure. */+bitset_container_t *bitset_container_create(void);++/* Create a bitset without zeroing the words. Caller must overwrite `words`+ * before the container is used. Return NULL in case of failure. */+bitset_container_t *bitset_container_create_uninitialized(void);++/* Free memory. */+void bitset_container_free(bitset_container_t *bitset);++/* Clear bitset (sets bits to 0). */+void bitset_container_clear(bitset_container_t *bitset);++/* Set all bits to 1. */+void bitset_container_set_all(bitset_container_t *bitset);++/* Duplicate bitset */+bitset_container_t *bitset_container_clone(const bitset_container_t *src);++/* Set the bit in [begin,end). WARNING: as of April 2016, this method is slow+ * and+ * should not be used in performance-sensitive code. Ever.  */+void bitset_container_set_range(bitset_container_t *bitset, uint32_t begin,+                                uint32_t end);++#if defined(CROARING_ASMBITMANIPOPTIMIZATION) && defined(__AVX2__)+/* Set the ith bit.  */+static inline void bitset_container_set(bitset_container_t *bitset,+                                        uint16_t pos) {+    uint64_t shift = 6;+    uint64_t offset;+    uint64_t p = pos;+    CROARING_ASM_SHIFT_RIGHT(p, shift, offset);+    uint64_t load = bitset->words[offset];+    CROARING_ASM_SET_BIT_INC_WAS_CLEAR(load, p, bitset->cardinality);+    bitset->words[offset] = load;+}++/* Unset the ith bit. Currently unused. Could be used for optimization. */+/*static inline void bitset_container_unset(bitset_container_t *bitset,+                                          uint16_t pos) {+    uint64_t shift = 6;+    uint64_t offset;+    uint64_t p = pos;+    CROARING_ASM_SHIFT_RIGHT(p, shift, offset);+    uint64_t load = bitset->words[offset];+    CROARING_ASM_CLEAR_BIT_DEC_WAS_SET(load, p, bitset->cardinality);+    bitset->words[offset] = load;+}*/++/* Add `pos' to `bitset'. Returns true if `pos' was not present. Might be slower+ * than bitset_container_set.  */+static inline bool bitset_container_add(bitset_container_t *bitset,+                                        uint16_t pos) {+    uint64_t shift = 6;+    uint64_t offset;+    uint64_t p = pos;+    CROARING_ASM_SHIFT_RIGHT(p, shift, offset);+    uint64_t load = bitset->words[offset];+    // could be possibly slightly further optimized+    const int32_t oldcard = bitset->cardinality;+    CROARING_ASM_SET_BIT_INC_WAS_CLEAR(load, p, bitset->cardinality);+    bitset->words[offset] = load;+    return bitset->cardinality - oldcard;+}++/* Remove `pos' from `bitset'. Returns true if `pos' was present.  Might be+ * slower than bitset_container_unset.  */+static inline bool bitset_container_remove(bitset_container_t *bitset,+                                           uint16_t pos) {+    uint64_t shift = 6;+    uint64_t offset;+    uint64_t p = pos;+    CROARING_ASM_SHIFT_RIGHT(p, shift, offset);+    uint64_t load = bitset->words[offset];+    // could be possibly slightly further optimized+    const int32_t oldcard = bitset->cardinality;+    CROARING_ASM_CLEAR_BIT_DEC_WAS_SET(load, p, bitset->cardinality);+    bitset->words[offset] = load;+    return oldcard - bitset->cardinality;+}++/* Get the value of the ith bit.  */+CROARING_ALLOW_UNALIGNED+inline bool bitset_container_get(const bitset_container_t *bitset,+                                 uint16_t pos) {+    uint64_t word = bitset->words[pos >> 6];+    const uint64_t p = pos;+    CROARING_ASM_INPLACESHIFT_RIGHT(word, p);+    return word & 1;+}++#else++/* Set the ith bit.  */+static inline void bitset_container_set(bitset_container_t *bitset,+                                        uint16_t pos) {+    const uint64_t old_word = bitset->words[pos >> 6];+    const int index = pos & 63;+    const uint64_t new_word = old_word | (UINT64_C(1) << index);+    bitset->cardinality += (uint32_t)((old_word ^ new_word) >> index);+    bitset->words[pos >> 6] = new_word;+}++/* Unset the ith bit. Currently unused.  */+/*static inline void bitset_container_unset(bitset_container_t *bitset,+                                          uint16_t pos) {+    const uint64_t old_word = bitset->words[pos >> 6];+    const int index = pos & 63;+    const uint64_t new_word = old_word & (~(UINT64_C(1) << index));+    bitset->cardinality -= (uint32_t)((old_word ^ new_word) >> index);+    bitset->words[pos >> 6] = new_word;+}*/++/* Add `pos' to `bitset'. Returns true if `pos' was not present. Might be slower+ * than bitset_container_set.  */+static inline bool bitset_container_add(bitset_container_t *bitset,+                                        uint16_t pos) {+    const uint64_t old_word = bitset->words[pos >> 6];+    const int index = pos & 63;+    const uint64_t new_word = old_word | (UINT64_C(1) << index);+    const uint64_t increment = (old_word ^ new_word) >> index;+    bitset->cardinality += (uint32_t)increment;+    bitset->words[pos >> 6] = new_word;+    return increment > 0;+}++/* Remove `pos' from `bitset'. Returns true if `pos' was present.  Might be+ * slower than bitset_container_unset.  */+static inline bool bitset_container_remove(bitset_container_t *bitset,+                                           uint16_t pos) {+    const uint64_t old_word = bitset->words[pos >> 6];+    const int index = pos & 63;+    const uint64_t new_word = old_word & (~(UINT64_C(1) << index));+    const uint64_t increment = (old_word ^ new_word) >> index;+    bitset->cardinality -= (uint32_t)increment;+    bitset->words[pos >> 6] = new_word;+    return increment > 0;+}++/* Get the value of the ith bit.  */+CROARING_ALLOW_UNALIGNED+inline bool bitset_container_get(const bitset_container_t *bitset,+                                 uint16_t pos) {+    const uint64_t word = bitset->words[pos >> 6];+    return (word >> (pos & 63)) & 1;+}++#endif++/*+ * Check if all bits are set in a range of positions from pos_start (included)+ * to pos_end (excluded).+ */+CROARING_ALLOW_UNALIGNED+static inline bool bitset_container_get_range(const bitset_container_t *bitset,+                                              uint32_t pos_start,+                                              uint32_t pos_end) {+    const uint32_t start = pos_start >> 6;+    const uint32_t end = pos_end >> 6;++    const uint64_t first = ~((1ULL << (pos_start & 0x3F)) - 1);+    const uint64_t last = (1ULL << (pos_end & 0x3F)) - 1;++    if (start == end)+        return ((bitset->words[end] & first & last) == (first & last));+    if ((bitset->words[start] & first) != first) return false;++    if ((end < BITSET_CONTAINER_SIZE_IN_WORDS) &&+        ((bitset->words[end] & last) != last)) {+        return false;+    }++    for (uint32_t i = start + 1;+         (i < BITSET_CONTAINER_SIZE_IN_WORDS) && (i < end); ++i) {+        if (bitset->words[i] != UINT64_C(0xFFFFFFFFFFFFFFFF)) return false;+    }++    return true;+}++/* Check whether `bitset' is present in `array'.  Calls bitset_container_get. */+inline bool bitset_container_contains(const bitset_container_t *bitset,+                                      uint16_t pos) {+    return bitset_container_get(bitset, pos);+}++/*+ * Check whether a range of bits from position `pos_start' (included) to+ * `pos_end' (excluded) is present in `bitset'.  Calls bitset_container_get_all.+ */+static inline bool bitset_container_contains_range(+    const bitset_container_t *bitset, uint32_t pos_start, uint32_t pos_end) {+    return bitset_container_get_range(bitset, pos_start, pos_end);+}++/* Get the number of bits set */+CROARING_ALLOW_UNALIGNED+static inline int bitset_container_cardinality(+    const bitset_container_t *bitset) {+    return bitset->cardinality;+}++/* Copy one container into another. We assume that they are distinct. */+void bitset_container_copy(const bitset_container_t *source,+                           bitset_container_t *dest);++/*  Add all the values [min,max) at a distance k*step from min: min,+ * min+step,.... */+void bitset_container_add_from_range(bitset_container_t *bitset, uint32_t min,+                                     uint32_t max, uint16_t step);++/* Get the number of bits set (force computation). This does not modify bitset.+ * To update the cardinality, you should do+ * bitset->cardinality =  bitset_container_compute_cardinality(bitset).*/+int bitset_container_compute_cardinality(const bitset_container_t *bitset);++/* Check whether this bitset is empty,+ *  it never modifies the bitset struct. */+static inline bool bitset_container_empty(const bitset_container_t *bitset) {+    if (bitset->cardinality == BITSET_UNKNOWN_CARDINALITY) {+        for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) {+            if ((bitset->words[i]) != 0) return false;+        }+        return true;+    }+    return bitset->cardinality == 0;+}++/* Get whether there is at least one bit set  (see bitset_container_empty for+   the reverse), the bitset is never modified */+static inline bool bitset_container_const_nonzero_cardinality(+    const bitset_container_t *bitset) {+    return !bitset_container_empty(bitset);+}++/*+ * Check whether the two bitsets intersect+ */+bool bitset_container_intersect(const bitset_container_t *src_1,+                                const bitset_container_t *src_2);++/* Computes the union of bitsets `src_1' and `src_2' into `dst'  and return the+ * cardinality. */+int bitset_container_or(const bitset_container_t *src_1,+                        const bitset_container_t *src_2,+                        bitset_container_t *dst);++/* Computes the union of bitsets `src_1' and `src_2' and return the cardinality.+ */+int bitset_container_or_justcard(const bitset_container_t *src_1,+                                 const bitset_container_t *src_2);++/* Computes the union of bitsets `src_1' and `src_2' into `dst' and return the+ * cardinality. Same as bitset_container_or. */+int bitset_container_union(const bitset_container_t *src_1,+                           const bitset_container_t *src_2,+                           bitset_container_t *dst);++/* Computes the union of bitsets `src_1' and `src_2'  and return the+ * cardinality. Same as bitset_container_or_justcard. */+int bitset_container_union_justcard(const bitset_container_t *src_1,+                                    const bitset_container_t *src_2);++/* Computes the union of bitsets `src_1' and `src_2' into `dst', but does+ * not update the cardinality. Provided to optimize chained operations. */+int bitset_container_union_nocard(const bitset_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  bitset_container_t *dst);++/* Computes the union of bitsets `src_1' and `src_2' into `dst', but does not+ * update the cardinality. Provided to optimize chained operations. */+int bitset_container_or_nocard(const bitset_container_t *src_1,+                               const bitset_container_t *src_2,+                               bitset_container_t *dst);++/* Computes the intersection of bitsets `src_1' and `src_2' into `dst' and+ * return the cardinality. */+int bitset_container_and(const bitset_container_t *src_1,+                         const bitset_container_t *src_2,+                         bitset_container_t *dst);++/* Computes the intersection of bitsets `src_1' and `src_2'  and return the+ * cardinality. */+int bitset_container_and_justcard(const bitset_container_t *src_1,+                                  const bitset_container_t *src_2);++/* Computes the intersection of bitsets `src_1' and `src_2' into `dst' and+ * return the cardinality. Same as bitset_container_and. */+int bitset_container_intersection(const bitset_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  bitset_container_t *dst);++/* Computes the intersection of bitsets `src_1' and `src_2' and return the+ * cardinality. Same as bitset_container_and_justcard. */+int bitset_container_intersection_justcard(const bitset_container_t *src_1,+                                           const bitset_container_t *src_2);++/* Computes the intersection of bitsets `src_1' and `src_2' into `dst', but does+ * not update the cardinality. Provided to optimize chained operations. */+int bitset_container_intersection_nocard(const bitset_container_t *src_1,+                                         const bitset_container_t *src_2,+                                         bitset_container_t *dst);++/* Computes the intersection of bitsets `src_1' and `src_2' into `dst', but does+ * not update the cardinality. Provided to optimize chained operations. */+int bitset_container_and_nocard(const bitset_container_t *src_1,+                                const bitset_container_t *src_2,+                                bitset_container_t *dst);++/* Computes the exclusive or of bitsets `src_1' and `src_2' into `dst' and+ * return the cardinality. */+int bitset_container_xor(const bitset_container_t *src_1,+                         const bitset_container_t *src_2,+                         bitset_container_t *dst);++/* Computes the exclusive or of bitsets `src_1' and `src_2' and return the+ * cardinality. */+int bitset_container_xor_justcard(const bitset_container_t *src_1,+                                  const bitset_container_t *src_2);++/* Computes the exclusive or of bitsets `src_1' and `src_2' into `dst', but does+ * not update the cardinality. Provided to optimize chained operations. */+int bitset_container_xor_nocard(const bitset_container_t *src_1,+                                const bitset_container_t *src_2,+                                bitset_container_t *dst);++/* Computes the and not of bitsets `src_1' and `src_2' into `dst' and return the+ * cardinality. */+int bitset_container_andnot(const bitset_container_t *src_1,+                            const bitset_container_t *src_2,+                            bitset_container_t *dst);++/* Computes the and not of bitsets `src_1' and `src_2'  and return the+ * cardinality. */+int bitset_container_andnot_justcard(const bitset_container_t *src_1,+                                     const bitset_container_t *src_2);++/* Computes the and not or of bitsets `src_1' and `src_2' into `dst', but does+ * not update the cardinality. Provided to optimize chained operations. */+int bitset_container_andnot_nocard(const bitset_container_t *src_1,+                                   const bitset_container_t *src_2,+                                   bitset_container_t *dst);++void bitset_container_offset(const bitset_container_t *c, container_t **loc,+                             container_t **hic, uint16_t offset);+/*+ * Write out the 16-bit integers contained in this container as a list of 32-bit+ * integers using base+ * as the starting value (it might be expected that base has zeros in its 16+ * least significant bits).+ * The function returns the number of values written.+ * The caller is responsible for allocating enough memory in out.+ * The out pointer should point to enough memory (the cardinality times 32+ * bits).+ */+int bitset_container_to_uint32_array(uint32_t *out,+                                     const bitset_container_t *bc,+                                     uint32_t base);++/*+ * Print this container using printf (useful for debugging).+ */+void bitset_container_printf(const bitset_container_t *v);++/*+ * Print this container using printf as a comma-separated list of 32-bit+ * integers starting at base.+ */+void bitset_container_printf_as_uint32_array(const bitset_container_t *v,+                                             uint32_t base);++bool bitset_container_validate(const bitset_container_t *v,+                               const char **reason);++/**+ * Return the serialized size in bytes of a container.+ */+static inline int32_t bitset_container_serialized_size_in_bytes(void) {+    return BITSET_CONTAINER_SIZE_IN_WORDS * 8;+}++/**+ * Return the the number of runs.+ */+int bitset_container_number_of_runs(bitset_container_t *bc);++bool bitset_container_iterate(const bitset_container_t *cont, uint32_t base,+                              roaring_iterator iterator, void *ptr);+bool bitset_container_iterate64(const bitset_container_t *cont, uint32_t base,+                                roaring_iterator64 iterator, uint64_t high_bits,+                                void *ptr);++/**+ * Writes the underlying array to buf, outputs how many bytes were written.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes written should be+ * bitset_container_size_in_bytes(container).+ */+int32_t bitset_container_write(const bitset_container_t *container, char *buf);++/**+ * Reads the instance from buf, outputs how many bytes were read.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes read should be bitset_container_size_in_bytes(container).+ * You need to provide the (known) cardinality.+ */+int32_t bitset_container_read(int32_t cardinality,+                              bitset_container_t *container, const char *buf);+/**+ * Return the serialized size in bytes of a container (see+ * bitset_container_write).+ * This is meant to be compatible with the Java and Go versions of Roaring and+ * assumes+ * that the cardinality of the container is already known or can be computed.+ */+static inline int32_t bitset_container_size_in_bytes(+    const bitset_container_t *container) {+    (void)container;+    return BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+}++/**+ * Return true if the two containers have the same content.+ */+bool bitset_container_equals(const bitset_container_t *container1,+                             const bitset_container_t *container2);++/**+ * Return true if container1 is a subset of container2.+ */+bool bitset_container_is_subset(const bitset_container_t *container1,+                                const bitset_container_t *container2);++/**+ * If the element of given rank is in this container, supposing that the first+ * element has rank start_rank, then the function returns true and sets element+ * accordingly.+ * Otherwise, it returns false and update start_rank.+ */+bool bitset_container_select(const bitset_container_t *container,+                             uint32_t *start_rank, uint32_t rank,+                             uint32_t *element);++/* Returns the smallest value (assumes not empty) */+uint16_t bitset_container_minimum(const bitset_container_t *container);++/* Returns the largest value (assumes not empty) */+uint16_t bitset_container_maximum(const bitset_container_t *container);++/* Returns the number of values equal or smaller than x */+int bitset_container_rank(const bitset_container_t *container, uint16_t x);++/* bulk version of bitset_container_rank(); return number of consumed elements+ */+uint32_t bitset_container_rank_many(const bitset_container_t *container,+                                    uint64_t start_rank, const uint32_t *begin,+                                    const uint32_t *end, uint64_t *ans);++/* Returns the index of x , if not exsist return -1 */+int bitset_container_get_index(const bitset_container_t *container, uint16_t x);++/* Returns the index of the first value equal or larger than x, or -1 */+int bitset_container_index_equalorlarger(const bitset_container_t *container,+                                         uint16_t x);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_BITSET_H_ */
+ cbits/CRoaring/include/roaring/containers/container_defs.h view
@@ -0,0 +1,99 @@+/*+ * container_defs.h+ *+ * Unlike containers.h (which is a file aggregating all the container includes,+ * like array.h, bitset.h, and run.h) this is a file included BY those headers+ * to do things like define the container base class `container_t`.+ */++#ifndef INCLUDE_CONTAINERS_CONTAINER_DEFS_H_+#define INCLUDE_CONTAINERS_CONTAINER_DEFS_H_++#ifdef __cplusplus+#include <type_traits>  // used by casting helper for compile-time check+#endif++// The preferences are a separate file to separate out tweakable parameters+#include <roaring/containers/perfparameters.h>++#ifdef __cplusplus+namespace roaring {+namespace internal {  // No extern "C" (contains template)+#endif++/*+ * Since roaring_array_t's definition is not opaque, the container type is+ * part of the API.  If it's not going to be `void*` then it needs a name, and+ * expectations are to prefix C library-exported names with `roaring_` etc.+ *+ * Rather than force the whole codebase to use the name `roaring_container_t`,+ * the few API appearances use the macro ROARING_CONTAINER_T.  Those includes+ * are prior to containers.h, so make a short private alias of `container_t`.+ * Then undefine the awkward macro so it's not used any more than it has to be.+ */+typedef ROARING_CONTAINER_T container_t;++/*+ * See ROARING_CONTAINER_T for notes on using container_t as a base class.+ * This macro helps make the following pattern look nicer:+ *+ *     #ifdef __cplusplus+ *     struct roaring_array_s : public container_t {+ *     #else+ *     struct roaring_array_s {+ *     #endif+ *         int32_t cardinality;+ *         int32_t capacity;+ *         uint16_t *array;+ *     }+ */+#if defined(__cplusplus)+#define STRUCT_CONTAINER(name) struct name : public container_t /* { ... } */+#else+#define STRUCT_CONTAINER(name) struct name /* { ... } */+#endif++/**+ * Since container_t* is not void* in C++, "dangerous" casts are not needed to+ * downcast; only a static_cast<> is needed.  Define a macro for static casting+ * which helps make casts more visible, and catches problems at compile-time+ * when building the C sources in C++ mode:+ *+ *     void some_func(container_t **c, ...) {  // double pointer, not single+ *         array_container_t *ac1 = (array_container_t *)(c);  // uncaught!!+ *+ *         array_container_t *ac2 = CAST(array_container_t *, c)  // C++ errors+ *         array_container_t *ac3 = CAST_array(c);  // shorthand for #2, errors+ *     }+ *+ * Trickier to do is a cast from `container**` to `array_container_t**`.  This+ * needs a reinterpret_cast<>, which sacrifices safety...so a template is used+ * leveraging <type_traits> to make sure it's legal in the C++ build.+ */+#ifdef __cplusplus+#define CAST(type, value) static_cast<type>(value)+#define movable_CAST(type, value) movable_CAST_HELPER<type>(value)++template <typename PPDerived, typename Base>+PPDerived movable_CAST_HELPER(Base **ptr_to_ptr) {+    typedef typename std::remove_pointer<PPDerived>::type PDerived;+    typedef typename std::remove_pointer<PDerived>::type Derived;+    static_assert(std::is_base_of<Base, Derived>::value,+                  "use movable_CAST() for container_t** => xxx_container_t**");+    return reinterpret_cast<Derived **>(ptr_to_ptr);+}+#else+#define CAST(type, value) ((type)value)+#define movable_CAST(type, value) ((type)value)+#endif++// Use for converting e.g. an `array_container_t**` to a `container_t**`+//+#define movable_CAST_base(c) movable_CAST(container_t **, c)++#ifdef __cplusplus+}+}  // namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_CONTAINER_DEFS_H_ */
+ cbits/CRoaring/include/roaring/containers/containers.h view
@@ -0,0 +1,2710 @@+/*+ * containers.h+ *+ * This header is the central internal interface for Roaring container+ * operations. It ties together the concrete container types (array, bitset,+ * run, and shared containers), their type codes, common helper functions, and+ * the mixed-operation headers used to combine different representations.+ *+ * In practice, it acts as the dispatch layer that lets higher-level bitmap+ * code manipulate containers through a uniform interface while still selecting+ * type-specific implementations when needed.+ */+#ifndef CONTAINERS_CONTAINERS_H+#define CONTAINERS_CONTAINERS_H++#include <assert.h>+#include <stdbool.h>+#include <stdio.h>++#include <roaring/bitset_util.h>+#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_andnot.h>+#include <roaring/containers/mixed_equal.h>+#include <roaring/containers/mixed_intersection.h>+#include <roaring/containers/mixed_negation.h>+#include <roaring/containers/mixed_subset.h>+#include <roaring/containers/mixed_union.h>+#include <roaring/containers/mixed_xor.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++// would enum be possible or better?++/**+ * The switch case statements follow+ * BITSET_CONTAINER_TYPE -- ARRAY_CONTAINER_TYPE -- RUN_CONTAINER_TYPE+ * so it makes more sense to number them 1, 2, 3 (in the vague hope that the+ * compiler might exploit this ordering).+ */++#define BITSET_CONTAINER_TYPE 1+#define ARRAY_CONTAINER_TYPE 2+#define RUN_CONTAINER_TYPE 3+#define SHARED_CONTAINER_TYPE 4++/**+ * Macros for pairing container type codes, suitable for switch statements.+ * Use PAIR_CONTAINER_TYPES() for the switch, CONTAINER_PAIR() for the cases:+ *+ *     switch (PAIR_CONTAINER_TYPES(type1, type2)) {+ *        case CONTAINER_PAIR(BITSET,ARRAY):+ *        ...+ *     }+ */+#define PAIR_CONTAINER_TYPES(type1, type2) (4 * (type1) + (type2))++#define CONTAINER_PAIR(name1, name2) \+    (4 * (name1##_CONTAINER_TYPE) + (name2##_CONTAINER_TYPE))++/**+ * A shared container is a wrapper around a container+ * with reference counting.+ */+STRUCT_CONTAINER(shared_container_s) {+    container_t *container;+    uint8_t typecode;+    croaring_refcount_t counter;  // to be managed atomically+};++typedef struct shared_container_s shared_container_t;++#define CAST_shared(c) CAST(shared_container_t *, c)  // safer downcast+#define const_CAST_shared(c) CAST(const shared_container_t *, c)+#define movable_CAST_shared(c) movable_CAST(shared_container_t **, c)++/*+ * With copy_on_write = true+ *  Create a new shared container if the typecode is not SHARED_CONTAINER_TYPE,+ * otherwise, increase the count+ * If copy_on_write = false, then clone.+ * Return NULL in case of failure.+ **/+container_t *get_copy_of_container(container_t *container, uint8_t *typecode,+                                   bool copy_on_write);++/* Frees a shared container (actually decrement its counter and only frees when+ * the counter falls to zero). */+void shared_container_free(shared_container_t *container);++/* extract a copy from the shared container, freeing the shared container if+there is just one instance left,+clone instances when the counter is higher than one+*/+container_t *shared_container_extract_copy(shared_container_t *container,+                                           uint8_t *typecode);++/* access to container underneath */+static inline const container_t *container_unwrap_shared(+    const container_t *candidate_shared_container, uint8_t *type) {+    if (*type == SHARED_CONTAINER_TYPE) {+        *type = const_CAST_shared(candidate_shared_container)->typecode;+        assert(*type != SHARED_CONTAINER_TYPE);+        return const_CAST_shared(candidate_shared_container)->container;+    } else {+        return candidate_shared_container;+    }+}++/* access to container underneath */+static inline container_t *container_mutable_unwrap_shared(container_t *c,+                                                           uint8_t *type) {+    if (*type == SHARED_CONTAINER_TYPE) {  // the passed in container is shared+        *type = CAST_shared(c)->typecode;+        assert(*type != SHARED_CONTAINER_TYPE);+        return CAST_shared(c)->container;  // return the enclosed container+    } else {+        return c;  // wasn't shared, so return as-is+    }+}++/* access to container underneath and queries its type */+static inline uint8_t get_container_type(const container_t *c, uint8_t type) {+    if (type == SHARED_CONTAINER_TYPE) {+        return const_CAST_shared(c)->typecode;+    } else {+        return type;+    }+}++/**+ * Copies a container, requires a typecode. This allocates new memory, caller+ * is responsible for deallocation. If the container is not shared, then it is+ * physically cloned. Sharable containers are not cloneable.+ */+container_t *container_clone(const container_t *container, uint8_t typecode);++/* access to container underneath, cloning it if needed */+static inline container_t *get_writable_copy_if_shared(container_t *c,+                                                       uint8_t *type) {+    if (*type == SHARED_CONTAINER_TYPE) {  // shared, return enclosed container+        return shared_container_extract_copy(CAST_shared(c), type);+    } else {+        return c;  // not shared, so return as-is+    }+}++/**+ * End of shared container code+ */++static const char *container_names[] = {"bitset", "array", "run", "shared"};+static const char *shared_container_names[] = {+    "bitset (shared)", "array (shared)", "run (shared)"};++// no matter what the initial container was, convert it to a bitset+// if a new container is produced, caller responsible for freeing the previous+// one+// container should not be a shared container+static inline bitset_container_t *container_to_bitset(container_t *c,+                                                      uint8_t typecode) {+    bitset_container_t *result = NULL;+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return CAST_bitset(c);  // nothing to do+        case ARRAY_CONTAINER_TYPE:+            result = bitset_container_from_array(CAST_array(c));+            return result;+        case RUN_CONTAINER_TYPE:+            result = bitset_container_from_run(CAST_run(c));+            return result;+        case SHARED_CONTAINER_TYPE:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * Get the container name from the typecode+ * (unused at time of writing)+ */+/*static inline const char *get_container_name(uint8_t typecode) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return container_names[0];+        case ARRAY_CONTAINER_TYPE:+            return container_names[1];+        case RUN_CONTAINER_TYPE:+            return container_names[2];+        case SHARED_CONTAINER_TYPE:+            return container_names[3];+        default:+            assert(false);+            roaring_unreachable;+            return "unknown";+    }+}*/++static inline const char *get_full_container_name(const container_t *c,+                                                  uint8_t typecode) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return container_names[0];+        case ARRAY_CONTAINER_TYPE:+            return container_names[1];+        case RUN_CONTAINER_TYPE:+            return container_names[2];+        case SHARED_CONTAINER_TYPE:+            switch (const_CAST_shared(c)->typecode) {+                case BITSET_CONTAINER_TYPE:+                    return shared_container_names[0];+                case ARRAY_CONTAINER_TYPE:+                    return shared_container_names[1];+                case RUN_CONTAINER_TYPE:+                    return shared_container_names[2];+                default:+                    assert(false);+                    roaring_unreachable;+                    return "unknown";+            }+            break;+        default:+            assert(false);+            roaring_unreachable;+            return "unknown";+    }+    roaring_unreachable;+    return NULL;+}++/**+ * Get the container cardinality (number of elements), requires a  typecode+ */+static inline int container_get_cardinality(const container_t *c,+                                            uint8_t typecode) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_cardinality(const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_cardinality(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_cardinality(const_CAST_run(c));+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++// returns true if a container is known to be full. Note that a lazy bitset+// container+// might be full without us knowing+static inline bool container_is_full(const container_t *c, uint8_t typecode) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_cardinality(const_CAST_bitset(c)) ==+                   (1 << 16);+        case ARRAY_CONTAINER_TYPE:+            return array_container_cardinality(const_CAST_array(c)) ==+                   (1 << 16);+        case RUN_CONTAINER_TYPE:+            return run_container_is_full(const_CAST_run(c));+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++static inline int container_shrink_to_fit(container_t *c, uint8_t type) {+    // Other bitmaps, maybe on other threads, read a shared container's array:+    // only its sole holder may move it.+    if (type == SHARED_CONTAINER_TYPE &&+        !croaring_refcount_is_unique(&CAST_shared(c)->counter)) {+        return 0;+    }+    c = container_mutable_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return 0;  // no shrinking possible+        case ARRAY_CONTAINER_TYPE:+            return array_container_shrink_to_fit(CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_shrink_to_fit(CAST_run(c));+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * make a container with a run of ones+ */+/* initially always use a run container, even if an array might be+ * marginally+ * smaller */+static inline container_t *container_range_of_ones(uint32_t range_start,+                                                   uint32_t range_end,+                                                   uint8_t *result_type) {+    assert(range_end >= range_start);+    uint64_t cardinality = range_end - range_start + 1;+    if (cardinality <= 2) {+        *result_type = ARRAY_CONTAINER_TYPE;+        return array_container_create_range(range_start, range_end);+    } else {+        *result_type = RUN_CONTAINER_TYPE;+        return run_container_create_range(range_start, range_end);+    }+}++/*  Create a container with all the values between in [min,max) at a+    distance k*step from min. */+static inline container_t *container_from_range(uint8_t *type, uint32_t min,+                                                uint32_t max, uint16_t step) {+    if (step == 0) return NULL;  // being paranoid+    if (step == 1) {+        return container_range_of_ones(min, max, type);+        // Note: the result is not always a run (need to check the cardinality)+        //*type = RUN_CONTAINER_TYPE;+        // return run_container_create_range(min, max);+    }+    int size = (max - min + step - 1) / step;+    if (size <= DEFAULT_MAX_SIZE) {  // array container+        *type = ARRAY_CONTAINER_TYPE;+        array_container_t *array = array_container_create_given_capacity(size);+        array_container_add_from_range(array, min, max, step);+        assert(array->cardinality == size);+        return array;+    } else {  // bitset container+        *type = BITSET_CONTAINER_TYPE;+        bitset_container_t *bitset = bitset_container_create();+        bitset_container_add_from_range(bitset, min, max, step);+        assert(bitset->cardinality == size);+        return bitset;+    }+}++/**+ * "repair" the container after lazy operations.+ */+static inline container_t *container_repair_after_lazy(container_t *c,+                                                       uint8_t *type) {+    c = get_writable_copy_if_shared(c, type);  // !!! unnecessary cloning+    container_t *result = NULL;+    switch (*type) {+        case BITSET_CONTAINER_TYPE: {+            bitset_container_t *bc = CAST_bitset(c);+            bc->cardinality = bitset_container_compute_cardinality(bc);+            if (bc->cardinality <= DEFAULT_MAX_SIZE) {+                result = array_container_from_bitset(bc);+                bitset_container_free(bc);+                *type = ARRAY_CONTAINER_TYPE;+                return result;+            }+            return c;+        }+        case ARRAY_CONTAINER_TYPE:+            return c;  // nothing to do+        case RUN_CONTAINER_TYPE:+            return convert_run_to_efficient_container_and_free(CAST_run(c),+                                                               type);+        case SHARED_CONTAINER_TYPE:+            assert(false);+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * Writes the underlying array to buf, outputs how many bytes were written.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes written should be+ * container_write(container, buf).+ *+ */+static inline int32_t container_write(const container_t *c, uint8_t typecode,+                                      char *buf) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_write(const_CAST_bitset(c), buf);+        case ARRAY_CONTAINER_TYPE:+            return array_container_write(const_CAST_array(c), buf);+        case RUN_CONTAINER_TYPE:+            return run_container_write(const_CAST_run(c), buf);+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * Get the container size in bytes under portable serialization (see+ * container_write), requires a+ * typecode+ */+static inline int32_t container_size_in_bytes(const container_t *c,+                                              uint8_t typecode) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_size_in_bytes(const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_size_in_bytes(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_size_in_bytes(const_CAST_run(c));+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * print the container (useful for debugging), requires a  typecode+ */+void container_printf(const container_t *container, uint8_t typecode);++/**+ * print the content of the container as a comma-separated list of 32-bit values+ * starting at base, requires a  typecode+ */+void container_printf_as_uint32_array(const container_t *container,+                                      uint8_t typecode, uint32_t base);++bool container_internal_validate(const container_t *container, uint8_t typecode,+                                 const char **reason);++/**+ * Checks whether a container is not empty, requires a  typecode+ */+static inline bool container_nonzero_cardinality(const container_t *c,+                                                 uint8_t typecode) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_const_nonzero_cardinality(+                const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_nonzero_cardinality(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_nonzero_cardinality(const_CAST_run(c));+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * Recover memory from a container, requires a  typecode+ */+void container_free(container_t *container, uint8_t typecode);++/**+ * Convert a container to an array of values, requires a  typecode as well as a+ * "base" (most significant values)+ * Returns number of ints added.+ */+static inline int container_to_uint32_array(uint32_t *output,+                                            const container_t *c,+                                            uint8_t typecode, uint32_t base) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_to_uint32_array(output,+                                                    const_CAST_bitset(c), base);+        case ARRAY_CONTAINER_TYPE:+            return array_container_to_uint32_array(output, const_CAST_array(c),+                                                   base);+        case RUN_CONTAINER_TYPE:+            return run_container_to_uint32_array(output, const_CAST_run(c),+                                                 base);+    }+    assert(false);+    roaring_unreachable;+    return 0;  // unreached+}++/**+ * Add a value to a container, requires a  typecode, fills in new_typecode and+ * return (possibly different) container.+ * This function may allocate a new container, and caller is responsible for+ * memory deallocation+ */+static inline container_t *container_add(+    container_t *c, uint16_t val,+    uint8_t typecode,  // !!! should be second argument?+    uint8_t *new_typecode) {+    c = get_writable_copy_if_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            bitset_container_set(CAST_bitset(c), val);+            *new_typecode = BITSET_CONTAINER_TYPE;+            return c;+        case ARRAY_CONTAINER_TYPE: {+            array_container_t *ac = CAST_array(c);+            if (array_container_try_add(ac, val, DEFAULT_MAX_SIZE) != -1) {+                *new_typecode = ARRAY_CONTAINER_TYPE;+                return ac;+            } else {+                bitset_container_t *bitset = bitset_container_from_array(ac);+                bitset_container_add(bitset, val);+                *new_typecode = BITSET_CONTAINER_TYPE;+                return bitset;+            }+        } break;+        case RUN_CONTAINER_TYPE:+            // per Java, no container type adjustments are done (revisit?)+            run_container_add(CAST_run(c), val);+            *new_typecode = RUN_CONTAINER_TYPE;+            return c;+        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Remove a value from a container, requires a  typecode, fills in new_typecode+ * and+ * return (possibly different) container.+ * This function may allocate a new container, and caller is responsible for+ * memory deallocation+ */+static inline container_t *container_remove(+    container_t *c, uint16_t val,+    uint8_t typecode,  // !!! should be second argument?+    uint8_t *new_typecode) {+    c = get_writable_copy_if_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            if (bitset_container_remove(CAST_bitset(c), val)) {+                int card = bitset_container_cardinality(CAST_bitset(c));+                if (card <= DEFAULT_MAX_SIZE) {+                    *new_typecode = ARRAY_CONTAINER_TYPE;+                    return array_container_from_bitset(CAST_bitset(c));+                }+            }+            *new_typecode = typecode;+            return c;+        case ARRAY_CONTAINER_TYPE:+            *new_typecode = typecode;+            array_container_remove(CAST_array(c), val);+            return c;+        case RUN_CONTAINER_TYPE:+            // per Java, no container type adjustments are done (revisit?)+            run_container_remove(CAST_run(c), val);+            *new_typecode = RUN_CONTAINER_TYPE;+            return c;+        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Check whether a value is in a container, requires a typecode+ */+inline bool container_contains(+    const container_t *c, uint16_t val,+    uint8_t typecode  // !!! should be second argument?+) {+    if (typecode == SHARED_CONTAINER_TYPE) {+        typecode = const_CAST_shared(c)->typecode;+        assert(typecode != SHARED_CONTAINER_TYPE);+        c = const_CAST_shared(c)->container;+    }++    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_get(const_CAST_bitset(c), val);+        case ARRAY_CONTAINER_TYPE:+            return array_container_contains(const_CAST_array(c), val);+        case RUN_CONTAINER_TYPE:+            return run_container_contains(const_CAST_run(c), val);+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++/**+ * Check whether a range of values from range_start (included) to range_end+ * (excluded) is in a container, requires a typecode+ */+static inline bool container_contains_range(+    const container_t *c, uint32_t range_start, uint32_t range_end,+    uint8_t typecode  // !!! should be second argument?+) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_get_range(const_CAST_bitset(c), range_start,+                                              range_end);+        case ARRAY_CONTAINER_TYPE:+            return array_container_contains_range(const_CAST_array(c),+                                                  range_start, range_end);+        case RUN_CONTAINER_TYPE:+            return run_container_contains_range(const_CAST_run(c), range_start,+                                                range_end);+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++/**+ * Returns true if the two containers have the same content. Note that+ * two containers having different types can be "equal" in this sense.+ */+static inline bool container_equals(const container_t *c1, uint8_t type1,+                                    const container_t *c2, uint8_t type2) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            return bitset_container_equals(const_CAST_bitset(c1),+                                           const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, RUN):+            return run_container_equals_bitset(const_CAST_run(c2),+                                               const_CAST_bitset(c1));++        case CONTAINER_PAIR(RUN, BITSET):+            return run_container_equals_bitset(const_CAST_run(c1),+                                               const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, ARRAY):+            // java would always return false?+            return array_container_equal_bitset(const_CAST_array(c2),+                                                const_CAST_bitset(c1));++        case CONTAINER_PAIR(ARRAY, BITSET):+            // java would always return false?+            return array_container_equal_bitset(const_CAST_array(c1),+                                                const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, RUN):+            return run_container_equals_array(const_CAST_run(c2),+                                              const_CAST_array(c1));++        case CONTAINER_PAIR(RUN, ARRAY):+            return run_container_equals_array(const_CAST_run(c1),+                                              const_CAST_array(c2));++        case CONTAINER_PAIR(ARRAY, ARRAY):+            return array_container_equals(const_CAST_array(c1),+                                          const_CAST_array(c2));++        case CONTAINER_PAIR(RUN, RUN):+            return run_container_equals(const_CAST_run(c1), const_CAST_run(c2));++        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++/**+ * Returns true if the container c1 is a subset of the container c2. Note that+ * c1 can be a subset of c2 even if they have a different type.+ */+static inline bool container_is_subset(const container_t *c1, uint8_t type1,+                                       const container_t *c2, uint8_t type2) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            return bitset_container_is_subset(const_CAST_bitset(c1),+                                              const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, RUN):+            return bitset_container_is_subset_run(const_CAST_bitset(c1),+                                                  const_CAST_run(c2));++        case CONTAINER_PAIR(RUN, BITSET):+            return run_container_is_subset_bitset(const_CAST_run(c1),+                                                  const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, ARRAY):+            return false;  // by construction, size(c1) > size(c2)++        case CONTAINER_PAIR(ARRAY, BITSET):+            return array_container_is_subset_bitset(const_CAST_array(c1),+                                                    const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, RUN):+            return array_container_is_subset_run(const_CAST_array(c1),+                                                 const_CAST_run(c2));++        case CONTAINER_PAIR(RUN, ARRAY):+            return run_container_is_subset_array(const_CAST_run(c1),+                                                 const_CAST_array(c2));++        case CONTAINER_PAIR(ARRAY, ARRAY):+            return array_container_is_subset(const_CAST_array(c1),+                                             const_CAST_array(c2));++        case CONTAINER_PAIR(RUN, RUN):+            return run_container_is_subset(const_CAST_run(c1),+                                           const_CAST_run(c2));++        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++// macro-izations possibilities for generic non-inplace binary-op dispatch++/**+ * Compute intersection between two containers, generate a new container (having+ * type result_type), requires a typecode. This allocates new memory, caller+ * is responsible for deallocation.+ */+static inline container_t *container_and(const container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type =+                bitset_bitset_container_intersection(+                    const_CAST_bitset(c1), const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            result = array_container_create();+            array_container_intersection(+                const_CAST_array(c1), const_CAST_array(c2), CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            result = run_container_create();+            run_container_intersection(const_CAST_run(c1), const_CAST_run(c2),+                                       CAST_run(result));+            return convert_run_to_efficient_container_and_free(CAST_run(result),+                                                               result_type);++        case CONTAINER_PAIR(BITSET, ARRAY):+            result = array_container_create();+            array_bitset_container_intersection(const_CAST_array(c2),+                                                const_CAST_bitset(c1),+                                                CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            result = array_container_create();+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_bitset_container_intersection(const_CAST_array(c1),+                                                const_CAST_bitset(c2),+                                                CAST_array(result));+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            *result_type =+                run_bitset_container_intersection(+                    const_CAST_run(c2), const_CAST_bitset(c1), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type =+                run_bitset_container_intersection(+                    const_CAST_run(c1), const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = array_container_create();+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_run_container_intersection(+                const_CAST_array(c1), const_CAST_run(c2), CAST_array(result));+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            result = array_container_create();+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_run_container_intersection(+                const_CAST_array(c2), const_CAST_run(c1), CAST_array(result));+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Compute the size of the intersection between two containers.+ */+static inline int container_and_cardinality(const container_t *c1,+                                            uint8_t type1,+                                            const container_t *c2,+                                            uint8_t type2) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            return bitset_container_and_justcard(const_CAST_bitset(c1),+                                                 const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, ARRAY):+            return array_container_intersection_cardinality(+                const_CAST_array(c1), const_CAST_array(c2));++        case CONTAINER_PAIR(RUN, RUN):+            return run_container_intersection_cardinality(const_CAST_run(c1),+                                                          const_CAST_run(c2));++        case CONTAINER_PAIR(BITSET, ARRAY):+            return array_bitset_container_intersection_cardinality(+                const_CAST_array(c2), const_CAST_bitset(c1));++        case CONTAINER_PAIR(ARRAY, BITSET):+            return array_bitset_container_intersection_cardinality(+                const_CAST_array(c1), const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, RUN):+            return run_bitset_container_intersection_cardinality(+                const_CAST_run(c2), const_CAST_bitset(c1));++        case CONTAINER_PAIR(RUN, BITSET):+            return run_bitset_container_intersection_cardinality(+                const_CAST_run(c1), const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, RUN):+            return array_run_container_intersection_cardinality(+                const_CAST_array(c1), const_CAST_run(c2));++        case CONTAINER_PAIR(RUN, ARRAY):+            return array_run_container_intersection_cardinality(+                const_CAST_array(c2), const_CAST_run(c1));++        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++/**+ * Check whether two containers intersect.+ */+static inline bool container_intersect(const container_t *c1, uint8_t type1,+                                       const container_t *c2, uint8_t type2) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            return bitset_container_intersect(const_CAST_bitset(c1),+                                              const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, ARRAY):+            return array_container_intersect(const_CAST_array(c1),+                                             const_CAST_array(c2));++        case CONTAINER_PAIR(RUN, RUN):+            return run_container_intersect(const_CAST_run(c1),+                                           const_CAST_run(c2));++        case CONTAINER_PAIR(BITSET, ARRAY):+            return array_bitset_container_intersect(const_CAST_array(c2),+                                                    const_CAST_bitset(c1));++        case CONTAINER_PAIR(ARRAY, BITSET):+            return array_bitset_container_intersect(const_CAST_array(c1),+                                                    const_CAST_bitset(c2));++        case CONTAINER_PAIR(BITSET, RUN):+            return run_bitset_container_intersect(const_CAST_run(c2),+                                                  const_CAST_bitset(c1));++        case CONTAINER_PAIR(RUN, BITSET):+            return run_bitset_container_intersect(const_CAST_run(c1),+                                                  const_CAST_bitset(c2));++        case CONTAINER_PAIR(ARRAY, RUN):+            return array_run_container_intersect(const_CAST_array(c1),+                                                 const_CAST_run(c2));++        case CONTAINER_PAIR(RUN, ARRAY):+            return array_run_container_intersect(const_CAST_array(c2),+                                                 const_CAST_run(c1));++        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++/**+ * Compute intersection between two containers, with result in the first+ container if possible. If the returned pointer is identical to c1,+ then the container has been modified. If the returned pointer is different+ from c1, then a new container has been created and the caller is responsible+ for freeing it.+ The type of the first container may change. Returns the modified+ (and possibly new) container.+*/+static inline container_t *container_iand(container_t *c1, uint8_t type1,+                                          const container_t *c2, uint8_t type2,+                                          uint8_t *result_type) {+    c1 = get_writable_copy_if_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type = bitset_bitset_container_intersection_inplace(+                               CAST_bitset(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            array_container_intersection_inplace(CAST_array(c1),+                                                 const_CAST_array(c2));+            *result_type = ARRAY_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(RUN, RUN):+            result = run_container_create();+            run_container_intersection(const_CAST_run(c1), const_CAST_run(c2),+                                       CAST_run(result));+            // as of January 2016, Java code used non-in-place intersection for+            // two runcontainers+            return convert_run_to_efficient_container_and_free(CAST_run(result),+                                                               result_type);++        case CONTAINER_PAIR(BITSET, ARRAY):+            // c1 is a bitmap so no inplace possible+            result = array_container_create();+            array_bitset_container_intersection(const_CAST_array(c2),+                                                const_CAST_bitset(c1),+                                                CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_bitset_container_intersection(+                const_CAST_array(c1), const_CAST_bitset(c2),+                CAST_array(c1));  // result is allowed to be same as c1+            return c1;++        case CONTAINER_PAIR(BITSET, RUN):+            // will attempt in-place computation+            *result_type = run_bitset_container_intersection(+                               const_CAST_run(c2), const_CAST_bitset(c1), &c1)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type =+                run_bitset_container_intersection(+                    const_CAST_run(c1), const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = array_container_create();+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_run_container_intersection(+                const_CAST_array(c1), const_CAST_run(c2), CAST_array(result));+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            result = array_container_create();+            *result_type = ARRAY_CONTAINER_TYPE;  // never bitset+            array_run_container_intersection(+                const_CAST_array(c2), const_CAST_run(c1), CAST_array(result));+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Compute union between two containers, generate a new container (having type+ * result_type), requires a typecode. This allocates new memory, caller+ * is responsible for deallocation.+ */+static inline container_t *container_or(const container_t *c1, uint8_t type1,+                                        const container_t *c2, uint8_t type2,+                                        uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            result = bitset_container_create();+            bitset_container_or(const_CAST_bitset(c1), const_CAST_bitset(c2),+                                CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type =+                array_array_container_union(const_CAST_array(c1),+                                            const_CAST_array(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            result = run_container_create();+            run_container_union(const_CAST_run(c1), const_CAST_run(c2),+                                CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // todo: could be optimized since will never convert to array+            result = convert_run_to_efficient_container_and_free(+                CAST_run(result), result_type);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            result = bitset_container_create();+            array_bitset_container_union(const_CAST_array(c2),+                                         const_CAST_bitset(c1),+                                         CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            result = bitset_container_create();+            array_bitset_container_union(const_CAST_array(c1),+                                         const_CAST_bitset(c2),+                                         CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c2), CAST_run(result));+                return result;+            }+            result = bitset_container_create();+            run_bitset_container_union(+                const_CAST_run(c2), const_CAST_bitset(c1), CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            if (run_container_is_full(const_CAST_run(c1))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c1), CAST_run(result));+                return result;+            }+            result = bitset_container_create();+            run_bitset_container_union(+                const_CAST_run(c1), const_CAST_bitset(c2), CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c1), const_CAST_run(c2),+                                      CAST_run(result));+            result = convert_run_to_efficient_container_and_free(+                CAST_run(result), result_type);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c2), const_CAST_run(c1),+                                      CAST_run(result));+            result = convert_run_to_efficient_container_and_free(+                CAST_run(result), result_type);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;  // unreached+    }+}++/**+ * Compute union between two containers, generate a new container (having type+ * result_type), requires a typecode. This allocates new memory, caller+ * is responsible for deallocation.+ *+ * This lazy version delays some operations such as the maintenance of the+ * cardinality. It requires repair later on the generated containers.+ */+static inline container_t *container_lazy_or(const container_t *c1,+                                             uint8_t type1,+                                             const container_t *c2,+                                             uint8_t type2,+                                             uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            result = bitset_container_create();+            bitset_container_or_nocard(const_CAST_bitset(c1),+                                       const_CAST_bitset(c2),+                                       CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type =+                array_array_container_lazy_union(const_CAST_array(c1),+                                                 const_CAST_array(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            result = run_container_create();+            run_container_union(const_CAST_run(c1), const_CAST_run(c2),+                                CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // we are being lazy+            result = convert_run_to_efficient_container_and_free(+                CAST_run(result), result_type);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            result = bitset_container_create();+            array_bitset_container_lazy_union(const_CAST_array(c2),+                                              const_CAST_bitset(c1),+                                              CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            result = bitset_container_create();+            array_bitset_container_lazy_union(const_CAST_array(c1),+                                              const_CAST_bitset(c2),+                                              CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c2), CAST_run(result));+                return result;+            }+            result = bitset_container_create();+            run_bitset_container_lazy_union(const_CAST_run(c2),+                                            const_CAST_bitset(c1),+                                            CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            if (run_container_is_full(const_CAST_run(c1))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c1), CAST_run(result));+                return result;+            }+            result = bitset_container_create();+            run_bitset_container_lazy_union(const_CAST_run(c1),+                                            const_CAST_bitset(c2),+                                            CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c1), const_CAST_run(c2),+                                      CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container(result, result_type);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c2), const_CAST_run(c1),+                                      CAST_run(result));  // TODO make lazy+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container(result, result_type);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;  // unreached+    }+}++/**+ * Compute the union between two containers, with result in the first container.+ * If the returned pointer is identical to c1, then the container has been+ * modified.+ * If the returned pointer is different from c1, then a new container has been+ * created and the caller is responsible for freeing it.+ * The type of the first container may change. Returns the modified+ * (and possibly new) container+ */+static inline container_t *container_ior(container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type) {+    c1 = get_writable_copy_if_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            bitset_container_or(const_CAST_bitset(c1), const_CAST_bitset(c2),+                                CAST_bitset(c1));+#if CROARING_OR_BITSET_CONVERSION_TO_FULL+            if (CAST_bitset(c1)->cardinality == (1 << 16)) {  // we convert+                result = run_container_create_range(0, (1 << 16));+                *result_type = RUN_CONTAINER_TYPE;+                return result;+            }+#endif+            *result_type = BITSET_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type = array_array_container_inplace_union(+                               CAST_array(c1), const_CAST_array(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            if ((result == NULL) && (*result_type == ARRAY_CONTAINER_TYPE)) {+                return c1;  // the computation was done in-place!+            }+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            run_container_union_inplace(CAST_run(c1), const_CAST_run(c2));+            return convert_run_to_efficient_container(CAST_run(c1),+                                                      result_type);++        case CONTAINER_PAIR(BITSET, ARRAY):+            array_bitset_container_union(+                const_CAST_array(c2), const_CAST_bitset(c1), CAST_bitset(c1));+            *result_type = BITSET_CONTAINER_TYPE;  // never array+            return c1;++        case CONTAINER_PAIR(ARRAY, BITSET):+            // c1 is an array, so no in-place possible+            result = bitset_container_create();+            *result_type = BITSET_CONTAINER_TYPE;+            array_bitset_container_union(const_CAST_array(c1),+                                         const_CAST_bitset(c2),+                                         CAST_bitset(result));+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c2), CAST_run(result));+                return result;+            }+            run_bitset_container_union(const_CAST_run(c2),+                                       const_CAST_bitset(c1),+                                       CAST_bitset(c1));  // allowed+            *result_type = BITSET_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(RUN, BITSET):+            if (run_container_is_full(const_CAST_run(c1))) {+                *result_type = RUN_CONTAINER_TYPE;+                return c1;+            }+            result = bitset_container_create();+            run_bitset_container_union(+                const_CAST_run(c1), const_CAST_bitset(c2), CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c1), const_CAST_run(c2),+                                      CAST_run(result));+            result = convert_run_to_efficient_container_and_free(+                CAST_run(result), result_type);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            array_run_container_inplace_union(const_CAST_array(c2),+                                              CAST_run(c1));+            c1 = convert_run_to_efficient_container(CAST_run(c1), result_type);+            return c1;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Compute the union between two containers, with result in the first container.+ * If the returned pointer is identical to c1, then the container has been+ * modified.+ * If the returned pointer is different from c1, then a new container has been+ * created and the caller is responsible for freeing it.+ * The type of the first container may change. Returns the modified+ * (and possibly new) container+ *+ * This lazy version delays some operations such as the maintenance of the+ * cardinality. It requires repair later on the generated containers.+ */+static inline container_t *container_lazy_ior(container_t *c1, uint8_t type1,+                                              const container_t *c2,+                                              uint8_t type2,+                                              uint8_t *result_type) {+    assert(type1 != SHARED_CONTAINER_TYPE);+    // c1 = get_writable_copy_if_shared(c1,&type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+#if CROARING_LAZY_OR_BITSET_CONVERSION_TO_FULL+            // if we have two bitsets, we might as well compute the cardinality+            bitset_container_or(const_CAST_bitset(c1), const_CAST_bitset(c2),+                                CAST_bitset(c1));+            // it is possible that two bitsets can lead to a full container+            if (CAST_bitset(c1)->cardinality == (1 << 16)) {  // we convert+                result = run_container_create_range(0, (1 << 16));+                *result_type = RUN_CONTAINER_TYPE;+                return result;+            }+#else+            bitset_container_or_nocard(const_CAST_bitset(c1),+                                       const_CAST_bitset(c2), CAST_bitset(c1));++#endif+            *result_type = BITSET_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type = array_array_container_lazy_inplace_union(+                               CAST_array(c1), const_CAST_array(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            if ((result == NULL) && (*result_type == ARRAY_CONTAINER_TYPE)) {+                return c1;  // the computation was done in-place!+            }+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            run_container_union_inplace(CAST_run(c1), const_CAST_run(c2));+            *result_type = RUN_CONTAINER_TYPE;+            return convert_run_to_efficient_container(CAST_run(c1),+                                                      result_type);++        case CONTAINER_PAIR(BITSET, ARRAY):+            array_bitset_container_lazy_union(const_CAST_array(c2),+                                              const_CAST_bitset(c1),+                                              CAST_bitset(c1));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;                // never array+            return c1;++        case CONTAINER_PAIR(ARRAY, BITSET):+            // c1 is an array, so no in-place possible+            result = bitset_container_create();+            *result_type = BITSET_CONTAINER_TYPE;+            array_bitset_container_lazy_union(const_CAST_array(c1),+                                              const_CAST_bitset(c2),+                                              CAST_bitset(result));  // is lazy+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = run_container_create();+                *result_type = RUN_CONTAINER_TYPE;+                run_container_copy(const_CAST_run(c2), CAST_run(result));+                return result;+            }+            run_bitset_container_lazy_union(+                const_CAST_run(c2), const_CAST_bitset(c1),+                CAST_bitset(c1));  // allowed //  lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(RUN, BITSET):+            if (run_container_is_full(const_CAST_run(c1))) {+                *result_type = RUN_CONTAINER_TYPE;+                return c1;+            }+            result = bitset_container_create();+            run_bitset_container_lazy_union(const_CAST_run(c1),+                                            const_CAST_bitset(c2),+                                            CAST_bitset(result));  //  lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = run_container_create();+            array_run_container_union(const_CAST_array(c1), const_CAST_run(c2),+                                      CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container_and_free(result,+            // result_type);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            array_run_container_inplace_union(const_CAST_array(c2),+                                              CAST_run(c1));+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container_and_free(result,+            // result_type);+            return c1;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Compute symmetric difference (xor) between two containers, generate a new+ * container (having type result_type), requires a typecode. This allocates new+ * memory, caller is responsible for deallocation.+ */+static inline container_t *container_xor(const container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type =+                bitset_bitset_container_xor(const_CAST_bitset(c1),+                                            const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type =+                array_array_container_xor(const_CAST_array(c1),+                                          const_CAST_array(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            *result_type = (uint8_t)run_run_container_xor(+                const_CAST_run(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            *result_type =+                array_bitset_container_xor(const_CAST_array(c2),+                                           const_CAST_bitset(c1), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            *result_type =+                array_bitset_container_xor(const_CAST_array(c1),+                                           const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            *result_type =+                run_bitset_container_xor(const_CAST_run(c2),+                                         const_CAST_bitset(c1), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type =+                run_bitset_container_xor(const_CAST_run(c1),+                                         const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            *result_type = (uint8_t)array_run_container_xor(+                const_CAST_array(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            *result_type = (uint8_t)array_run_container_xor(+                const_CAST_array(c2), const_CAST_run(c1), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;  // unreached+    }+}++/* Applies an offset to the non-empty container 'c'.+ * The results are stored in new containers returned via 'lo' and 'hi', for the+ * low and high halves of the result (where the low half matches the original+ * key and the high one corresponds to values for the following key). Either one+ * of 'lo' and 'hi' are allowed to be 'NULL', but not both. Whenever one of them+ * is not 'NULL', it should point to a 'NULL' container. Whenever one of them is+ * 'NULL' the shifted elements for that part will not be computed. If either of+ * the resulting containers turns out to be empty, the pointed container will+ * remain 'NULL'.+ */+static inline void container_add_offset(const container_t *c, uint8_t type,+                                        container_t **lo, container_t **hi,+                                        uint16_t offset) {+    assert(offset != 0);+    assert(container_nonzero_cardinality(c, type));+    assert(lo != NULL || hi != NULL);+    assert(lo == NULL || *lo == NULL);+    assert(hi == NULL || *hi == NULL);++    switch (type) {+        case BITSET_CONTAINER_TYPE:+            bitset_container_offset(const_CAST_bitset(c), lo, hi, offset);+            break;+        case ARRAY_CONTAINER_TYPE:+            array_container_offset(const_CAST_array(c), lo, hi, offset);+            break;+        case RUN_CONTAINER_TYPE:+            run_container_offset(const_CAST_run(c), lo, hi, offset);+            break;+        default:+            assert(false);+            roaring_unreachable;+            break;+    }+}++/**+ * Compute xor between two containers, generate a new container (having type+ * result_type), requires a typecode. This allocates new memory, caller+ * is responsible for deallocation.+ *+ * This lazy version delays some operations such as the maintenance of the+ * cardinality. It requires repair later on the generated containers.+ */+static inline container_t *container_lazy_xor(const container_t *c1,+                                              uint8_t type1,+                                              const container_t *c2,+                                              uint8_t type2,+                                              uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            result = bitset_container_create();+            bitset_container_xor_nocard(const_CAST_bitset(c1),+                                        const_CAST_bitset(c2),+                                        CAST_bitset(result));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type =+                array_array_container_lazy_xor(const_CAST_array(c1),+                                               const_CAST_array(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            // nothing special done yet.+            *result_type = (uint8_t)run_run_container_xor(+                const_CAST_run(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            result = bitset_container_create();+            *result_type = BITSET_CONTAINER_TYPE;+            array_bitset_container_lazy_xor(const_CAST_array(c2),+                                            const_CAST_bitset(c1),+                                            CAST_bitset(result));+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            result = bitset_container_create();+            *result_type = BITSET_CONTAINER_TYPE;+            array_bitset_container_lazy_xor(const_CAST_array(c1),+                                            const_CAST_bitset(c2),+                                            CAST_bitset(result));+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            result = bitset_container_create();+            run_bitset_container_lazy_xor(+                const_CAST_run(c2), const_CAST_bitset(c1), CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            result = bitset_container_create();+            run_bitset_container_lazy_xor(+                const_CAST_run(c1), const_CAST_bitset(c2), CAST_bitset(result));+            *result_type = BITSET_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            result = run_container_create();+            array_run_container_lazy_xor(const_CAST_array(c1),+                                         const_CAST_run(c2), CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container(result, result_type);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            result = run_container_create();+            array_run_container_lazy_xor(const_CAST_array(c2),+                                         const_CAST_run(c1), CAST_run(result));+            *result_type = RUN_CONTAINER_TYPE;+            // next line skipped since we are lazy+            // result = convert_run_to_efficient_container(result, result_type);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;  // unreached+    }+}++/**+ * Compute the xor between two containers, with result in the first container.+ * If the returned pointer is identical to c1, then the container has been+ * modified.+ * If the returned pointer is different from c1, then a new container has been+ * created. The original container is freed by container_ixor.+ * The type of the first container may change. Returns the modified (and+ * possibly new) container.+ */+static inline container_t *container_ixor(container_t *c1, uint8_t type1,+                                          const container_t *c2, uint8_t type2,+                                          uint8_t *result_type) {+    c1 = get_writable_copy_if_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type = bitset_bitset_container_ixor(+                               CAST_bitset(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            *result_type = array_array_container_ixor(+                               CAST_array(c1), const_CAST_array(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            *result_type = (uint8_t)run_run_container_ixor(+                CAST_run(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            *result_type = bitset_array_container_ixor(+                               CAST_bitset(c1), const_CAST_array(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            *result_type = array_bitset_container_ixor(+                               CAST_array(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            *result_type = bitset_run_container_ixor(+                               CAST_bitset(c1), const_CAST_run(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;++            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type = run_bitset_container_ixor(+                               CAST_run(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            *result_type = (uint8_t)array_run_container_ixor(+                CAST_array(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            *result_type = (uint8_t)run_array_container_ixor(+                CAST_run(c1), const_CAST_array(c2), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Compute the xor between two containers, with result in the first container.+ * If the returned pointer is identical to c1, then the container has been+ * modified.+ * If the returned pointer is different from c1, then a new container has been+ * created and the caller is responsible for freeing it.+ * The type of the first container may change. Returns the modified+ * (and possibly new) container+ *+ * This lazy version delays some operations such as the maintenance of the+ * cardinality. It requires repair later on the generated containers.+ */+static inline container_t *container_lazy_ixor(container_t *c1, uint8_t type1,+                                               const container_t *c2,+                                               uint8_t type2,+                                               uint8_t *result_type) {+    assert(type1 != SHARED_CONTAINER_TYPE);+    // c1 = get_writable_copy_if_shared(c1,&type1);+    c2 = container_unwrap_shared(c2, &type2);+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            bitset_container_xor_nocard(CAST_bitset(c1), const_CAST_bitset(c2),+                                        CAST_bitset(c1));  // is lazy+            *result_type = BITSET_CONTAINER_TYPE;+            return c1;++        // TODO: other cases being lazy, esp. when we know inplace not likely+        // could see the corresponding code for union+        default:+            // we may have a dirty bitset (without a precomputed cardinality)+            // and calling container_ixor on it might be unsafe.+            if (type1 == BITSET_CONTAINER_TYPE) {+                bitset_container_t *bc = CAST_bitset(c1);+                if (bc->cardinality == BITSET_UNKNOWN_CARDINALITY) {+                    bc->cardinality = bitset_container_compute_cardinality(bc);+                }+            }+            return container_ixor(c1, type1, c2, type2, result_type);+    }+}++/**+ * Compute difference (andnot) between two containers, generate a new+ * container (having type result_type), requires a typecode. This allocates new+ * memory, caller is responsible for deallocation.+ */+static inline container_t *container_andnot(const container_t *c1,+                                            uint8_t type1,+                                            const container_t *c2,+                                            uint8_t type2,+                                            uint8_t *result_type) {+    c1 = container_unwrap_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type =+                bitset_bitset_container_andnot(const_CAST_bitset(c1),+                                               const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            result = array_container_create();+            array_array_container_andnot(+                const_CAST_array(c1), const_CAST_array(c2), CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = array_container_create();+                *result_type = ARRAY_CONTAINER_TYPE;+                return result;+            }+            *result_type = (uint8_t)run_run_container_andnot(+                const_CAST_run(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            *result_type =+                bitset_array_container_andnot(const_CAST_bitset(c1),+                                              const_CAST_array(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            result = array_container_create();+            array_bitset_container_andnot(const_CAST_array(c1),+                                          const_CAST_bitset(c2),+                                          CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(BITSET, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = array_container_create();+                *result_type = ARRAY_CONTAINER_TYPE;+                return result;+            }+            *result_type =+                bitset_run_container_andnot(const_CAST_bitset(c1),+                                            const_CAST_run(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type =+                run_bitset_container_andnot(const_CAST_run(c1),+                                            const_CAST_bitset(c2), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            if (run_container_is_full(const_CAST_run(c2))) {+                result = array_container_create();+                *result_type = ARRAY_CONTAINER_TYPE;+                return result;+            }+            result = array_container_create();+            array_run_container_andnot(const_CAST_array(c1), const_CAST_run(c2),+                                       CAST_array(result));+            *result_type = ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, ARRAY):+            *result_type = (uint8_t)run_array_container_andnot(+                const_CAST_run(c1), const_CAST_array(c2), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;  // unreached+    }+}++/**+ * Compute the andnot between two containers, with result in the first+ * container.+ * If the returned pointer is identical to c1, then the container has been+ * modified.+ * If the returned pointer is different from c1, then a new container has been+ * created. The original container is freed by container_iandnot.+ * The type of the first container may change. Returns the modified (and+ * possibly new) container.+ */+static inline container_t *container_iandnot(container_t *c1, uint8_t type1,+                                             const container_t *c2,+                                             uint8_t type2,+                                             uint8_t *result_type) {+    c1 = get_writable_copy_if_shared(c1, &type1);+    c2 = container_unwrap_shared(c2, &type2);+    container_t *result = NULL;+    switch (PAIR_CONTAINER_TYPES(type1, type2)) {+        case CONTAINER_PAIR(BITSET, BITSET):+            *result_type = bitset_bitset_container_iandnot(+                               CAST_bitset(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, ARRAY):+            array_array_container_iandnot(CAST_array(c1), const_CAST_array(c2));+            *result_type = ARRAY_CONTAINER_TYPE;+            return c1;++        case CONTAINER_PAIR(RUN, RUN):+            *result_type = (uint8_t)run_run_container_iandnot(+                CAST_run(c1), const_CAST_run(c2), &result);+            return result;++        case CONTAINER_PAIR(BITSET, ARRAY):+            *result_type = bitset_array_container_iandnot(+                               CAST_bitset(c1), const_CAST_array(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, BITSET):+            *result_type = ARRAY_CONTAINER_TYPE;+            array_bitset_container_iandnot(CAST_array(c1),+                                           const_CAST_bitset(c2));+            return c1;++        case CONTAINER_PAIR(BITSET, RUN):+            *result_type = bitset_run_container_iandnot(+                               CAST_bitset(c1), const_CAST_run(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(RUN, BITSET):+            *result_type = run_bitset_container_iandnot(+                               CAST_run(c1), const_CAST_bitset(c2), &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;++        case CONTAINER_PAIR(ARRAY, RUN):+            *result_type = ARRAY_CONTAINER_TYPE;+            array_run_container_iandnot(CAST_array(c1), const_CAST_run(c2));+            return c1;++        case CONTAINER_PAIR(RUN, ARRAY):+            *result_type = (uint8_t)run_array_container_iandnot(+                CAST_run(c1), const_CAST_array(c2), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++/**+ * Visit all values x of the container once, passing (base+x,ptr)+ * to iterator. You need to specify a container and its type.+ * Returns true if the iteration should continue.+ */+static inline bool container_iterate(const container_t *c, uint8_t type,+                                     uint32_t base, roaring_iterator iterator,+                                     void *ptr) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_iterate(const_CAST_bitset(c), base,+                                            iterator, ptr);+        case ARRAY_CONTAINER_TYPE:+            return array_container_iterate(const_CAST_array(c), base, iterator,+                                           ptr);+        case RUN_CONTAINER_TYPE:+            return run_container_iterate(const_CAST_run(c), base, iterator,+                                         ptr);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++static inline bool container_iterate64(const container_t *c, uint8_t type,+                                       uint32_t base,+                                       roaring_iterator64 iterator,+                                       uint64_t high_bits, void *ptr) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_iterate64(const_CAST_bitset(c), base,+                                              iterator, high_bits, ptr);+        case ARRAY_CONTAINER_TYPE:+            return array_container_iterate64(const_CAST_array(c), base,+                                             iterator, high_bits, ptr);+        case RUN_CONTAINER_TYPE:+            return run_container_iterate64(const_CAST_run(c), base, iterator,+                                           high_bits, ptr);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++static inline container_t *container_not(const container_t *c, uint8_t type,+                                         uint8_t *result_type) {+    c = container_unwrap_shared(c, &type);+    container_t *result = NULL;+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            *result_type =+                bitset_container_negation(const_CAST_bitset(c), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;+        case ARRAY_CONTAINER_TYPE:+            *result_type = (uint8_t)array_container_negation_range_to_any(+                const_CAST_array(c), 0, 1 << 16, &result);+            return result;+        case RUN_CONTAINER_TYPE:+            *result_type =+                (uint8_t)run_container_negation(const_CAST_run(c), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return NULL;+}++static inline container_t *container_not_range(const container_t *c,+                                               uint8_t type,+                                               uint32_t range_start,+                                               uint32_t range_end,+                                               uint8_t *result_type) {+    c = container_unwrap_shared(c, &type);+    container_t *result = NULL;+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            *result_type =+                bitset_container_negation_range(const_CAST_bitset(c),+                                                range_start, range_end, &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;+        case ARRAY_CONTAINER_TYPE:+            *result_type = (uint8_t)array_container_negation_range_to_any(+                const_CAST_array(c), range_start, range_end, &result);+            return result;+        case RUN_CONTAINER_TYPE:+            *result_type = (uint8_t)run_container_negation_range(+                const_CAST_run(c), range_start, range_end, &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return NULL;+}++static inline container_t *container_inot(container_t *c, uint8_t type,+                                          uint8_t *result_type) {+    c = get_writable_copy_if_shared(c, &type);+    container_t *result = NULL;+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            *result_type =+                bitset_container_negation_inplace(CAST_bitset(c), &result)+                    ? BITSET_CONTAINER_TYPE+                    : ARRAY_CONTAINER_TYPE;+            return result;+        case ARRAY_CONTAINER_TYPE:+            // will never be inplace+            *result_type = (uint8_t)array_container_negation_range_to_any(+                CAST_array(c), 0, 1 << 16, &result);+            array_container_free(CAST_array(c));+            return result;+        case RUN_CONTAINER_TYPE:+            *result_type =+                (uint8_t)run_container_negation_inplace(CAST_run(c), &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return NULL;+}++static inline container_t *container_inot_range(container_t *c, uint8_t type,+                                                uint32_t range_start,+                                                uint32_t range_end,+                                                uint8_t *result_type) {+    c = get_writable_copy_if_shared(c, &type);+    container_t *result = NULL;+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            *result_type = bitset_container_negation_range_inplace(+                               CAST_bitset(c), range_start, range_end, &result)+                               ? BITSET_CONTAINER_TYPE+                               : ARRAY_CONTAINER_TYPE;+            return result;+        case ARRAY_CONTAINER_TYPE:+            *result_type = (uint8_t)array_container_negation_range_to_any(+                CAST_array(c), range_start, range_end, &result);+            array_container_free(CAST_array(c));+            return result;+        case RUN_CONTAINER_TYPE:+            *result_type = (uint8_t)run_container_negation_range_inplace(+                CAST_run(c), range_start, range_end, &result);+            return result;++        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return NULL;+}++/**+ * If the element of given rank is in this container, supposing that+ * the first+ * element has rank start_rank, then the function returns true and+ * sets element+ * accordingly.+ * Otherwise, it returns false and update start_rank.+ */+static inline bool container_select(const container_t *c, uint8_t type,+                                    uint32_t *start_rank, uint32_t rank,+                                    uint32_t *element) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_select(const_CAST_bitset(c), start_rank,+                                           rank, element);+        case ARRAY_CONTAINER_TYPE:+            return array_container_select(const_CAST_array(c), start_rank, rank,+                                          element);+        case RUN_CONTAINER_TYPE:+            return run_container_select(const_CAST_run(c), start_rank, rank,+                                        element);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++static inline uint16_t container_maximum(const container_t *c, uint8_t type) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_maximum(const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_maximum(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_maximum(const_CAST_run(c));+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++static inline uint16_t container_minimum(const container_t *c, uint8_t type) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_minimum(const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_minimum(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_minimum(const_CAST_run(c));+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++// number of values smaller or equal to x+static inline int container_rank(const container_t *c, uint8_t type,+                                 uint16_t x) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_rank(const_CAST_bitset(c), x);+        case ARRAY_CONTAINER_TYPE:+            return array_container_rank(const_CAST_array(c), x);+        case RUN_CONTAINER_TYPE:+            return run_container_rank(const_CAST_run(c), x);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++// bulk version of container_rank(); return number of consumed elements+static inline uint32_t container_rank_many(const container_t *c, uint8_t type,+                                           uint64_t start_rank,+                                           const uint32_t *begin,+                                           const uint32_t *end, uint64_t *ans) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_rank_many(const_CAST_bitset(c), start_rank,+                                              begin, end, ans);+        case ARRAY_CONTAINER_TYPE:+            return array_container_rank_many(const_CAST_array(c), start_rank,+                                             begin, end, ans);+        case RUN_CONTAINER_TYPE:+            return run_container_rank_many(const_CAST_run(c), start_rank, begin,+                                           end, ans);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return 0;+}++// return the index of x, if not exsist return -1+static inline int container_get_index(const container_t *c, uint8_t type,+                                      uint16_t x) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_get_index(const_CAST_bitset(c), x);+        case ARRAY_CONTAINER_TYPE:+            return array_container_get_index(const_CAST_array(c), x);+        case RUN_CONTAINER_TYPE:+            return run_container_get_index(const_CAST_run(c), x);+        default:+            assert(false);+            roaring_unreachable;+    }+    assert(false);+    roaring_unreachable;+    return false;+}++/**+ * Add all values in range [min, max] to a given container.+ *+ * If the returned pointer is different from $container, then a new container+ * has been created and the caller is responsible for freeing it.+ * The type of the first container may change. Returns the modified+ * (and possibly new) container.+ */+static inline container_t *container_add_range(container_t *c, uint8_t type,+                                               uint32_t min, uint32_t max,+                                               uint8_t *result_type) {+    // NB: when selecting new container type, we perform only inexpensive checks+    switch (type) {+        case BITSET_CONTAINER_TYPE: {+            bitset_container_t *bitset = CAST_bitset(c);++            int32_t union_cardinality = 0;+            union_cardinality += bitset->cardinality;+            union_cardinality += max - min + 1;+            union_cardinality -=+                bitset_lenrange_cardinality(bitset->words, min, max - min);++            if (union_cardinality == INT32_C(0x10000)) {+                *result_type = RUN_CONTAINER_TYPE;+                return run_container_create_range(0, INT32_C(0x10000));+            } else {+                *result_type = BITSET_CONTAINER_TYPE;+                bitset_set_lenrange(bitset->words, min, max - min);+                bitset->cardinality = union_cardinality;+                return bitset;+            }+        }+        case ARRAY_CONTAINER_TYPE: {+            array_container_t *array = CAST_array(c);++            int32_t nvals_greater =+                count_greater(array->array, array->cardinality, (uint16_t)max);+            int32_t nvals_less =+                count_less(array->array, array->cardinality - nvals_greater,+                           (uint16_t)min);+            int32_t union_cardinality =+                nvals_less + (max - min + 1) + nvals_greater;++            if (union_cardinality == INT32_C(0x10000)) {+                *result_type = RUN_CONTAINER_TYPE;+                return run_container_create_range(0, INT32_C(0x10000));+            } else if (union_cardinality <= DEFAULT_MAX_SIZE) {+                *result_type = ARRAY_CONTAINER_TYPE;+                array_container_add_range_nvals(array, min, max, nvals_less,+                                                nvals_greater);+                return array;+            } else {+                *result_type = BITSET_CONTAINER_TYPE;+                bitset_container_t *bitset = bitset_container_from_array(array);+                bitset_set_lenrange(bitset->words, min, max - min);+                bitset->cardinality = union_cardinality;+                return bitset;+            }+        }+        case RUN_CONTAINER_TYPE: {+            run_container_t *run = CAST_run(c);++            int32_t nruns_greater =+                rle16_count_greater(run->runs, run->n_runs, (uint16_t)max);+            int32_t nruns_less = rle16_count_less(+                run->runs, run->n_runs - nruns_greater, (uint16_t)min);++            int32_t run_size_bytes =+                (nruns_less + 1 + nruns_greater) * sizeof(rle16_t);+            int32_t bitset_size_bytes =+                BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);++            if (run_size_bytes <= bitset_size_bytes) {+                run_container_add_range_nruns(run, min, max, nruns_less,+                                              nruns_greater);+                *result_type = RUN_CONTAINER_TYPE;+                return run;+            } else {+                return container_from_run_range(run, min, max, result_type);+            }+        }+        default:+            roaring_unreachable;+    }+}++/*+ * Removes all elements in range [min, max].+ * Returns one of:+ *   - NULL if no elements left+ *   - pointer to the original container+ *   - pointer to a newly-allocated container (if it is more efficient)+ *+ * If the returned pointer is different from $container, then a new container+ * has been created and the caller is responsible for freeing the original+ * container.+ */+static inline container_t *container_remove_range(container_t *c, uint8_t type,+                                                  uint32_t min, uint32_t max,+                                                  uint8_t *result_type) {+    switch (type) {+        case BITSET_CONTAINER_TYPE: {+            bitset_container_t *bitset = CAST_bitset(c);++            int32_t result_cardinality =+                bitset->cardinality -+                bitset_lenrange_cardinality(bitset->words, min, max - min);++            if (result_cardinality == 0) {+                return NULL;+            } else if (result_cardinality <= DEFAULT_MAX_SIZE) {+                *result_type = ARRAY_CONTAINER_TYPE;+                bitset_reset_range(bitset->words, min, max + 1);+                bitset->cardinality = result_cardinality;+                return array_container_from_bitset(bitset);+            } else {+                *result_type = BITSET_CONTAINER_TYPE;+                bitset_reset_range(bitset->words, min, max + 1);+                bitset->cardinality = result_cardinality;+                return bitset;+            }+        }+        case ARRAY_CONTAINER_TYPE: {+            array_container_t *array = CAST_array(c);++            int32_t nvals_greater =+                count_greater(array->array, array->cardinality, (uint16_t)max);+            int32_t nvals_less =+                count_less(array->array, array->cardinality - nvals_greater,+                           (uint16_t)min);+            int32_t result_cardinality = nvals_less + nvals_greater;++            if (result_cardinality == 0) {+                return NULL;+            } else {+                *result_type = ARRAY_CONTAINER_TYPE;+                array_container_remove_range(+                    array, nvals_less, array->cardinality - result_cardinality);+                return array;+            }+        }+        case RUN_CONTAINER_TYPE: {+            run_container_t *run = CAST_run(c);++            if (run->n_runs == 0) {+                return NULL;+            }+            if (min <= run_container_minimum(run) &&+                max >= run_container_maximum(run)) {+                return NULL;+            }++            run_container_remove_range(run, min, max);+            return convert_run_to_efficient_container(run, result_type);+        }+        default:+            roaring_unreachable;+    }+}++#ifdef __cplusplus+using api::roaring_container_iterator_t;+#endif++/**+ * Initializes the iterator at the first entry in the container.+ */+roaring_container_iterator_t container_init_iterator(const container_t *c,+                                                     uint8_t typecode,+                                                     uint16_t *value);++/**+ * Initializes the iterator at the last entry in the container.+ */+roaring_container_iterator_t container_init_iterator_last(const container_t *c,+                                                          uint8_t typecode,+                                                          uint16_t *value);++/**+ * Moves the iterator to the next entry. Returns true and sets `value` if a+ * value is present.+ */+CROARING_ALLOW_UNALIGNED+inline bool container_iterator_next(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            it->index++;++            uint32_t wordindex = it->index / 64;+            if (wordindex >= BITSET_CONTAINER_SIZE_IN_WORDS) {+                return false;+            }++            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX << (it->index % 64));+            // next part could be optimized/simplified+            while (word == 0 &&+                   (wordindex + 1 < BITSET_CONTAINER_SIZE_IN_WORDS)) {+                wordindex++;+                word = bc->words[wordindex];+            }+            if (word != 0) {+                it->index = wordindex * 64 + roaring_trailing_zeroes(word);+                *value = it->index;+                return true;+            }+            return false;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            it->index++;+            if (it->index < ac->cardinality) {+                *value = ac->array[it->index];+                return true;+            }+            return false;+        }+        case RUN_CONTAINER_TYPE: {+            if (*value == UINT16_MAX) {  // Avoid overflow to zero+                return false;+            }++            const run_container_t *rc = const_CAST_run(c);+            uint32_t limit =+                rc->runs[it->index].value + rc->runs[it->index].length;+            if (*value < limit) {+                (*value)++;+                return true;+            }++            it->index++;+            if (it->index < rc->n_runs) {+                *value = rc->runs[it->index].value;+                return true;+            }+            return false;+        }+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++/**+ * Moves the iterator to the previous entry. Returns true and sets `value` if a+ * value is present.+ */+CROARING_ALLOW_UNALIGNED+inline bool container_iterator_prev(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            if (--it->index < 0) {+                return false;+            }++            const bitset_container_t *bc = const_CAST_bitset(c);+            int32_t wordindex = it->index / 64;+            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX >> (63 - (it->index % 64)));++            while (word == 0 && --wordindex >= 0) {+                word = bc->words[wordindex];+            }+            if (word == 0) {+                return false;+            }++            it->index = (wordindex * 64) + (63 - roaring_leading_zeroes(word));+            *value = it->index;+            return true;+        }+        case ARRAY_CONTAINER_TYPE: {+            if (--it->index < 0) {+                return false;+            }+            const array_container_t *ac = const_CAST_array(c);+            *value = ac->array[it->index];+            return true;+        }+        case RUN_CONTAINER_TYPE: {+            if (*value == 0) {+                return false;+            }++            const run_container_t *rc = const_CAST_run(c);+            (*value)--;+            if (*value >= rc->runs[it->index].value) {+                return true;+            }++            if (--it->index < 0) {+                return false;+            }++            *value = rc->runs[it->index].value + rc->runs[it->index].length;+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++/**+ * Moves the iterator to the smallest entry that is greater than or equal to+ * `val`. Returns true and sets `value_out` if a value is present. `value_out`+ * should be initialized to a value.+ */+bool container_iterator_lower_bound(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value_out, uint16_t val);++/**+ * Reads up to `count` entries from the container, and writes them into `buf`+ * as `high16 | entry`. Returns true and sets `value_out` if a value is present+ * after reading the entries. Sets `consumed` to the number of values read.+ * `count` should be greater than zero.+ */+bool container_iterator_read_into_uint32(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint32_t high16, uint32_t *buf,+                                         uint32_t count, uint32_t *consumed,+                                         uint16_t *value_out);++/**+ * Reads up to `count` entries from the container, and writes them into `buf`+ * as `high48 | entry`. Returns true and sets `value_out` if a value is present+ * after reading the entries. Sets `consumed` to the number of values read.+ * `count` should be greater than zero.+ */+bool container_iterator_read_into_uint64(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint64_t high48, uint64_t *buf,+                                         uint32_t count, uint32_t *consumed,+                                         uint16_t *value_out);++/**+ * Reads up to `count` entries backward from the container, writing them into+ * `buf` as `high16 | entry` in descending order. Returns true and sets+ * `value_out` if a value is present before the entries read. Sets `consumed`+ * to the number of values read. `count` should be greater than zero.+ *+ * `value_out` must be initialized to the current value yielded by the iterator.+ */+bool container_iterator_read_backward_into_uint32(+    const container_t *c, uint8_t typecode, roaring_container_iterator_t *it,+    uint32_t high16, uint32_t *buf, uint32_t count, uint32_t *consumed,+    uint16_t *value_out);++/**+ * Reads up to `count` entries backward from the container, writing them into+ * `buf` as `high48 | entry` in descending order. Returns true and sets+ * `value_out` if a value is present before the entries read. Sets `consumed`+ * to the number of values read. `count` should be greater than zero.+ *+ * `value_out` must be initialized to the current value yielded by the iterator.+ */+bool container_iterator_read_backward_into_uint64(+    const container_t *c, uint8_t typecode, roaring_container_iterator_t *it,+    uint64_t high48, uint64_t *buf, uint32_t count, uint32_t *consumed,+    uint16_t *value_out);++/**+ * Skips the next `skip_count` entries in the container iterator. Returns true+ * and sets `value_out` if a value is present after skipping. Returns false if+ * the end of the container is reached during the skip operation. Sets+ * consumed_count to the number of values actually skipped (which may be less+ * than skip_count if the end of the container is reached).+ *+ * value_out must be initialized to the previous value yielded by the iterator.+ *+ * skip_count must be greater than zero.+ */+bool container_iterator_skip(const container_t *c, uint8_t typecode,+                             roaring_container_iterator_t *it,+                             uint32_t skip_count, uint32_t *consumed_count,+                             uint16_t *value_out);++/**+ * Skips the previous `skip_count` entries in the container iterator (moves+ * backwards). Returns true and sets `value_out` if a value is present after+ * skipping backwards. Returns false if the beginning of the container is+ * reached during the skip operation. Sets consumed_count to the number of+ * values actually skipped backwards (which may be less than skip_count if+ * the beginning of the container is reached).+ *+ * value_out must be initialized to the current value yielded by the iterator.+ *+ * skip_count must be greater than zero.+ */+bool container_iterator_skip_backward(const container_t *c, uint8_t typecode,+                                      roaring_container_iterator_t *it,+                                      uint32_t skip_count,+                                      uint32_t *consumed_count,+                                      uint16_t *value_out);++/**+ * Finds the end of the consecutive run starting at the current iterator+ * position within a container. Returns the low16 of the last consecutive+ * value. If there are more values in the container after the run,+ * *has_more is set to true, the iterator is positioned at the next value,+ * and *value is updated to that value. Otherwise *has_more is set to false.+ *+ * *value must be the low 16 bits of the current value at the iterator's+ * position on entry.+ */+uint16_t container_iterator_find_run_end(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint16_t *value, bool *has_more);++/**+ * Finds the start of the consecutive run ending at the current iterator+ * position within a container. Returns the low16 of the first consecutive+ * value. If there are more values in the container before the run,+ * *has_more is set to true, the iterator is positioned at the previous value,+ * and *value is updated to that value. Otherwise *has_more is set to false.+ *+ * *value must be the low 16 bits of the current value at the iterator's+ * position on entry.+ */+uint16_t container_iterator_find_run_start(const container_t *c,+                                           uint8_t typecode,+                                           roaring_container_iterator_t *it,+                                           uint16_t *value, bool *has_more);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif
+ cbits/CRoaring/include/roaring/containers/convert.h view
@@ -0,0 +1,82 @@+/*+ * convert.h+ *+ * This header declares conversion helpers between the different Roaring+ * container representations: array, bitset, and run containers. These+ * routines are used when an operation produces data better represented in a+ * different form, or when the library wants to switch to the most space-+ * efficient container type.+ *+ * In addition to direct conversions, the file also provides helpers that+ * choose between candidate result representations based on cardinality and+ * storage efficiency.+ */++#ifndef INCLUDE_CONTAINERS_CONVERT_H_+#define INCLUDE_CONTAINERS_CONVERT_H_++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Convert an array into a bitset. The input container is not freed or modified.+ */+bitset_container_t *bitset_container_from_array(const array_container_t *arr);++/* Convert a run into a bitset. The input container is not freed or modified. */+bitset_container_t *bitset_container_from_run(const run_container_t *arr);++/* Convert a run into an array. The input container is not freed or modified. */+array_container_t *array_container_from_run(const run_container_t *arr);++/* Convert a bitset into an array. The input container is not freed or modified.+ */+array_container_t *array_container_from_bitset(const bitset_container_t *bits);++/* Convert an array into a run. The input container is not freed or modified.+ */+run_container_t *run_container_from_array(const array_container_t *c);++/* convert a run into either an array or a bitset+ * might free the container. This does not free the input run container. */+container_t *convert_to_bitset_or_array_container(run_container_t *rc,+                                                  int32_t card,+                                                  uint8_t *resulttype);++/* convert containers to and from runcontainers, as is most space efficient.+ * The container might be freed. */+container_t *convert_run_optimize(container_t *c, uint8_t typecode_original,+                                  uint8_t *typecode_after);++/* converts a run container to either an array or a bitset, IF it saves space.+ */+/* If a conversion occurs, the caller is responsible to free the original+ * container and+ * he becomes reponsible to free the new one. */+container_t *convert_run_to_efficient_container(run_container_t *c,+                                                uint8_t *typecode_after);++// like convert_run_to_efficient_container but frees the old result if needed+container_t *convert_run_to_efficient_container_and_free(+    run_container_t *c, uint8_t *typecode_after);++/**+ * Create new container which is a union of run container and+ * range [min, max]. Caller is responsible for freeing run container.+ */+container_t *container_from_run_range(const run_container_t *run, uint32_t min,+                                      uint32_t max, uint8_t *typecode_after);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_CONVERT_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_andnot.h view
@@ -0,0 +1,191 @@+/*+ * mixed_andnot.h+ *+ * This header declares mixed-container difference operations of the form+ * `A \ B` (also called andnot) between Roaring container types such as array,+ * bitset, and run containers. These helpers are used when the operands have+ * different internal representations and the result may need to change+ * representation depending on density.+ *+ * The file includes both allocating and inplace-oriented variants so callers+ * can either materialize a fresh result or reuse storage when that is+ * efficient and semantically allowed.+ */+#ifndef INCLUDE_CONTAINERS_MIXED_ANDNOT_H_+#define INCLUDE_CONTAINERS_MIXED_ANDNOT_H_++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst, a valid array container that could be the same as dst.*/+void array_bitset_container_andnot(const array_container_t *src_1,+                                   const bitset_container_t *src_2,+                                   array_container_t *dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * src_1 */++void array_bitset_container_iandnot(array_container_t *src_1,+                                    const bitset_container_t *src_2);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst, which does not initially have a valid container.+ * Return true for a bitset result; false for array+ */++bool bitset_array_container_andnot(const bitset_container_t *src_1,+                                   const array_container_t *src_2,+                                   container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_array_container_iandnot(bitset_container_t *src_1,+                                    const array_container_t *src_2,+                                    container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_andnot(const run_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_iandnot(run_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset").  dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool bitset_run_container_andnot(const bitset_container_t *src_1,+                                 const run_container_t *src_2,+                                 container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_run_container_iandnot(bitset_container_t *src_1,+                                  const run_container_t *src_2,+                                  container_t **dst);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any type of container.+ */++int run_array_container_andnot(const run_container_t *src_1,+                               const array_container_t *src_2,+                               container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++int run_array_container_iandnot(run_container_t *src_1,+                                const array_container_t *src_2,+                                container_t **dst);++/* dst must be a valid array container, allowed to be src_1 */++void array_run_container_andnot(const array_container_t *src_1,+                                const run_container_t *src_2,+                                array_container_t *dst);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++void array_run_container_iandnot(array_container_t *src_1,+                                 const run_container_t *src_2);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int run_run_container_andnot(const run_container_t *src_1,+                             const run_container_t *src_2, container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++int run_run_container_iandnot(run_container_t *src_1,+                              const run_container_t *src_2, container_t **dst);++/*+ * dst is a valid array container and may be the same as src_1+ */++void array_array_container_andnot(const array_container_t *src_1,+                                  const array_container_t *src_2,+                                  array_container_t *dst);++/* inplace array-array andnot will always be able to reuse the space of+ * src_1 */+void array_array_container_iandnot(array_container_t *src_1,+                                   const array_container_t *src_2);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially). Return value is+ * "dst is a bitset"+ */++bool bitset_bitset_container_andnot(const bitset_container_t *src_1,+                                    const bitset_container_t *src_2,+                                    container_t **dst);++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_bitset_container_iandnot(bitset_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     container_t **dst);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif
+ cbits/CRoaring/include/roaring/containers/mixed_equal.h view
@@ -0,0 +1,42 @@+/*+ * mixed_equal.h+ *+ */++#ifndef CONTAINERS_MIXED_EQUAL_H_+#define CONTAINERS_MIXED_EQUAL_H_++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/**+ * Return true if the two containers have the same content.+ */+bool array_container_equal_bitset(const array_container_t* container1,+                                  const bitset_container_t* container2);++/**+ * Return true if the two containers have the same content.+ */+bool run_container_equals_array(const run_container_t* container1,+                                const array_container_t* container2);+/**+ * Return true if the two containers have the same content.+ */+bool run_container_equals_bitset(const run_container_t* container1,+                                 const bitset_container_t* container2);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* CONTAINERS_MIXED_EQUAL_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_intersection.h view
@@ -0,0 +1,110 @@+/*+ * mixed_intersection.h+ *+ * This header declares intersection operations between different Roaring+ * container types, such as array, bitset, and run containers. These mixed+ * routines are used when the input containers have different internal+ * representations and the implementation must choose the appropriate result+ * form based on the data.+ *+ * In addition to materializing intersections, the file also provides helpers+ * for intersection cardinality, intersection predicates, and selected inplace+ * variants.+ */++#ifndef INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_+#define INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_++/* These functions appear to exclude cases where the+ * inputs have the same type and the output is guaranteed+ * to have the same type as the inputs.  Eg, array intersection+ */++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the intersection of src_1 and src_2 and write the result to+ * dst. It is allowed for dst to be equal to src_1. We assume that dst is a+ * valid container. */+void array_bitset_container_intersection(const array_container_t *src_1,+                                         const bitset_container_t *src_2,+                                         array_container_t *dst);++/* Compute the size of the intersection of src_1 and src_2. */+int array_bitset_container_intersection_cardinality(+    const array_container_t *src_1, const bitset_container_t *src_2);++/* Checking whether src_1 and src_2 intersect. */+bool array_bitset_container_intersect(const array_container_t *src_1,+                                      const bitset_container_t *src_2);++/*+ * Compute the intersection between src_1 and src_2 and write the result+ * to *dst. If the return function is true, the result is a bitset_container_t+ * otherwise is a array_container_t. We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool bitset_bitset_container_intersection(const bitset_container_t *src_1,+                                          const bitset_container_t *src_2,+                                          container_t **dst);++/* Compute the intersection between src_1 and src_2 and write the result to+ * dst. It is allowed for dst to be equal to src_1. We assume that dst is a+ * valid container. */+void array_run_container_intersection(const array_container_t *src_1,+                                      const run_container_t *src_2,+                                      array_container_t *dst);++/* Compute the intersection between src_1 and src_2 and write the result to+ * *dst. If the result is true then the result is a bitset_container_t+ * otherwise is a array_container_t.+ * If *dst == src_2, then an in-place intersection is attempted+ **/+bool run_bitset_container_intersection(const run_container_t *src_1,+                                       const bitset_container_t *src_2,+                                       container_t **dst);++/* Compute the size of the intersection between src_1 and src_2 . */+int array_run_container_intersection_cardinality(const array_container_t *src_1,+                                                 const run_container_t *src_2);++/* Compute the size of the intersection  between src_1 and src_2+ **/+int run_bitset_container_intersection_cardinality(+    const run_container_t *src_1, const bitset_container_t *src_2);++/* Check that src_1 and src_2 intersect. */+bool array_run_container_intersect(const array_container_t *src_1,+                                   const run_container_t *src_2);++/* Check that src_1 and src_2 intersect.+ **/+bool run_bitset_container_intersect(const run_container_t *src_1,+                                    const bitset_container_t *src_2);++/*+ * Same as bitset_bitset_container_intersection except that if the output is to+ * be a+ * bitset_container_t, then src_1 is modified and no allocation is made.+ * If the output is to be an array_container_t, then caller is responsible+ * to free the container.+ * In all cases, the result is in *dst.+ */+bool bitset_bitset_container_intersection_inplace(+    bitset_container_t *src_1, const bitset_container_t *src_2,+    container_t **dst);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_negation.h view
@@ -0,0 +1,160 @@+/*+ * mixed_negation.h+ *+ * This header declares negation (complement) operations for Roaring+ * containers, both over the full 16-bit container domain and over specified+ * subranges. Depending on the input representation and the density of the+ * complement, the result may need to switch between array, bitset, and run+ * containers.+ *+ * The file includes both allocating and inplace-oriented variants so callers+ * can choose between simple result construction and reuse of an existing+ * container when that is practical.+ */++#ifndef INCLUDE_CONTAINERS_MIXED_NEGATION_H_+#define INCLUDE_CONTAINERS_MIXED_NEGATION_H_++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Negation across the entire range of the container.+ * Compute the  negation of src  and write the result+ * to *dst. The complement of a+ * sufficiently sparse set will always be dense and a hence a bitmap+ * We assume that dst is pre-allocated and a valid bitset container+ * There can be no in-place version.+ */+void array_container_negation(const array_container_t *src,+                              bitset_container_t *dst);++/* Negation across the entire range of the container+ * Compute the  negation of src  and write the result+ * to *dst.  A true return value indicates a bitset result,+ * otherwise the result is an array container.+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool bitset_container_negation(const bitset_container_t *src,+                               container_t **dst);++/* inplace version */+/*+ * Same as bitset_container_negation except that if the output is to+ * be a+ * bitset_container_t, then src is modified and no allocation is made.+ * If the output is to be an array_container_t, then caller is responsible+ * to free the container.+ * In all cases, the result is in *dst.+ */+bool bitset_container_negation_inplace(bitset_container_t *src,+                                       container_t **dst);++/* Negation across the entire range of container+ * Compute the  negation of src  and write the result+ * to *dst.+ * Return values are the *_TYPECODES as defined * in containers.h+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+int run_container_negation(const run_container_t *src, container_t **dst);++/*+ * Same as run_container_negation except that if the output is to+ * be a+ * run_container_t, and has the capacity to hold the result,+ * then src is modified and no allocation is made.+ * In all cases, the result is in *dst.+ */+int run_container_negation_inplace(run_container_t *src, container_t **dst);++/* Negation across a range of the container.+ * Compute the  negation of src  and write the result+ * to *dst. Returns true if the result is a bitset container+ * and false for an array container.  *dst is not preallocated.+ */+bool array_container_negation_range(const array_container_t *src,+                                    const int range_start, const int range_end,+                                    container_t **dst);++/* Even when the result would fit, it is unclear how to make an+ * inplace version without inefficient copying.  Thus this routine+ * may be a wrapper for the non-in-place version+ */+bool array_container_negation_range_inplace(array_container_t *src,+                                            const int range_start,+                                            const int range_end,+                                            container_t **dst);++/* Negation across a range of the container.+ * Compute the negation of src and write the result to *dst, which is not+ * preallocated. Unlike array_container_negation_range, the result may be a+ * run container: the negation of a sparse array over a large range has few+ * runs. Returns the typecode of the result.+ */+int array_container_negation_range_to_any(const array_container_t *src,+                                          const int range_start,+                                          const int range_end,+                                          container_t **dst);++/* Negation across a range of the container+ * Compute the  negation of src  and write the result+ * to *dst.  A true return value indicates a bitset result,+ * otherwise the result is an array container.+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool bitset_container_negation_range(const bitset_container_t *src,+                                     const int range_start, const int range_end,+                                     container_t **dst);++/* inplace version */+/*+ * Same as bitset_container_negation except that if the output is to+ * be a+ * bitset_container_t, then src is modified and no allocation is made.+ * If the output is to be an array_container_t, then caller is responsible+ * to free the container.+ * In all cases, the result is in *dst.+ */+bool bitset_container_negation_range_inplace(bitset_container_t *src,+                                             const int range_start,+                                             const int range_end,+                                             container_t **dst);++/* Negation across a range of container+ * Compute the  negation of src  and write the result+ * to *dst.  Return values are the *_TYPECODES as defined * in containers.h+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+int run_container_negation_range(const run_container_t *src,+                                 const int range_start, const int range_end,+                                 container_t **dst);++/*+ * Same as run_container_negation except that if the output is to+ * be a+ * run_container_t, and has the capacity to hold the result,+ * then src is modified and no allocation is made.+ * In all cases, the result is in *dst.+ */+int run_container_negation_range_inplace(run_container_t *src,+                                         const int range_start,+                                         const int range_end,+                                         container_t **dst);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_MIXED_NEGATION_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_subset.h view
@@ -0,0 +1,61 @@+/*+ * mixed_subset.h+ *+ * This header declares subset-checking routines between different Roaring+ * container types. These helpers are used when two containers do not share the+ * same representation and a direct type-specific subset predicate is needed.+ *+ * Each function answers whether all values from one container are contained in+ * another, across combinations of array, bitset, and run containers.+ */++#ifndef CONTAINERS_MIXED_SUBSET_H_+#define CONTAINERS_MIXED_SUBSET_H_++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/**+ * Return true if container1 is a subset of container2.+ */+bool array_container_is_subset_bitset(const array_container_t* container1,+                                      const bitset_container_t* container2);++/**+ * Return true if container1 is a subset of container2.+ */+bool run_container_is_subset_array(const run_container_t* container1,+                                   const array_container_t* container2);++/**+ * Return true if container1 is a subset of container2.+ */+bool array_container_is_subset_run(const array_container_t* container1,+                                   const run_container_t* container2);++/**+ * Return true if container1 is a subset of container2.+ */+bool run_container_is_subset_bitset(const run_container_t* container1,+                                    const bitset_container_t* container2);++/**+ * Return true if container1 is a subset of container2.+ */+bool bitset_container_is_subset_run(const bitset_container_t* container1,+                                    const run_container_t* container2);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* CONTAINERS_MIXED_SUBSET_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_union.h view
@@ -0,0 +1,125 @@+/*+ * mixed_union.h+ *+ * This header declares union operations between different Roaring container+ * types, such as array, bitset, and run containers. These mixed-operation+ * helpers are used when the two input containers do not share the same+ * representation and the result may need to stay in, or be converted to, a+ * representation chosen according to the data.+ *+ * The file includes regular, lazy, and inplace variants so callers can select+ * between fully maintained results and faster deferred-maintenance paths.+ */++#ifndef INCLUDE_CONTAINERS_MIXED_UNION_H_+#define INCLUDE_CONTAINERS_MIXED_UNION_H_++/* These functions appear to exclude cases where the+ * inputs have the same type and the output is guaranteed+ * to have the same type as the inputs.  Eg, bitset unions+ */++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the union of src_1 and src_2 and write the result to+ * dst. It is allowed for src_2 to be dst.   */+void array_bitset_container_union(const array_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  bitset_container_t *dst);++/* Compute the union of src_1 and src_2 and write the result to+ * dst. It is allowed for src_2 to be dst.  This version does not+ * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY). */+void array_bitset_container_lazy_union(const array_container_t *src_1,+                                       const bitset_container_t *src_2,+                                       bitset_container_t *dst);++/*+ * Compute the union between src_1 and src_2 and write the result+ * to *dst. If the return function is true, the result is a bitset_container_t+ * otherwise is a array_container_t. We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool array_array_container_union(const array_container_t *src_1,+                                 const array_container_t *src_2,+                                 container_t **dst);++/*+ * Compute the union between src_1 and src_2 and write the result+ * to *dst if it cannot be written to src_1. If the return function is true,+ * the result is a bitset_container_t+ * otherwise is a array_container_t. When the result is an array_container_t, it+ * it either written to src_1 (if *dst is null) or to *dst.+ * If the result is a bitset_container_t and *dst is null, then there was a+ * failure.+ */+bool array_array_container_inplace_union(array_container_t *src_1,+                                         const array_container_t *src_2,+                                         container_t **dst);++/*+ * Same as array_array_container_union except that it will more eagerly produce+ * a bitset.+ */+bool array_array_container_lazy_union(const array_container_t *src_1,+                                      const array_container_t *src_2,+                                      container_t **dst);++/*+ * Same as array_array_container_inplace_union except that it will more eagerly+ * produce a bitset.+ */+bool array_array_container_lazy_inplace_union(array_container_t *src_1,+                                              const array_container_t *src_2,+                                              container_t **dst);++/* Compute the union of src_1 and src_2 and write the result to+ * dst. We assume that dst is a+ * valid container. The result might need to be further converted to array or+ * bitset container,+ * the caller is responsible for the eventual conversion. */+void array_run_container_union(const array_container_t *src_1,+                               const run_container_t *src_2,+                               run_container_t *dst);++/* Compute the union of src_1 and src_2 and write the result to+ * src2. The result might need to be further converted to array or+ * bitset container,+ * the caller is responsible for the eventual conversion. */+void array_run_container_inplace_union(const array_container_t *src_1,+                                       run_container_t *src_2);++/* Compute the union of src_1 and src_2 and write the result to+ * dst. It is allowed for dst to be src_2.+ * If run_container_is_full(src_1) is true, you must not be calling this+ *function.+ **/+void run_bitset_container_union(const run_container_t *src_1,+                                const bitset_container_t *src_2,+                                bitset_container_t *dst);++/* Compute the union of src_1 and src_2 and write the result to+ * dst. It is allowed for dst to be src_2.  This version does not+ * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY).+ * If run_container_is_full(src_1) is true, you must not be calling this+ * function.+ * */+void run_bitset_container_lazy_union(const run_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     bitset_container_t *dst);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_MIXED_UNION_H_ */
+ cbits/CRoaring/include/roaring/containers/mixed_xor.h view
@@ -0,0 +1,185 @@+/*+ * mixed_xor.h+ *+ * This header declares XOR operations between different Roaring container+ * types, such as array, bitset, and run containers. These "mixed" routines+ * handle cases where the two inputs do not share the same representation and+ * where the most appropriate output representation may depend on the data.+ *+ * It includes regular, lazy, and inplace variants so higher-level bitmap code+ * can choose between fully normalized results and faster deferred-maintenance+ * paths.+ */++#ifndef INCLUDE_CONTAINERS_MIXED_XOR_H_+#define INCLUDE_CONTAINERS_MIXED_XOR_H_++/* These functions appear to exclude cases where the+ * inputs have the same type and the output is guaranteed+ * to have the same type as the inputs.  Eg, bitset unions+ */++/*+ * Java implementation (as of May 2016) for array_run, run_run+ * and  bitset_run don't do anything different for inplace.+ * (They are not truly in place.)+ */++#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>++// #include "containers.h"++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially).+ * Result is true iff dst is a bitset  */+bool array_bitset_container_xor(const array_container_t *src_1,+                                const bitset_container_t *src_2,+                                container_t **dst);++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. It is allowed for src_2 to be dst.  This version does not+ * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY).+ */++void array_bitset_container_lazy_xor(const array_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     bitset_container_t *dst);+/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially). Return value is+ * "dst is a bitset"+ */++bool bitset_bitset_container_xor(const bitset_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst);++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_xor(const run_container_t *src_1,+                              const bitset_container_t *src_2,+                              container_t **dst);++/* lazy xor.  Dst is initialized and may be equal to src_2.+ *  Result is left as a bitset container, even if actual+ *  cardinality would dictate an array container.+ */++void run_bitset_container_lazy_xor(const run_container_t *src_1,+                                   const bitset_container_t *src_2,+                                   bitset_container_t *dst);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int array_run_container_xor(const array_container_t *src_1,+                            const run_container_t *src_2, container_t **dst);++/* dst does not initially have a valid container.  Creates either+ * an array or a bitset container, indicated by return code+ */++bool array_array_container_xor(const array_container_t *src_1,+                               const array_container_t *src_2,+                               container_t **dst);++/* dst does not initially have a valid container.  Creates either+ * an array or a bitset container, indicated by return code.+ * A bitset container will not have a valid cardinality and the+ * container type might not be correct for the actual cardinality+ */++bool array_array_container_lazy_xor(const array_container_t *src_1,+                                    const array_container_t *src_2,+                                    container_t **dst);++/* Dst is a valid run container. (Can it be src_2? Let's say not.)+ * Leaves result as run container, even if other options are+ * smaller.+ */++void array_run_container_lazy_xor(const array_container_t *src_1,+                                  const run_container_t *src_2,+                                  run_container_t *dst);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int run_run_container_xor(const run_container_t *src_1,+                          const run_container_t *src_2, container_t **dst);++/* INPLACE versions (initial implementation may not exploit all inplace+ * opportunities (if any...)+ */++/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_array_container_ixor(bitset_container_t *src_1,+                                 const array_container_t *src_2,+                                 container_t **dst);++bool bitset_bitset_container_ixor(bitset_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  container_t **dst);++bool array_bitset_container_ixor(array_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst);++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_ixor(run_container_t *src_1,+                               const bitset_container_t *src_2,+                               container_t **dst);++bool bitset_run_container_ixor(bitset_container_t *src_1,+                               const run_container_t *src_2, container_t **dst);++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int array_run_container_ixor(array_container_t *src_1,+                             const run_container_t *src_2, container_t **dst);++int run_array_container_ixor(run_container_t *src_1,+                             const array_container_t *src_2, container_t **dst);++bool array_array_container_ixor(array_container_t *src_1,+                                const array_container_t *src_2,+                                container_t **dst);++int run_run_container_ixor(run_container_t *src_1, const run_container_t *src_2,+                           container_t **dst);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif
+ cbits/CRoaring/include/roaring/containers/perfparameters.h view
@@ -0,0 +1,60 @@+/*+ * perfparameters.h+ *+ * This header centralizes a small set of performance-tuning constants and+ * heuristic defaults used by CRoaring container code. These parameters control+ * decisions such as initial container sizing and when lazy or eager operations+ * may convert between container representations.+ *+ * In practice, these values encode trade-offs between memory use, allocation+ * overhead, and execution speed for common workloads.+ */+#ifndef PERFPARAMETERS_H_+#define PERFPARAMETERS_H_++#include <stdbool.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/**+During lazy computations, we can transform array containers into bitset+containers as+long as we can expect them to have  ARRAY_LAZY_LOWERBOUND values.+*/+enum { ARRAY_LAZY_LOWERBOUND = 1024 };++/* default initial size of a run container+   setting it to zero delays the malloc.*/+enum { RUN_DEFAULT_INIT_SIZE = 0 };++/* default initial size of an array container+   setting it to zero delays the malloc */+enum { ARRAY_DEFAULT_INIT_SIZE = 0 };++/* automatic bitset conversion during lazy or */+#ifndef CROARING_LAZY_OR_BITSET_CONVERSION+#define CROARING_LAZY_OR_BITSET_CONVERSION true+#endif++/* automatically attempt to convert a bitset to a full run during lazy+ * evaluation */+#ifndef CROARING_LAZY_OR_BITSET_CONVERSION_TO_FULL+#define CROARING_LAZY_OR_BITSET_CONVERSION_TO_FULL true+#endif++/* automatically attempt to convert a bitset to a full run */+#ifndef CROARING_OR_BITSET_CONVERSION_TO_FULL+#define CROARING_OR_BITSET_CONVERSION_TO_FULL true+#endif++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif
+ cbits/CRoaring/include/roaring/containers/run.h view
@@ -0,0 +1,770 @@+/*+ * run.h+ *+ * Run containers store a set of 16-bit integers as a sorted array of+ * non-overlapping runs. Each run is represented by a starting value and a+ * length, encoding one contiguous interval of present integers.+ *+ * This representation is effective when the data contains long consecutive+ * ranges because it compresses many adjacent values into a small number of+ * run records while still supporting search and set operations over the+ * interval list.+ */++#ifndef INCLUDE_CONTAINERS_RUN_H_+#define INCLUDE_CONTAINERS_RUN_H_++#include <roaring/roaring_types.h>  // roaring_iterator++// Include other headers after roaring_types.h+#include <assert.h>+#include <stdbool.h>+#include <stdint.h>+#include <string.h>++#include <roaring/array_util.h>  // binarySearch()/memequals() for inlining+#include <roaring/containers/container_defs.h>  // container_t, perfparameters+#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {++// Note: in pure C++ code, you should avoid putting `using` in header files+using api::roaring_iterator;+using api::roaring_iterator64;++namespace internal {+#endif++/* struct rle16_s - run length pair+ *+ * @value:  start position of the run+ * @length: length of the run is `length + 1`+ *+ * An RLE pair {v, l} would represent the integers between the interval+ * [v, v+l+1], e.g. {3, 2} = [3, 4, 5].+ */+struct rle16_s {+    uint16_t value;+    uint16_t length;+};++typedef struct rle16_s rle16_t;++#ifdef __cplusplus+#define CROARING_MAKE_RLE16(val, len) \+    { (uint16_t)(val), (uint16_t)(len) }  // no tagged structs until c++20+#else+#define CROARING_MAKE_RLE16(val, len) \+    (rle16_t) { .value = (uint16_t)(val), .length = (uint16_t)(len) }+#endif++/* struct run_container_s - run container bitmap+ *+ * @n_runs:   number of rle_t pairs in `runs`.+ * @capacity: capacity in rle_t pairs `runs` can hold.+ * @runs:     pairs of rle_t.+ */+STRUCT_CONTAINER(run_container_s) {+    int32_t n_runs;+    int32_t capacity;+    rle16_t *runs;+};++typedef struct run_container_s run_container_t;++#define CAST_run(c) CAST(run_container_t *, c)  // safer downcast+#define const_CAST_run(c) CAST(const run_container_t *, c)+#define movable_CAST_run(c) movable_CAST(run_container_t **, c)++/* Create a new run container. Return NULL in case of failure. */+run_container_t *run_container_create(void);++/* Create a new run container with given capacity. Return NULL in case of+ * failure. */+run_container_t *run_container_create_given_capacity(int32_t size);++/*+ * Shrink the capacity to the actual size, return the number of bytes saved.+ */+int run_container_shrink_to_fit(run_container_t *src);++/* Free memory owned by `run'. */+void run_container_free(run_container_t *run);++/* Duplicate container */+run_container_t *run_container_clone(const run_container_t *src);++/*+ * Effectively deletes the value at index index, repacking data.+ */+static inline void recoverRoomAtIndex(run_container_t *run, uint16_t index) {+    memmove(run->runs + index, run->runs + (1 + index),+            (run->n_runs - index - 1) * sizeof(rle16_t));+    run->n_runs--;+}++/**+ * Good old binary search through rle data+ */+CROARING_ALLOW_UNALIGNED+inline int32_t interleavedBinarySearch(const rle16_t *array, int32_t lenarray,+                                       uint16_t ikey) {+    int32_t low = 0;+    int32_t high = lenarray - 1;+    while (low <= high) {+        int32_t middleIndex = (low + high) >> 1;+        uint16_t middleValue = array[middleIndex].value;+        if (middleValue < ikey) {+            low = middleIndex + 1;+        } else if (middleValue > ikey) {+            high = middleIndex - 1;+        } else {+            return middleIndex;+        }+    }+    return -(low + 1);+}++/*+ * Returns index of the run which contains $ikey+ */+static inline int32_t rle16_find_run(const rle16_t *array, int32_t lenarray,+                                     uint16_t ikey) {+    int32_t low = 0;+    int32_t high = lenarray - 1;+    while (low <= high) {+        int32_t middleIndex = (low + high) >> 1;+        uint16_t min = array[middleIndex].value;+        uint16_t max = array[middleIndex].value + array[middleIndex].length;+        if (ikey > max) {+            low = middleIndex + 1;+        } else if (ikey < min) {+            high = middleIndex - 1;+        } else {+            return middleIndex;+        }+    }+    return -(low + 1);+}++/**+ * Returns number of runs which can'be be merged with the key because they+ * are less than the key.+ * Note that [5,6,7,8] can be merged with the key 9 and won't be counted.+ */+static inline int32_t rle16_count_less(const rle16_t *array, int32_t lenarray,+                                       uint16_t key) {+    if (lenarray == 0) return 0;+    int32_t low = 0;+    int32_t high = lenarray - 1;+    while (low <= high) {+        int32_t middleIndex = (low + high) >> 1;+        uint16_t min_value = array[middleIndex].value;+        uint16_t max_value =+            array[middleIndex].value + array[middleIndex].length;+        if (max_value + UINT32_C(1) < key) {  // uint32 arithmetic+            low = middleIndex + 1;+        } else if (key < min_value) {+            high = middleIndex - 1;+        } else {+            return middleIndex;+        }+    }+    return low;+}++static inline int32_t rle16_count_greater(const rle16_t *array,+                                          int32_t lenarray, uint16_t key) {+    if (lenarray == 0) return 0;+    int32_t low = 0;+    int32_t high = lenarray - 1;+    while (low <= high) {+        int32_t middleIndex = (low + high) >> 1;+        uint16_t min_value = array[middleIndex].value;+        uint16_t max_value =+            array[middleIndex].value + array[middleIndex].length;+        if (max_value < key) {+            low = middleIndex + 1;+        } else if (key + UINT32_C(1) < min_value) {  // uint32 arithmetic+            high = middleIndex - 1;+        } else {+            return lenarray - (middleIndex + 1);+        }+    }+    return lenarray - low;+}++/**+ * increase capacity to at least min. Whether the+ * existing data needs to be copied over depends on copy. If "copy" is false,+ * then the new content will be uninitialized, otherwise a copy is made.+ */+void run_container_grow(run_container_t *run, int32_t min, bool copy);++/**+ * Moves the data so that we can write data at index+ */+static inline void makeRoomAtIndex(run_container_t *run, uint16_t index) {+    /* This function calls realloc + memmove sequentially to move by one index.+     * Potentially copying twice the array.+     */+    if (run->n_runs + 1 > run->capacity)+        run_container_grow(run, run->n_runs + 1, true);+    memmove(run->runs + 1 + index, run->runs + index,+            (run->n_runs - index) * sizeof(rle16_t));+    run->n_runs++;+}++/* Add `pos' to `run'. Returns true if `pos' was not present. */+bool run_container_add(run_container_t *run, uint16_t pos);++/* Remove `pos' from `run'. Returns true if `pos' was present. */+static inline bool run_container_remove(run_container_t *run, uint16_t pos) {+    int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos);+    if (index >= 0) {+        int32_t le = run->runs[index].length;+        if (le == 0) {+            recoverRoomAtIndex(run, (uint16_t)index);+        } else {+            run->runs[index].value++;+            run->runs[index].length--;+        }+        return true;+    }+    index = -index - 2;  // points to preceding value, possibly -1+    if (index >= 0) {    // possible match+        int32_t offset = pos - run->runs[index].value;+        int32_t le = run->runs[index].length;+        if (offset < le) {+            // need to break in two+            run->runs[index].length = (uint16_t)(offset - 1);+            // need to insert+            uint16_t newvalue = pos + 1;+            int32_t newlength = le - offset - 1;+            makeRoomAtIndex(run, (uint16_t)(index + 1));+            run->runs[index + 1].value = newvalue;+            run->runs[index + 1].length = (uint16_t)newlength;+            return true;++        } else if (offset == le) {+            run->runs[index].length--;+            return true;+        }+    }+    // no match+    return false;+}++/* Check whether `pos' is present in `run'.  */+CROARING_ALLOW_UNALIGNED+inline bool run_container_contains(const run_container_t *run, uint16_t pos) {+    int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos);+    if (index >= 0) return true;+    index = -index - 2;  // points to preceding value, possibly -1+    if (index != -1) {   // possible match+        int32_t offset = pos - run->runs[index].value;+        int32_t le = run->runs[index].length;+        if (offset <= le) return true;+    }+    return false;+}++/*+ * Check whether all positions in a range of positions from pos_start (included)+ * to pos_end (excluded) is present in `run'.+ */+CROARING_ALLOW_UNALIGNED+static inline bool run_container_contains_range(const run_container_t *run,+                                                uint32_t pos_start,+                                                uint32_t pos_end) {+    uint32_t count = 0;+    int32_t index =+        interleavedBinarySearch(run->runs, run->n_runs, (uint16_t)pos_start);+    if (index < 0) {+        index = -index - 2;+        if ((index == -1) ||+            ((pos_start - run->runs[index].value) > run->runs[index].length)) {+            return false;+        }+    }+    for (int32_t i = index; i < run->n_runs; ++i) {+        const uint32_t stop = run->runs[i].value + run->runs[i].length;+        if (run->runs[i].value >= pos_end) break;+        if (stop >= pos_end) {+            count += (((pos_end - run->runs[i].value) > 0)+                          ? (pos_end - run->runs[i].value)+                          : 0);+            break;+        }+        const uint32_t min = (stop - pos_start) > 0 ? (stop - pos_start) : 0;+        count += (min < run->runs[i].length) ? min : run->runs[i].length;+    }+    return count >= (pos_end - pos_start - 1);+}++/* Get the cardinality of `run'. Requires an actual computation. */+int run_container_cardinality(const run_container_t *run);++/* Card > 0?, see run_container_empty for the reverse */+static inline bool run_container_nonzero_cardinality(+    const run_container_t *run) {+    return run->n_runs > 0;  // runs never empty+}++/* Card == 0?, see run_container_nonzero_cardinality for the reverse */+static inline bool run_container_empty(const run_container_t *run) {+    return run->n_runs == 0;  // runs never empty+}++/* Copy one container into another. We assume that they are distinct. */+void run_container_copy(const run_container_t *src, run_container_t *dst);++/**+ * Append run described by vl to the run container, possibly merging.+ * It is assumed that the run would be inserted at the end of the container, no+ * check is made.+ * It is assumed that the run container has the necessary capacity: caller is+ * responsible for checking memory capacity.+ *+ *+ * This is not a safe function, it is meant for performance: use with care.+ */+static inline void run_container_append(run_container_t *run, rle16_t vl,+                                        rle16_t *previousrl) {+    const uint32_t previousend = previousrl->value + previousrl->length;+    if (vl.value > previousend + 1) {  // we add a new one+        run->runs[run->n_runs] = vl;+        run->n_runs++;+        *previousrl = vl;+    } else {+        uint32_t newend = vl.value + vl.length + UINT32_C(1);+        if (newend > previousend) {  // we merge+            previousrl->length = (uint16_t)(newend - 1 - previousrl->value);+            run->runs[run->n_runs - 1] = *previousrl;+        }+    }+}++/**+ * Like run_container_append but it is assumed that the content of run is empty.+ */+static inline rle16_t run_container_append_first(run_container_t *run,+                                                 rle16_t vl) {+    run->runs[run->n_runs] = vl;+    run->n_runs++;+    return vl;+}++/**+ * append a single value  given by val to the run container, possibly merging.+ * It is assumed that the value would be inserted at the end of the container,+ * no check is made.+ * It is assumed that the run container has the necessary capacity: caller is+ * responsible for checking memory capacity.+ *+ * This is not a safe function, it is meant for performance: use with care.+ */+static inline void run_container_append_value(run_container_t *run,+                                              uint16_t val,+                                              rle16_t *previousrl) {+    const uint32_t previousend = previousrl->value + previousrl->length;+    if (val > previousend + 1) {  // we add a new one+        *previousrl = CROARING_MAKE_RLE16(val, 0);+        run->runs[run->n_runs] = *previousrl;+        run->n_runs++;+    } else if (val == previousend + 1) {  // we merge+        previousrl->length++;+        run->runs[run->n_runs - 1] = *previousrl;+    }+}++/**+ * Like run_container_append_value but it is assumed that the content of run is+ * empty.+ */+static inline rle16_t run_container_append_value_first(run_container_t *run,+                                                       uint16_t val) {+    rle16_t newrle = CROARING_MAKE_RLE16(val, 0);+    run->runs[run->n_runs] = newrle;+    run->n_runs++;+    return newrle;+}++/* Check whether the container spans the whole chunk (cardinality = 1<<16).+ * This check can be done in constant time (inexpensive). */+CROARING_ALLOW_UNALIGNED+static inline bool run_container_is_full(const run_container_t *run) {+    rle16_t vl = run->runs[0];+    return (run->n_runs == 1) && (vl.value == 0) && (vl.length == 0xFFFF);+}++/* Compute the union of `src_1' and `src_2' and write the result to `dst'+ * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */+void run_container_union(const run_container_t *src_1,+                         const run_container_t *src_2, run_container_t *dst);++/* Compute the union of `src_1' and `src_2' and write the result to `src_1' */+void run_container_union_inplace(run_container_t *src_1,+                                 const run_container_t *src_2);++/* Compute the intersection of src_1 and src_2 and write the result to+ * dst. It is assumed that dst is distinct from both src_1 and src_2. */+void run_container_intersection(const run_container_t *src_1,+                                const run_container_t *src_2,+                                run_container_t *dst);++/* Compute the size of the intersection of src_1 and src_2 . */+int run_container_intersection_cardinality(const run_container_t *src_1,+                                           const run_container_t *src_2);++/* Check whether src_1 and src_2 intersect. */+bool run_container_intersect(const run_container_t *src_1,+                             const run_container_t *src_2);++/* Compute the symmetric difference of `src_1' and `src_2' and write the result+ * to `dst'+ * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */+void run_container_xor(const run_container_t *src_1,+                       const run_container_t *src_2, run_container_t *dst);++/*+ * Write out the 16-bit integers contained in this container as a list of 32-bit+ * integers using base+ * as the starting value (it might be expected that base has zeros in its 16+ * least significant bits).+ * The function returns the number of values written.+ * The caller is responsible for allocating enough memory in out.+ */+int run_container_to_uint32_array(void *vout, const run_container_t *cont,+                                  uint32_t base);++/*+ * Print this container using printf (useful for debugging).+ */+void run_container_printf(const run_container_t *v);++/*+ * Print this container using printf as a comma-separated list of 32-bit+ * integers starting at base.+ */+void run_container_printf_as_uint32_array(const run_container_t *v,+                                          uint32_t base);++bool run_container_validate(const run_container_t *run, const char **reason);++/**+ * Return the serialized size in bytes of a container having "num_runs" runs.+ */+static inline int32_t run_container_serialized_size_in_bytes(int32_t num_runs) {+    return sizeof(uint16_t) ++           sizeof(rle16_t) * num_runs;  // each run requires 2 2-byte entries.+}++bool run_container_iterate(const run_container_t *cont, uint32_t base,+                           roaring_iterator iterator, void *ptr);+bool run_container_iterate64(const run_container_t *cont, uint32_t base,+                             roaring_iterator64 iterator, uint64_t high_bits,+                             void *ptr);++/**+ * Writes the underlying array to buf, outputs how many bytes were written.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes written should be run_container_size_in_bytes(container).+ */+int32_t run_container_write(const run_container_t *container, char *buf);++/**+ * Reads the instance from buf, outputs how many bytes were read.+ * This is meant to be byte-by-byte compatible with the Java and Go versions of+ * Roaring.+ * The number of bytes read should be bitset_container_size_in_bytes(container).+ * The cardinality parameter is provided for consistency with other containers,+ * but+ * it might be effectively ignored..+ */+int32_t run_container_read(int32_t cardinality, run_container_t *container,+                           const char *buf);++/**+ * Return the serialized size in bytes of a container (see run_container_write).+ * This is meant to be compatible with the Java and Go versions of Roaring.+ */+CROARING_ALLOW_UNALIGNED+static inline int32_t run_container_size_in_bytes(+    const run_container_t *container) {+    return run_container_serialized_size_in_bytes(container->n_runs);+}++/**+ * Return true if the two containers have the same content.+ */+CROARING_ALLOW_UNALIGNED+static inline bool run_container_equals(const run_container_t *container1,+                                        const run_container_t *container2) {+    if (container1->n_runs != container2->n_runs) {+        return false;+    }+    return memequals(container1->runs, container2->runs,+                     container1->n_runs * sizeof(rle16_t));+}++/**+ * Return true if container1 is a subset of container2.+ */+bool run_container_is_subset(const run_container_t *container1,+                             const run_container_t *container2);++/**+ * Appends the run [start, start + length] to the n_runs sorted runs in `runs',+ * xoring it with the last run, and returns the new number of runs. `runs' must+ * have room for n_runs + 1 runs. Follows the Java implementation closely.+ */+static inline int32_t run_append_exclusive(rle16_t *runs, int32_t n_runs,+                                           uint16_t start, uint16_t length) {+    rle16_t *last_run = n_runs ? runs + (n_runs - 1) : NULL;+    const int old_end = n_runs ? last_run->value + last_run->length + 1 : -1;+    if (start > old_end) {+        runs[n_runs] = CROARING_MAKE_RLE16(start, length);+        return n_runs + 1;+    }+    if (start == old_end) {  // we merge+        last_run->length += length + 1;+        return n_runs;+    }+    const int new_end = start + length + 1;+    if (start == last_run->value) {  // wipe out previous+        if (new_end < old_end) {+            *last_run = CROARING_MAKE_RLE16(new_end, old_end - new_end - 1);+        } else if (new_end > old_end) {+            *last_run = CROARING_MAKE_RLE16(old_end, new_end - old_end - 1);+        } else {+            return n_runs - 1;+        }+        return n_runs;+    }+    last_run->length = start - last_run->value - 1;+    if (new_end < old_end) {+        runs[n_runs] = CROARING_MAKE_RLE16(new_end, old_end - new_end - 1);+        return n_runs + 1;+    } else if (new_end > old_end) {+        runs[n_runs] = CROARING_MAKE_RLE16(old_end, new_end - old_end - 1);+        return n_runs + 1;+    }+    return n_runs;+}++/**+ * Used in a start-finish scan that appends segments, for XOR and NOT+ */++void run_container_smart_append_exclusive(run_container_t *src,+                                          const uint16_t start,+                                          const uint16_t length);++/**+ * The new container consists of a single run [start,stop).+ * It is required that stop>start, the caller is responsability for this check.+ * It is required that stop <= (1<<16), the caller is responsability for this+ * check. The cardinality of the created container is stop - start. Returns NULL+ * on failure+ */+static inline run_container_t *run_container_create_range(uint32_t start,+                                                          uint32_t stop) {+    run_container_t *rc = run_container_create_given_capacity(1);+    if (rc) {+        rle16_t r;+        r.value = (uint16_t)start;+        r.length = (uint16_t)(stop - start - 1);+        run_container_append_first(rc, r);+    }+    return rc;+}++/**+ * If the element of given rank is in this container, supposing that the first+ * element has rank start_rank, then the function returns true and sets element+ * accordingly.+ * Otherwise, it returns false and update start_rank.+ */+bool run_container_select(const run_container_t *container,+                          uint32_t *start_rank, uint32_t rank,+                          uint32_t *element);++/* Compute the difference of src_1 and src_2 and write the result to+ * dst. It is assumed that dst is distinct from both src_1 and src_2. */++void run_container_andnot(const run_container_t *src_1,+                          const run_container_t *src_2, run_container_t *dst);++void run_container_offset(const run_container_t *c, container_t **loc,+                          container_t **hic, uint16_t offset);++/* Returns the smallest value (assumes not empty) */+inline uint16_t run_container_minimum(const run_container_t *run) {+    if (run->n_runs == 0) return 0;+    return run->runs[0].value;+}++/* Returns the largest value (assumes not empty) */+CROARING_ALLOW_UNALIGNED+inline uint16_t run_container_maximum(const run_container_t *run) {+    if (run->n_runs == 0) return 0;+    return run->runs[run->n_runs - 1].value + run->runs[run->n_runs - 1].length;+}++/* Returns the number of values equal or smaller than x */+int run_container_rank(const run_container_t *arr, uint16_t x);++/* bulk version of run_container_rank(); return number of consumed elements */+uint32_t run_container_rank_many(const run_container_t *arr,+                                 uint64_t start_rank, const uint32_t *begin,+                                 const uint32_t *end, uint64_t *ans);++/* Returns the index of x, if not exsist return -1 */+int run_container_get_index(const run_container_t *arr, uint16_t x);++/* Returns the index of the first run containing a value at least as large as x,+ * or -1 */+inline int run_container_index_equalorlarger(const run_container_t *arr,+                                             uint16_t x) {+    int32_t index = interleavedBinarySearch(arr->runs, arr->n_runs, x);+    if (index >= 0) return index;+    index = -index - 2;  // points to preceding run, possibly -1+    if (index != -1) {   // possible match+        int32_t offset = x - arr->runs[index].value;+        int32_t le = arr->runs[index].length;+        if (offset <= le) return index;+    }+    index += 1;+    if (index < arr->n_runs) {+        return index;+    }+    return -1;+}++/*+ * Add all values in range [min, max] using hint.+ */+static inline void run_container_add_range_nruns(run_container_t *run,+                                                 uint32_t min, uint32_t max,+                                                 int32_t nruns_less,+                                                 int32_t nruns_greater) {+    int32_t nruns_common = run->n_runs - nruns_less - nruns_greater;+    if (nruns_common == 0) {+        makeRoomAtIndex(run, (uint16_t)nruns_less);+        run->runs[nruns_less].value = (uint16_t)min;+        run->runs[nruns_less].length = (uint16_t)(max - min);+    } else {+        uint32_t common_min = run->runs[nruns_less].value;+        uint32_t common_max = run->runs[nruns_less + nruns_common - 1].value ++                              run->runs[nruns_less + nruns_common - 1].length;+        uint32_t result_min = (common_min < min) ? common_min : min;+        uint32_t result_max = (common_max > max) ? common_max : max;++        run->runs[nruns_less].value = (uint16_t)result_min;+        run->runs[nruns_less].length = (uint16_t)(result_max - result_min);++        memmove(&(run->runs[nruns_less + 1]),+                &(run->runs[run->n_runs - nruns_greater]),+                nruns_greater * sizeof(rle16_t));+        run->n_runs = nruns_less + 1 + nruns_greater;+    }+}++/**+ * Add all values in range [min, max]. This function is currently unused+ * and left as documentation.+ */+/*static inline void run_container_add_range(run_container_t* run,+                                           uint32_t min, uint32_t max) {+    int32_t nruns_greater = rle16_count_greater(run->runs, run->n_runs, max);+    int32_t nruns_less = rle16_count_less(run->runs, run->n_runs -+nruns_greater, min); run_container_add_range_nruns(run, min, max, nruns_less,+nruns_greater);+}*/++/**+ * Shifts last $count elements either left (distance < 0) or right (distance >+ * 0)+ */+static inline void run_container_shift_tail(run_container_t *run, int32_t count,+                                            int32_t distance) {+    if (distance > 0) {+        if (run->capacity < count + distance) {+            run_container_grow(run, count + distance, true);+        }+    }+    int32_t srcpos = run->n_runs - count;+    int32_t dstpos = srcpos + distance;+    memmove(&(run->runs[dstpos]), &(run->runs[srcpos]),+            sizeof(rle16_t) * count);+    run->n_runs += distance;+}++/**+ * Remove all elements in range [min, max]+ */+static inline void run_container_remove_range(run_container_t *run,+                                              uint32_t min, uint32_t max) {+    int32_t first = rle16_find_run(run->runs, run->n_runs, (uint16_t)min);+    int32_t last = rle16_find_run(run->runs, run->n_runs, (uint16_t)max);++    if (first >= 0 && min > run->runs[first].value &&+        max < ((uint32_t)run->runs[first].value ++               (uint32_t)run->runs[first].length)) {+        // split this run into two adjacent runs++        // right subinterval+        makeRoomAtIndex(run, (uint16_t)(first + 1));+        run->runs[first + 1].value = (uint16_t)(max + 1);+        run->runs[first + 1].length =+            (uint16_t)((run->runs[first].value + run->runs[first].length) -+                       (max + 1));++        // left subinterval+        run->runs[first].length =+            (uint16_t)((min - 1) - run->runs[first].value);++        return;+    }++    // update left-most partial run+    if (first >= 0) {+        if (min > run->runs[first].value) {+            run->runs[first].length =+                (uint16_t)((min - 1) - run->runs[first].value);+            first++;+        }+    } else {+        first = -first - 1;+    }++    // update right-most run+    if (last >= 0) {+        uint16_t run_max = run->runs[last].value + run->runs[last].length;+        if (run_max > max) {+            run->runs[last].value = (uint16_t)(max + 1);+            run->runs[last].length = (uint16_t)(run_max - (max + 1));+            last--;+        }+    } else {+        last = (-last - 1) - 1;+    }++    // remove intermediate runs+    if (first <= last) {+        run_container_shift_tail(run, run->n_runs - (last + 1),+                                 -(last - first + 1));+    }+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#endif /* INCLUDE_CONTAINERS_RUN_H_ */
+ cbits/CRoaring/include/roaring/isadetection.h view
@@ -0,0 +1,68 @@+/*+ * isadetection.h+ *+ * This header declares the small interface used to detect instruction-set+ * capabilities relevant to CRoaring's optimized kernels on x64 platforms. It+ * also defines compile-time feature macros that indicate whether the compiler+ * toolchain is capable of building AVX-512 code paths.+ *+ * The resulting flags are used to decide whether accelerated implementations,+ * such as AVX2 or AVX-512 variants, can be selected safely at runtime.+ */+#ifndef ROARING_ISADETECTION_H+#define ROARING_ISADETECTION_H++#include <roaring/portability.h>++#if CROARING_IS_X64  // x64++#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#ifdef __has_include+// We want to make sure that the AVX-512 functions are only built on compilers+// fully supporting AVX-512.+#if __has_include(<avx512vbmi2intrin.h>)+#define CROARING_COMPILER_SUPPORTS_AVX512 1+#endif  // #if __has_include(<avx512vbmi2intrin.h>)+#endif  // #ifdef __has_include++// Visual Studio 2019 and up support AVX-512+#ifdef _MSC_VER+#if _MSC_VER >= 1920+#define CROARING_COMPILER_SUPPORTS_AVX512 1+#endif  // #if _MSC_VER >= 1920+#endif  // #ifdef _MSC_VER++#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#define CROARING_COMPILER_SUPPORTS_AVX512 0+#endif  // #ifndef CROARING_COMPILER_SUPPORTS_AVX512+#endif  // #ifndef CROARING_COMPILER_SUPPORTS_AVX512++#endif  // CROARING_IS_X64++#if CROARING_IS_X64 || CROARING_IS_ARM64+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif+enum {+    // x64+    ROARING_SUPPORTS_AVX2 = 1,+    ROARING_SUPPORTS_AVX512 = 2,+    // ARM64+    ROARING_SUPPORTS_SVE = 4,+    ROARING_SUPPORTS_SVE2 = 8,+};+/**+ * Returns a bitmask of the ROARING_SUPPORTS_* flags describing the+ * instruction sets supported by the current processor and operating system.+ * The result is computed once and cached.+ */+int croaring_hardware_support(void);+#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif+#endif  // CROARING_IS_X64 || CROARING_IS_ARM64+#endif  // ROARING_ISADETECTION_H
+ cbits/CRoaring/include/roaring/memory.h view
@@ -0,0 +1,66 @@+/*+ * memory.h+ *+ * This header defines CRoaring's memory-allocation abstraction layer. It+ * declares the function pointer types and hook structure used to override the+ * library's malloc/realloc/calloc/free and aligned allocation routines, along+ * with the wrapper functions used throughout the codebase.+ *+ * This allows applications to integrate CRoaring with custom allocators,+ * memory trackers, arenas, or platform-specific aligned allocation policies+ * without changing the rest of the library code.+ */+#ifndef INCLUDE_ROARING_MEMORY_H_+#define INCLUDE_ROARING_MEMORY_H_++#include <stdbool.h>+#include <stddef.h>  // for size_t++#ifdef __cplusplus+extern "C" {+#endif++typedef void* (*roaring_malloc_p)(size_t);+typedef void* (*roaring_realloc_p)(void*, size_t);+typedef void* (*roaring_calloc_p)(size_t, size_t);+typedef void (*roaring_free_p)(void*);+typedef void* (*roaring_aligned_malloc_p)(size_t, size_t);+typedef void (*roaring_aligned_free_p)(void*);++typedef struct roaring_memory_s {+    roaring_malloc_p malloc;+    roaring_realloc_p realloc;+    roaring_calloc_p calloc;+    roaring_free_p free;+    roaring_aligned_malloc_p aligned_malloc;+    roaring_aligned_free_p aligned_free;+} roaring_memory_t;++void roaring_init_memory_hook(roaring_memory_t memory_hook);++void* roaring_malloc(size_t);+void* roaring_realloc(void*, size_t);+void* roaring_calloc(size_t, size_t);+void roaring_free(void*);+void* roaring_aligned_malloc(size_t, size_t);+void roaring_aligned_free(void*);++// Internal: shrinks p to size bytes, freeing it at 0 as realloc(p, 0) varies.+// Returns the block to keep, p if realloc fails; *shrunk: whether it shrank.+static inline void* croaring_realloc_shrink(void* p, size_t size,+                                            bool* shrunk) {+    if (size == 0) {+        roaring_free(p);+        *shrunk = true;+        return NULL;+    }+    void* q = roaring_realloc(p, size);+    *shrunk = q != NULL;+    return q != NULL ? q : p;+}++#ifdef __cplusplus+}+#endif++#endif  // INCLUDE_ROARING_MEMORY_H_
+ cbits/CRoaring/include/roaring/misc/configreport.h view
@@ -0,0 +1,297 @@+/*+ * configreport.h+ *+ * This header provides lightweight helpers to report basic build and runtime+ * configuration information, such as processor family, compiler identity,+ * optimization status, endianness, and selected hardware characteristics.+ *+ * It is mainly intended for diagnostics, benchmarking output, and quick+ * inspection of the environment in which CRoaring or code using CRoaring was+ * compiled and is being executed.+ *+ * If this gets compiled into a different execution unit than the CRoaring+ * library, the functions croaring_hardware_support() & ROARING_SUPPORTS_AVX512+ * and croaring_hardware_support() & ROARING_SUPPORTS_AVX2 *may* trigger an+ * additional call to dynamic_croaring_detect_supported_architectures().+ */+#ifndef CROARING_INCLUDE_MISC_CONFIGREPORT_H_+#define CROARING_INCLUDE_MISC_CONFIGREPORT_H_++#include <stddef.h>  // for size_t+#include <stdint.h>+#include <stdio.h>++#include <roaring/portability.h>+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace misc {+#endif /* __cplusplus */++#if CROARING_IS_X64+// useful for basic info (0)+static inline void native_cpuid(unsigned int *eax, unsigned int *ebx,+                                unsigned int *ecx, unsigned int *edx) {+#ifdef CROARING_INLINE_ASM+    __asm volatile("cpuid"+                   : "=a"(*eax), "=b"(*ebx), "=c"(*ecx), "=d"(*edx)+                   : "0"(*eax), "2"(*ecx));+#else+    (void)eax;+    (void)ebx;+    (void)ecx;+    (void)edx;+#endif /* CROARING_INLINE_ASM */+}++// CPUID instruction takes no parameters as CPUID implicitly uses the EAX+// register.+// The EAX register should be loaded with a value specifying what information to+// return+static inline void cpuinfo(int code, int *eax, int *ebx, int *ecx, int *edx) {+#ifdef CROARING_INLINE_ASM+    __asm__ volatile("cpuid;"  //  call cpuid instruction+                     : "=a"(*eax), "=b"(*ebx), "=c"(*ecx),+                       "=d"(*edx)  // output equal to "movl  %%eax %1"+                     : "a"(code)   // input equal to "movl %1, %%eax"+                     //:"%eax","%ebx","%ecx","%edx"// clobbered register+    );+#else+    (void)code;+    (void)eax;+    (void)ebx;+    (void)ecx;+    (void)edx;+#endif /* CROARING_INLINE_ASM */+}++static inline int computecacheline() {+    int eax = 0, ebx = 0, ecx = 0, edx = 0;+    cpuinfo((int)0x80000006, &eax, &ebx, &ecx, &edx);+    return ecx & 0xFF;+}++// this is quite imperfect, but can be handy+static inline const char *guessprocessor() {+    unsigned eax = 1, ebx = 0, ecx = 0, edx = 0;+    native_cpuid(&eax, &ebx, &ecx, &edx);+    const char *codename;+    switch (eax >> 4) {+        case 0x506E:+            codename = "Skylake";+            break;+        case 0x406C:+            codename = "CherryTrail";+            break;+        case 0x306D:+            codename = "Broadwell";+            break;+        case 0x306C:+            codename = "Haswell";+            break;+        case 0x306A:+            codename = "IvyBridge";+            break;+        case 0x206A:+        case 0x206D:+            codename = "SandyBridge";+            break;+        case 0x2065:+        case 0x206C:+        case 0x206F:+            codename = "Westmere";+            break;+        case 0x106E:+        case 0x106A:+        case 0x206E:+            codename = "Nehalem";+            break;+        case 0x1067:+        case 0x106D:+            codename = "Penryn";+            break;+        case 0x006F:+        case 0x1066:+            codename = "Merom";+            break;+        case 0x0066:+            codename = "Presler";+            break;+        case 0x0063:+        case 0x0064:+            codename = "Prescott";+            break;+        case 0x006D:+            codename = "Dothan";+            break;+        case 0x0366:+            codename = "Cedarview";+            break;+        case 0x0266:+            codename = "Lincroft";+            break;+        case 0x016C:+            codename = "Pineview";+            break;+        case 0x706e:+        case 0x606a:+            codename = "Icelake";+            break;+        case 0x706a:+        case 0x506c:+            codename = "Goldmont";+            break;+        case 0x806c:+        case 0x806d:+            codename = "TigerLake";+            break;+        case 0x806e:+        case 0x906e:+            codename = "Kabylake";+            break;+        case 0xa065:+        case 0xa066:+            codename = "Cometlake";+            break;+        case 0xa067:+            codename = "Rocketlake";+            break;+        case 0x9067:+        case 0x906a:+            codename = "Alderlake";+            break;+        case 0xb067:+            codename = "Raptorlake";+            break;+        case 0x30f1:+        case 0x60f0:+        case 0x70f1:+        case 0x60f8:+        case 0x90f0:+            codename = "Zen2";+            break;+        case 0x20f10:+        case 0x50f00:+            codename = "Zen3";+            break;+        case 0x40f40:+            codename = "Zen3+";+            break;+        case 0x60f10:+            codename = "Zen4";+            break;+        default:+            codename = "unknown";+            break;+    }+    return codename;+}++static inline void tellmeall() {+#if CROARING_IS_BIG_ENDIAN+    printf("# big-endian system detected\n");+#endif /* CROARING_IS_BIG_ENDIAN */+    printf("# x64 processor: %s\n", guessprocessor());+#if defined(__clang__)+    printf("# compiler: clang %d.%d.%d\n", __clang_major__, __clang_minor__,+           __clang_patchlevel__);+#elif defined(__GNUC__)+    printf(" compiler: gcc %d.%d.%d\n", __GNUC__, __GNUC_MINOR__,+           __GNUC_PATCHLEVEL__);+#elif defined(_MSC_VER)+    printf("# compiler: MSVC %d\n", _MSC_VER);+#elif defined(__VERSION__)+    printf("# compiler: %s\n", __VERSION__);+#else+    printf("# compiler: unknown\n");+#endif /* compiler detection */++#if defined(__OPTIMIZE__)+    printf("# optimization: on\n");+#elif defined(__OPTIMIZE_SIZE__)+    printf("# size optimization: on\n");+#elif defined(_MSC_VER) && !defined(_DEBUG)+    printf("# optimization: on\n");+#else+    printf("# optimization: off\n");+#endif /* optimization detection */++    if ((sizeof(int) != 4) || (sizeof(long) != 8)) {+        printf("number of bytes: int = %lu long = %lu \n",+               (long unsigned int)sizeof(size_t),+               (long unsigned int)sizeof(int));+    }+#if __BIG_ENDIAN__+    printf("# you have a big endian machine\n");+#endif /* __BIG_ENDIAN__ */+#if __CHAR_BIT__+    if (__CHAR_BIT__ != 8)+        printf("# on your machine, chars don't have 8bits???\n");+#endif /* __CHAR_BIT__ */+    if (computecacheline() != 64)+        printf("# cache line: %d bytes\n", computecacheline());+}+#else++static inline void tellmeall() {+#if CROARING_IS_BIG_ENDIAN+    printf("# big-endian system detected\n");+#endif /* CROARING_IS_BIG_ENDIAN */+#if CROARING_IS_X64+    printf("# x64 processor detected\n");+#elif defined(CROARING_IS_RISCV64)+    printf("# RISC-V 64-bit processor detected\n");+#elif defined(CROARING_IS_RISCV)+    printf("# RISC-V processor detected\n");+#elif defined(__arm__) || defined(__aarch64__) || defined(__arm64__) || \+    defined(_M_ARM) || defined(_M_ARM64)+    printf("# ARM processor detected\n");+#else+    printf("# Non-x64, non-ARM, non-RISC-V processor\n");+#endif /* architecture detection */+#if defined(__clang__)+    printf("# compiler: clang %d.%d.%d\n", __clang_major__, __clang_minor__,+           __clang_patchlevel__);+#elif defined(__GNUC__)+    printf(" compiler: gcc %d.%d.%d\n", __GNUC__, __GNUC_MINOR__,+           __GNUC_PATCHLEVEL__);+#elif defined(_MSC_VER)+    printf("# compiler: MSVC %d\n", _MSC_VER);+#elif defined(__VERSION__)+    printf("# compiler: %s\n", __VERSION__);+#else+    printf("# compiler: unknown\n");+#endif /* compiler detection */++#if defined(__OPTIMIZE__)+    printf("# optimization: on\n");+#elif defined(__OPTIMIZE_SIZE__)+    printf("# size optimization: on\n");+#elif defined(_MSC_VER) && !defined(_DEBUG)+    printf("# optimization: on\n");+#else+    printf("# optimization: off\n");+#endif /* optimization detection */+    if ((sizeof(int) != 4) || (sizeof(long) != 8)) {+        printf("number of bytes: int = %lu long = %lu \n",+               (long unsigned int)sizeof(size_t),+               (long unsigned int)sizeof(int));+    }+#if __BIG_ENDIAN__+    printf("# you have a big endian machine\n");+#endif /* __BIG_ENDIAN__ */+#if __CHAR_BIT__+    if (__CHAR_BIT__ != 8)+        printf("# on your machine, chars don't have 8bits???\n");+#endif /* __CHAR_BIT__ */+}++#endif /* CROARING_IS_X64 */++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace misc {+#endif /* __cplusplus */++#endif /* CROARING_INCLUDE_MISC_CONFIGREPORT_H_ */
+ cbits/CRoaring/include/roaring/portability.h view
@@ -0,0 +1,709 @@+/*+ * portability.h+ *+ * This header centralizes compiler-, platform-, and architecture-specific+ * portability definitions used throughout CRoaring. It provides feature+ * detection, calling-convention and attribute macros, intrinsic and inline+ * assembly enablement, endianness helpers, alignment annotations, atomic+ * reference-count support, and other low-level compatibility glue.+ *+ * The goal is to keep these conditional definitions in one place so the rest+ * of the codebase can rely on a more uniform interface across GCC, Clang,+ * MSVC, x86/x64, ARM/NEON, and other supported environments.+ */++/**+ * All macros should be prefixed with either CROARING or ROARING.+ * The library uses both ROARING_...+ * as well as CROARING_ as prefixes. The ROARING_ prefix is for+ * macros that are provided by the build system or that are closely+ * related to the format. The header macros may also use ROARING_.+ * The CROARING_ prefix is for internal macros that a user is unlikely+ * to ever interact with.+ */++#ifndef CROARING_INCLUDE_PORTABILITY_H_+#define CROARING_INCLUDE_PORTABILITY_H_++// Users who need _GNU_SOURCE should define it?+// #ifndef _GNU_SOURCE+// #define _GNU_SOURCE 1+// #endif  // _GNU_SOURCE+#ifndef __STDC_FORMAT_MACROS+#define __STDC_FORMAT_MACROS 1+#endif  // __STDC_FORMAT_MACROS++#ifdef _MSC_VER+#define CROARING_VISUAL_STUDIO 1+/**+ * We want to differentiate carefully between+ * clang under visual studio and regular visual+ * studio.+ */+#ifdef __clang__+// clang under visual studio+#define CROARING_CLANG_VISUAL_STUDIO 1+#else+// just regular visual studio (best guess)+#define CROARING_REGULAR_VISUAL_STUDIO 1+#endif  // __clang__+#endif  // _MSC_VER+#ifndef CROARING_VISUAL_STUDIO+#define CROARING_VISUAL_STUDIO 0+#endif+#ifndef CROARING_CLANG_VISUAL_STUDIO+#define CROARING_CLANG_VISUAL_STUDIO 0+#endif+#ifndef CROARING_REGULAR_VISUAL_STUDIO+#define CROARING_REGULAR_VISUAL_STUDIO 0+#endif++#include <stdbool.h>+#include <stdint.h>+#include <stdlib.h>  // will provide posix_memalign with _POSIX_C_SOURCE as defined above+#ifdef __GLIBC__+#include <malloc.h>  // this should never be needed but there are some reports that it is needed.+#endif+// alignas/alignof are keywords in C++ and in C23+, where <stdalign.h> is+// deprecated. Only include it for C11..C17.+#if !defined(__cplusplus) && \+    (!defined(__STDC_VERSION__) || __STDC_VERSION__ < 202311L)+#include <stdalign.h>+#endif++#ifdef __cplusplus+extern "C" {  // portability definitions are in global scope, not a namespace+#endif++#if defined(__SIZEOF_LONG_LONG__) && __SIZEOF_LONG_LONG__ != 8+#error This code assumes  64-bit long longs (by use of the GCC intrinsics). Your system is not currently supported.+#endif++#if CROARING_REGULAR_VISUAL_STUDIO+#ifndef __restrict__+#define __restrict__ __restrict+#endif  // __restrict__+#endif  // CROARING_REGULAR_VISUAL_STUDIO++#if defined(__riscv) || defined(_M_RISCV32) || defined(_M_RISCV64)+#define CROARING_IS_RISCV 1++#if (defined(__riscv_xlen) && (__riscv_xlen == 64)) || defined(_M_RISCV64)+#define CROARING_IS_RISCV64 1+#endif++#elif defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))+// we have an x64 processor+// (ARM64EC also defines _M_X64, but it targets ARM64 and is handled below)+#define CROARING_IS_X64 1++#if defined(_MSC_VER) && (_MSC_VER < 1910)+// Old visual studio systems won't support AVX2 well.+#undef CROARING_IS_X64+#endif++#if defined(__clang_major__) && (__clang_major__ <= 8) && !defined(__AVX2__)+// Older versions of clang have a bug affecting us+// https://stackoverflow.com/questions/57228537/how-does-one-use-pragma-clang-attribute-push-with-c-namespaces+#undef CROARING_IS_X64+#endif++#ifdef ROARING_DISABLE_X64+#undef CROARING_IS_X64+#endif+// we include the intrinsic header+#if !CROARING_REGULAR_VISUAL_STUDIO+/* Non-Microsoft C/C++-compatible compiler */+#include <x86intrin.h>  // on some recent GCC, this will declare posix_memalign++#if CROARING_CLANG_VISUAL_STUDIO++/**+ * You are not supposed, normally, to include these+ * headers directly. Instead you should either include intrin.h+ * or x86intrin.h. However, when compiling with clang+ * under Windows (i.e., when _MSC_VER is set), these headers+ * only get included *if* the corresponding features are detected+ * from macros:+ * e.g., if __AVX2__ is set... in turn,  we normally set these+ * macros by compiling against the corresponding architecture+ * (e.g., arch:AVX2, -mavx2, etc.) which compiles the whole+ * software with these advanced instructions. These headers would+ * normally guard against such usage, but we carefully included+ * <x86intrin.h>  (or <intrin.h>) before, so the headers+ * are fooled.+ */+// To avoid reordering imports:+// clang-format off+#include <bmiintrin.h>   // for _blsr_u64+#include <lzcntintrin.h> // for  __lzcnt64+#include <immintrin.h>   // for most things (AVX2, AVX512, _popcnt64)+#include <smmintrin.h>+#include <tmmintrin.h>+#include <avxintrin.h>+#include <avx2intrin.h>+#include <wmmintrin.h>+#if _MSC_VER >= 1920+// Important: we need the AVX-512 headers:+#include <avx512fintrin.h>+#include <avx512dqintrin.h>+#include <avx512cdintrin.h>+#include <avx512bwintrin.h>+#include <avx512vlintrin.h>+#include <avx512vbmiintrin.h>+#include <avx512vbmi2intrin.h>+#include <avx512vpopcntdqintrin.h>+// clang-format on+#endif  // _MSC_VER >= 1920+// unfortunately, we may not get _blsr_u64, but, thankfully, clang+// has it as a macro.+#ifndef _blsr_u64+// we roll our own+#define _blsr_u64(n) ((n - 1) & n)+#endif  //  _blsr_u64+#endif  // SIMDJSON_CLANG_VISUAL_STUDIO++#endif  // CROARING_REGULAR_VISUAL_STUDIO+#endif  // defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))++#if defined(__aarch64__) || defined(_M_ARM64) || defined(_M_ARM64EC)+// we have a 64-bit ARM processor+#define CROARING_IS_ARM64 1+#endif++#if !defined(CROARING_USENEON) && !defined(DISABLENEON) && defined(__ARM_NEON)+#define CROARING_USENEON+#endif+#if defined(CROARING_USENEON)+#include <arm_neon.h>+#endif++#if defined(__e2k__)+// we have an e2k (Elbrus-2000) processor+#define CROARING_IS_E2K 1+#endif++#if !CROARING_REGULAR_VISUAL_STUDIO && !defined(CROARING_IS_E2K) && !__FILC__+/* Non-Microsoft C/C++-compatible compiler, assumes that it supports inline+ * assembly */+#define CROARING_INLINE_ASM 1+#endif  // _MSC_VER++#if CROARING_REGULAR_VISUAL_STUDIO+/* Microsoft C/C++-compatible compiler */+#include <intrin.h>++#ifndef __clang__  // if one compiles with MSVC *with* clang, then these+                   // intrinsics are defined!!!+#define CROARING_INTRINSICS 1+// sadly there is no way to check whether we are missing these intrinsics+// specifically.++/* wrappers for Visual Studio built-ins that look like gcc built-ins+ * __builtin_ctzll */+/** result might be undefined when input_num is zero */+inline int roaring_trailing_zeroes(unsigned long long input_num) {+    unsigned long index;+#ifdef _WIN64  // highly recommended!!!+    _BitScanForward64(&index, input_num);+#else   // if we must support 32-bit Windows+    if ((uint32_t)input_num != 0) {+        _BitScanForward(&index, (uint32_t)input_num);+    } else {+        _BitScanForward(&index, (uint32_t)(input_num >> 32));+        index += 32;+    }+#endif  // _WIN64+    return index;+}++/* wrappers for Visual Studio built-ins that look like gcc built-ins+ * __builtin_clzll */+/** result might be undefined when input_num is zero */+inline int roaring_leading_zeroes(unsigned long long input_num) {+    unsigned long index;+#ifdef _WIN64  // highly recommended!!!+    _BitScanReverse64(&index, input_num);+#else   // if we must support 32-bit Windows+    if (input_num > 0xFFFFFFFF) {+        _BitScanReverse(&index, (uint32_t)(input_num >> 32));+        index += 32;+    } else {+        _BitScanReverse(&index, (uint32_t)(input_num));+    }+#endif  // _WIN64+    return 63 - index;+}++/* Use #define so this is effective even under /Ob0 (no inline) */+#define roaring_unreachable __assume(0)+#endif  // __clang__++#endif  // CROARING_REGULAR_VISUAL_STUDIO++#ifndef CROARING_INTRINSICS+#define CROARING_INTRINSICS 1+#define roaring_unreachable __builtin_unreachable()+/** result might be undefined when input_num is zero */+inline int roaring_trailing_zeroes(unsigned long long input_num) {+    return __builtin_ctzll(input_num);+}+/** result might be undefined when input_num is zero */+inline int roaring_leading_zeroes(unsigned long long input_num) {+    return __builtin_clzll(input_num);+}+#endif++#if CROARING_REGULAR_VISUAL_STUDIO+#define ALIGNED(x) __declspec(align(x))+#elif defined(__GNUC__) || defined(__clang__)+#define ALIGNED(x) __attribute__((aligned(x)))+#else+#warning "Warning. Unrecognized compiler."+#define ALIGNED(x)+#endif++#if defined(__GNUC__) || defined(__clang__)+#define CROARING_WARN_UNUSED __attribute__((warn_unused_result))+#else+#define CROARING_WARN_UNUSED+#endif++#define IS_BIG_ENDIAN (*(uint16_t *)"\0\xff" < 0x100)++#ifdef CROARING_USENEON+// we can always compute the popcount fast.+#elif (defined(_M_ARM) || defined(_M_ARM64)) && \+    ((defined(_WIN64) || defined(_WIN32)) &&    \+     defined(CROARING_REGULAR_VISUAL_STUDIO) && \+     CROARING_REGULAR_VISUAL_STUDIO)+// we will need this function:+static inline int roaring_hamming_backup(uint64_t x) {+    uint64_t c1 = UINT64_C(0x5555555555555555);+    uint64_t c2 = UINT64_C(0x3333333333333333);+    uint64_t c4 = UINT64_C(0x0F0F0F0F0F0F0F0F);+    x -= (x >> 1) & c1;+    x = ((x >> 2) & c2) + (x & c2);+    x = (x + (x >> 4)) & c4;+    x *= UINT64_C(0x0101010101010101);+    return x >> 56;+}+#endif++static inline int roaring_hamming(uint64_t x) {+#if defined(_WIN64) && defined(CROARING_REGULAR_VISUAL_STUDIO) && \+    CROARING_REGULAR_VISUAL_STUDIO+#ifdef CROARING_USENEON+    return vaddv_u8(vcnt_u8(vcreate_u8(input_num)));+#elif defined(_M_ARM64)+    return roaring_hamming_backup(x);+    // (int) _CountOneBits64(x); is unavailable+#else   // _M_ARM64+    return (int)__popcnt64(x);+#endif  // _M_ARM64+#elif defined(_WIN32) && defined(CROARING_REGULAR_VISUAL_STUDIO) && \+    CROARING_REGULAR_VISUAL_STUDIO+#ifdef _M_ARM+    return roaring_hamming_backup(x);+    // _CountOneBits is unavailable+#else   // _M_ARM+    return (int)__popcnt((unsigned int)x) ++           (int)__popcnt((unsigned int)(x >> 32));+#endif  // _M_ARM+#else+    return __builtin_popcountll(x);+#endif+}++#ifndef UINT64_C+#define UINT64_C(c) (c##ULL)+#endif  // UINT64_C++#ifndef UINT32_C+#define UINT32_C(c) (c##UL)+#endif  // UINT32_C++#ifdef __cplusplus+}  // extern "C" {+#endif  // __cplusplus++// this is almost standard?+#undef STRINGIFY_IMPLEMENTATION_+#undef STRINGIFY+#define STRINGIFY_IMPLEMENTATION_(a) #a+#define STRINGIFY(a) STRINGIFY_IMPLEMENTATION_(a)++// Our fast kernels require 64-bit systems.+//+// On 32-bit x86, we lack 64-bit popcnt, lzcnt, blsr instructions.+// Furthermore, the number of SIMD registers is reduced.+//+// On 32-bit ARM, we would have smaller registers.+//+// The library should still have the fallback kernel. It is+// slower, but it should run everywhere.++//+// Enable valid runtime implementations, and select+// CROARING_BUILTIN_IMPLEMENTATION+//++// We are going to use runtime dispatch.+#if CROARING_IS_X64+#ifdef __clang__+// clang does not have GCC push pop+// warning: clang attribute push can't be used within a namespace in clang up+// til 8.0 so CROARING_TARGET_REGION and CROARING_UNTARGET_REGION must be+// *outside* of a namespace.+#define CROARING_TARGET_REGION(T)                                      \+    _Pragma(STRINGIFY(clang attribute push(__attribute__((target(T))), \+                                           apply_to = function)))+#define CROARING_UNTARGET_REGION _Pragma("clang attribute pop")+#elif defined(__GNUC__)+// GCC is easier+#define CROARING_TARGET_REGION(T) \+    _Pragma("GCC push_options") _Pragma(STRINGIFY(GCC target(T)))+#define CROARING_UNTARGET_REGION _Pragma("GCC pop_options")+#endif  // clang then gcc++#endif  // CROARING_IS_X64++// Default target region macros don't do anything.+#ifndef CROARING_TARGET_REGION+#define CROARING_TARGET_REGION(T)+#define CROARING_UNTARGET_REGION+#endif++#define CROARING_TARGET_AVX2 \+    CROARING_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")+#define CROARING_TARGET_AVX512                                         \+    CROARING_TARGET_REGION(                                            \+        "avx2,bmi,bmi2,pclmul,lzcnt,popcnt,avx512f,avx512dq,avx512bw," \+        "avx512vbmi2,avx512bitalg,avx512vpopcntdq")+#define CROARING_UNTARGET_AVX2 CROARING_UNTARGET_REGION+#define CROARING_UNTARGET_AVX512 CROARING_UNTARGET_REGION++#ifdef __AVX2__+// No need for runtime dispatching.+// It is unnecessary and harmful to old clang to tag regions.+#undef CROARING_TARGET_AVX2+#define CROARING_TARGET_AVX2+#undef CROARING_UNTARGET_AVX2+#define CROARING_UNTARGET_AVX2+#endif++#if defined(__AVX512F__) && defined(__AVX512DQ__) && defined(__AVX512BW__) && \+    defined(__AVX512VBMI2__) && defined(__AVX512BITALG__) &&                  \+    defined(__AVX512VPOPCNTDQ__)+// No need for runtime dispatching.+// It is unnecessary and harmful to old clang to tag regions.+#undef CROARING_TARGET_AVX512+#define CROARING_TARGET_AVX512+#undef CROARING_UNTARGET_AVX512+#define CROARING_UNTARGET_AVX512+#endif++// Allow unaligned memory access+#if defined(__GNUC__) || defined(__clang__)+#define CROARING_ALLOW_UNALIGNED __attribute__((no_sanitize("alignment")))+#else+#define CROARING_ALLOW_UNALIGNED+#endif++#if defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__)+#define CROARING_IS_BIG_ENDIAN (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)+#elif defined(_WIN32)+#define CROARING_IS_BIG_ENDIAN 0+#else+#if defined(__APPLE__) || \+    defined(__FreeBSD__)  // defined __BYTE_ORDER__ && defined+                          // __ORDER_BIG_ENDIAN__+#include <machine/endian.h>+#elif defined(sun) || \+    defined(__sun)  // defined(__APPLE__) || defined(__FreeBSD__)+#include <sys/byteorder.h>+#else  // defined(__APPLE__) || defined(__FreeBSD__)++#ifdef __has_include+#if __has_include(<endian.h>)+#include <endian.h>+#endif  //__has_include(<endian.h>)+#endif  //__has_include++#endif  // defined(__APPLE__) || defined(__FreeBSD__)++#ifndef !defined(__BYTE_ORDER__) || !defined(__ORDER_LITTLE_ENDIAN__)+#define CROARING_IS_BIG_ENDIAN 0+#endif++#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define CROARING_IS_BIG_ENDIAN 0+#else  // __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define CROARING_IS_BIG_ENDIAN 1+#endif  // __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#endif++// Host <-> big endian conversion.+#if CROARING_IS_BIG_ENDIAN+#define croaring_htobe64(x) (x)++#elif defined(_WIN32) || defined(_WIN64)  // CROARING_IS_BIG_ENDIAN+#include <stdlib.h>+#define croaring_htobe64(x) _byteswap_uint64(x)++#elif defined(__APPLE__)  // CROARING_IS_BIG_ENDIAN+#include <libkern/OSByteOrder.h>+#define croaring_htobe64(x) OSSwapInt64(x)++#elif defined(__has_include) && \+    __has_include(              \+        <byteswap.h>)  && (defined(__linux__) || defined(__FreeBSD__))  // CROARING_IS_BIG_ENDIAN+#include <byteswap.h>+#if defined(__linux__)+#define croaring_htobe64(x) bswap_64(x)+#elif defined(__FreeBSD__)+#include <sys/endian.h>+#define croaring_htobe64(x) bswap64(x)+#else+#warning "Unknown platform, report as an error"+#endif++#else  // CROARING_IS_BIG_ENDIAN+// Gets compiled to bswap or equivalent on most compilers.+#define croaring_htobe64(x)                                                    \+    (((x & 0x00000000000000FFULL) << 56) |                                     \+     ((x & 0x000000000000FF00ULL) << 40) |                                     \+     ((x & 0x0000000000FF0000ULL) << 24) |                                     \+     ((x & 0x00000000FF000000ULL) << 8) | ((x & 0x000000FF00000000ULL) >> 8) | \+     ((x & 0x0000FF0000000000ULL) >> 24) |                                     \+     ((x & 0x00FF000000000000ULL) >> 40) |                                     \+     ((x & 0xFF00000000000000ULL) >> 56))+#endif  // CROARING_IS_BIG_ENDIAN+#define croaring_be64toh(x) croaring_htobe64(x)+// End of host <-> big endian conversion.++// Host <-> little-endian conversion helpers.+//+// The CRoaring "portable" serialization format (and the regular+// roaring_bitmap_serialize / Roaring64Map::write formats which build on it)+// is defined to be little-endian on the wire. Code that reads or writes+// multi-byte integers to such buffers must convert between host and+// little-endian byte order. On little-endian hosts these are no-ops; on+// big-endian hosts they swap bytes.+//+// The "frozen" format is intentionally non-portable and uses native byte+// order; it must not use these helpers.+#if CROARING_IS_BIG_ENDIAN++static inline uint16_t croaring_bswap16(uint16_t x) {+    return (uint16_t)((x << 8) | (x >> 8));+}++static inline uint32_t croaring_bswap32(uint32_t x) {+    return ((x & 0x000000FFU) << 24) | ((x & 0x0000FF00U) << 8) |+           ((x & 0x00FF0000U) >> 8) | ((x & 0xFF000000U) >> 24);+}++static inline uint64_t croaring_bswap64(uint64_t x) {+    return ((x & 0x00000000000000FFULL) << 56) |+           ((x & 0x000000000000FF00ULL) << 40) |+           ((x & 0x0000000000FF0000ULL) << 24) |+           ((x & 0x00000000FF000000ULL) << 8) |+           ((x & 0x000000FF00000000ULL) >> 8) |+           ((x & 0x0000FF0000000000ULL) >> 24) |+           ((x & 0x00FF000000000000ULL) >> 40) |+           ((x & 0xFF00000000000000ULL) >> 56);+}++#define croaring_htole16(x) croaring_bswap16(x)+#define croaring_htole32(x) croaring_bswap32(x)+#define croaring_htole64(x) croaring_bswap64(x)++#else  // CROARING_IS_BIG_ENDIAN++#define croaring_htole16(x) (x)+#define croaring_htole32(x) (x)+#define croaring_htole64(x) (x)++#endif  // CROARING_IS_BIG_ENDIAN++#define croaring_letoh16(x) croaring_htole16(x)+#define croaring_letoh32(x) croaring_htole32(x)+#define croaring_letoh64(x) croaring_htole64(x)++// Defines for the possible CROARING atomic implementations+#define CROARING_ATOMIC_IMPL_NONE 1+#define CROARING_ATOMIC_IMPL_CPP 2+#define CROARING_ATOMIC_IMPL_C 3+#define CROARING_ATOMIC_IMPL_C_WINDOWS 4++// If the use has forced a specific implementation, use that, otherwise,+// figure out the best implementation we can use.+#if !defined(CROARING_ATOMIC_IMPL)+#if defined(__cplusplus) && __cplusplus >= 201103L+#ifdef __has_include+#if __has_include(<atomic>)+#define CROARING_ATOMIC_IMPL CROARING_ATOMIC_IMPL_CPP+#endif  //__has_include(<atomic>)+#else+// We lack __has_include to check:+#define CROARING_ATOMIC_IMPL CROARING_ATOMIC_IMPL_CPP+#endif  //__has_include+#elif __STDC_VERSION__ >= 201112L && !defined(__STDC_NO_ATOMICS__)+#define CROARING_ATOMIC_IMPL CROARING_ATOMIC_IMPL_C+#elif CROARING_REGULAR_VISUAL_STUDIO+// https://www.technetworkhub.com/c11-atomics-in-visual-studio-2022-version-17/+#define CROARING_ATOMIC_IMPL CROARING_ATOMIC_IMPL_C_WINDOWS+#endif+#endif  // !defined(CROARING_ATOMIC_IMPL)++#if CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_C+#include <stdatomic.h>+typedef _Atomic(uint32_t) croaring_refcount_t;++static inline void croaring_refcount_inc(croaring_refcount_t *val) {+    // Increasing the reference counter can always be done with+    // memory_order_relaxed: New references to an object can only be formed from+    // an existing reference, and passing an existing reference from one thread+    // to another must already provide any required synchronization.+    atomic_fetch_add_explicit(val, 1, memory_order_relaxed);+}++static inline bool croaring_refcount_dec(croaring_refcount_t *val) {+    // It is important to enforce any possible access to the object in one+    // thread (through an existing reference) to happen before deleting the+    // object in a different thread. This is achieved by a "release" operation+    // after dropping a reference (any access to the object through this+    // reference must obviously happened before), and an "acquire" operation+    // before deleting the object.+    bool is_zero = atomic_fetch_sub_explicit(val, 1, memory_order_release) == 1;+    if (is_zero) {+        atomic_thread_fence(memory_order_acquire);+    }+    return is_zero;+}++static inline uint32_t croaring_refcount_get(const croaring_refcount_t *val) {+    return atomic_load_explicit(val, memory_order_relaxed);+}++// Returns true if the caller holds the only reference, and may thus take+// exclusive ownership of the object. The "acquire" operation pairs with the+// "release" operation from dropping the other references.+static inline bool croaring_refcount_is_unique(const croaring_refcount_t *val) {+    return atomic_load_explicit(val, memory_order_acquire) == 1;+}+#elif CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_CPP+#include <atomic>+typedef std::atomic<uint32_t> croaring_refcount_t;++static inline void croaring_refcount_inc(croaring_refcount_t *val) {+    val->fetch_add(1, std::memory_order_relaxed);+}++static inline bool croaring_refcount_dec(croaring_refcount_t *val) {+    // See above comments on the c11 atomic implementation for memory ordering+    bool is_zero = val->fetch_sub(1, std::memory_order_release) == 1;+    if (is_zero) {+        std::atomic_thread_fence(std::memory_order_acquire);+    }+    return is_zero;+}++static inline uint32_t croaring_refcount_get(const croaring_refcount_t *val) {+    return val->load(std::memory_order_relaxed);+}++static inline bool croaring_refcount_is_unique(const croaring_refcount_t *val) {+    return val->load(std::memory_order_acquire) == 1;+}+#elif CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_C_WINDOWS+#include <intrin.h>+#pragma intrinsic(_InterlockedIncrement)+#pragma intrinsic(_InterlockedDecrement)+#pragma intrinsic(_InterlockedOr)++// _InterlockedIncrement and _InterlockedDecrement take a (signed) long, and+// overflow is defined to wrap, so we can pretend it is a uint32_t for our case+typedef volatile long croaring_refcount_t;++static inline void croaring_refcount_inc(croaring_refcount_t *val) {+    _InterlockedIncrement(val);+}++static inline bool croaring_refcount_dec(croaring_refcount_t *val) {+    return _InterlockedDecrement(val) == 0;+}++static inline uint32_t croaring_refcount_get(const croaring_refcount_t *val) {+    // Per+    // https://learn.microsoft.com/en-us/windows/win32/sync/interlocked-variable-access+    // > Simple reads and writes to properly-aligned 32-bit variables are atomic+    // > operations. In other words, you will not end up with only one portion+    // > of the variable updated; all bits are updated in an atomic fashion.+    return *val;+}++static inline bool croaring_refcount_is_unique(const croaring_refcount_t *val) {+    // A plain read would not provide the acquire semantics: _InterlockedOr+    // leaves the value alone while acting as a memory barrier.+    return _InterlockedOr((croaring_refcount_t *)val, 0) == 1;+}+#elif CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_NONE+#include <assert.h>+typedef uint32_t croaring_refcount_t;++static inline void croaring_refcount_inc(croaring_refcount_t *val) {+    *val += 1;+}++static inline bool croaring_refcount_dec(croaring_refcount_t *val) {+    assert(*val > 0);+    *val -= 1;+    return *val == 0;+}++static inline uint32_t croaring_refcount_get(const croaring_refcount_t *val) {+    return *val;+}++static inline bool croaring_refcount_is_unique(const croaring_refcount_t *val) {+    return *val == 1;+}+#else+#error "Unknown atomic implementation"+#endif++#if defined(__GNUC__) || defined(__clang__)+#define CROARING_DEPRECATED __attribute__((deprecated))+#elif defined(_MSC_VER)+#define CROARING_DEPRECATED __declspec(deprecated)+#else+#define CROARING_DEPRECATED+#endif  // defined(__GNUC__) || defined(__clang__)++// We want to initialize structs to zero portably (C and C++), without+// warnings. We can do mystruct s = CROARING_ZERO_INITIALIZER;+#if __cplusplus+#define CROARING_ZERO_INITIALIZER \+    {}+#else+#define CROARING_ZERO_INITIALIZER \+    { 0 }+#endif++#if defined(__cplusplus)+#define CROARING_STATIC_ASSERT(x, y) static_assert(x, y)+#else+#define CROARING_STATIC_ASSERT(x, y) _Static_assert(x, y)+#endif++// We need portability.h to be included first,+// but we also always want isadetection.h to be+// included (right after).+// See https://github.com/RoaringBitmap/CRoaring/issues/394+// There is no scenario where we want portability.h to+// be included, but not isadetection.h: the latter is a+// strict requirement.+#include <roaring/isadetection.h>  // include it last!+#endif                             /* INCLUDE_PORTABILITY_H_ */
+ cbits/CRoaring/include/roaring/roaring.h view
@@ -0,0 +1,1430 @@+/*+ * An implementation of Roaring Bitmaps in C.+ *+ * This is the main public header for the 32-bit CRoaring API. A Roaring bitmap+ * represents a set of unsigned 32-bit integers by partitioning the value space+ * into 16-bit chunks and storing each chunk in a container chosen to match the+ * local data density. Sparse chunks are typically kept as sorted arrays,+ * denser chunks as bitsets, and long consecutive runs as run containers.+ *+ * This hybrid representation aims to keep bitmaps compact while still+ * supporting fast membership tests, iteration, rank/select queries,+ * serialization, and set operations such as union, intersection, difference,+ * and symmetric difference.+ */++#ifndef ROARING_H+#define ROARING_H++#include <stdbool.h>+#include <stddef.h>  // for `size_t`+#include <stdint.h>++#include <roaring/roaring_types.h>++// Include other headers after roaring_types.h+#include <roaring/bitset/bitset.h>+#include <roaring/containers/containers.h>+#include <roaring/memory.h>+#include <roaring/portability.h>+#include <roaring/roaring_array.h>+#include <roaring/roaring_version.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace api {+#endif++typedef struct roaring_bitmap_s {+    roaring_array_t high_low_container;+} roaring_bitmap_t;++/**+ * Dynamically allocates a new bitmap (initially empty).+ * Returns NULL if the allocation fails.+ * Capacity is a performance hint for how many "containers" the data will need.+ * Client is responsible for calling `roaring_bitmap_free()`.+ */+roaring_bitmap_t *roaring_bitmap_create_with_capacity(uint32_t cap);++/**+ * Dynamically allocates a new bitmap (initially empty).+ * Returns NULL if the allocation fails.+ * Client is responsible for calling `roaring_bitmap_free()`.+ */+inline roaring_bitmap_t *roaring_bitmap_create(void) {+    return roaring_bitmap_create_with_capacity(0);+}++/**+ * Initialize a roaring bitmap structure in memory controlled by client.+ * Capacity is a performance hint for how many "containers" the data will need.+ * Can return false if auxiliary allocations fail when capacity greater than 0.+ */+bool roaring_bitmap_init_with_capacity(roaring_bitmap_t *r, uint32_t cap);++/**+ * Initialize a roaring bitmap structure in memory controlled by client.+ * The bitmap will be in a "clear" state, with no auxiliary allocations.+ * Since this performs no allocations, the function will not fail.+ */+inline void roaring_bitmap_init_cleared(roaring_bitmap_t *r) {+    roaring_bitmap_init_with_capacity(r, 0);+}++/**+ * Add all the values between min (included) and max (excluded) that are at a+ * distance k*step from min.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_from_range(uint64_t min, uint64_t max,+                                            uint32_t step);++/**+ * Creates a new bitmap from a pointer of uint32_t integers+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_of_ptr(size_t n_args, const uint32_t *vals);++/**+ * Check if the bitmap contains any shared containers.+ */+bool roaring_contains_shared(const roaring_bitmap_t *r);++/**+ * Unshare all shared containers.+ * Returns true if any unsharing was performed, false if there were no shared+ * containers.+ */+bool roaring_unshare_all(roaring_bitmap_t *r);++/*+ * Whether you want to use copy-on-write.+ * Saves memory and avoids copies, but needs more care in a threaded context.+ * Most users should ignore this flag.+ *+ * Note: If you do turn this flag to 'true', enabling COW, then ensure that you+ * do so for all of your bitmaps, since interactions between bitmaps with and+ * without COW is unsafe.+ *+ * When setting this flag to false, if any containers are shared, they+ * are unshared (cloned) immediately.+ */+inline bool roaring_bitmap_get_copy_on_write(const roaring_bitmap_t *r) {+    return r->high_low_container.flags & ROARING_FLAG_COW;+}+inline void roaring_bitmap_set_copy_on_write(roaring_bitmap_t *r, bool cow) {+    if (cow) {+        r->high_low_container.flags |= ROARING_FLAG_COW;+    } else {+        if (roaring_bitmap_get_copy_on_write(r)) {+            roaring_unshare_all(r);+        }+        r->high_low_container.flags &= ~ROARING_FLAG_COW;+    }+}++/**+ * Return a copy of the bitmap with all values shifted by offset.+ * The returned pointer may be NULL in case of errors. The caller is responsible+ * for freeing the return bitmap.+ */+roaring_bitmap_t *roaring_bitmap_add_offset(const roaring_bitmap_t *bm,+                                            int64_t offset);+/**+ * Describe the inner structure of the bitmap.+ */+void roaring_bitmap_printf_describe(const roaring_bitmap_t *r);++/**+ * Creates a new bitmap from a list of uint32_t integers+ *+ * This function is deprecated, use `roaring_bitmap_from` instead, which+ * doesn't require the number of elements to be passed in.+ *+ * @see roaring_bitmap_from+ */+CROARING_DEPRECATED roaring_bitmap_t *roaring_bitmap_of(size_t n, ...);++#ifdef __cplusplus+/**+ * Creates a new bitmap which contains all values passed in as arguments.+ *+ * To create a bitmap from a variable number of arguments, use the+ * `roaring_bitmap_of_ptr` function instead.+ */+// Use an immediately invoked closure, capturing by reference+// (in case __VA_ARGS__ refers to context outside the closure)+// Include a 0 at the beginning of the array to make the array length > 0+// (zero sized arrays are not valid in standard c/c++)+#define roaring_bitmap_from(...)                                              \+    [&]() {                                                                   \+        const uint32_t roaring_bitmap_from_array[] = {0, __VA_ARGS__};        \+        return roaring_bitmap_of_ptr((sizeof(roaring_bitmap_from_array) /     \+                                      sizeof(roaring_bitmap_from_array[0])) - \+                                         1,                                   \+                                     &roaring_bitmap_from_array[1]);          \+    }()+#else+/**+ * Creates a new bitmap which contains all values passed in as arguments.+ *+ * To create a bitmap from a variable number of arguments, use the+ * `roaring_bitmap_of_ptr` function instead.+ */+// While __VA_ARGS__ occurs twice in expansion, one of the times is in a sizeof+// expression, which is an unevaluated context, so it's even safe in the case+// where expressions passed have side effects (roaring64_bitmap_from(my_func(),+// ++i))+// Include a 0 at the beginning of the array to make the array length > 0+// (zero sized arrays are not valid in standard c/c++)+#define roaring_bitmap_from(...)                                             \+    roaring_bitmap_of_ptr(                                                   \+        (sizeof((const uint32_t[]){0, __VA_ARGS__}) / sizeof(uint32_t)) - 1, \+        &((const uint32_t[]){0, __VA_ARGS__})[1])+#endif++/**+ * Copies a bitmap (this does memory allocation).+ * The caller is responsible for memory management.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_copy(const roaring_bitmap_t *r);++/**+ * Copies a bitmap from src to dest. It is assumed that the pointer dest+ * is to an already allocated bitmap. The content of the dest bitmap is+ * freed/deleted.+ *+ * It might be preferable and simpler to call roaring_bitmap_copy except+ * that roaring_bitmap_overwrite can save on memory allocations.+ *+ * Returns true if successful, or false if there was an error. On failure,+ * the dest bitmap is left in a valid, empty state (even if it was not empty+ * before).+ */+bool roaring_bitmap_overwrite(roaring_bitmap_t *dest,+                              const roaring_bitmap_t *src);++/**+ * Print the content of the bitmap.+ */+void roaring_bitmap_printf(const roaring_bitmap_t *r);++/**+ * Computes the intersection between two bitmaps and returns new bitmap. The+ * caller is responsible for memory management.+ *+ * Performance hint: if you are computing the intersection between several+ * bitmaps, two-by-two, it is best to start with the smallest bitmap.+ * You may also rely on roaring_bitmap_and_inplace to avoid creating+ * many temporary bitmaps.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_and(const roaring_bitmap_t *r1,+                                     const roaring_bitmap_t *r2);++/**+ * Computes the size of the intersection between two bitmaps.+ */+uint64_t roaring_bitmap_and_cardinality(const roaring_bitmap_t *r1,+                                        const roaring_bitmap_t *r2);++/**+ * Check whether two bitmaps intersect.+ */+bool roaring_bitmap_intersect(const roaring_bitmap_t *r1,+                              const roaring_bitmap_t *r2);++/**+ * Check whether a bitmap and an open range intersect.+ * Values of y beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+bool roaring_bitmap_intersect_with_range(const roaring_bitmap_t *bm, uint64_t x,+                                         uint64_t y);++/**+ * Computes the Jaccard index between two bitmaps. (Also known as the Tanimoto+ * distance, or the Jaccard similarity coefficient)+ *+ * The Jaccard index is undefined if both bitmaps are empty.+ */+double roaring_bitmap_jaccard_index(const roaring_bitmap_t *r1,+                                    const roaring_bitmap_t *r2);++/**+ * Computes the size of the union between two bitmaps.+ */+uint64_t roaring_bitmap_or_cardinality(const roaring_bitmap_t *r1,+                                       const roaring_bitmap_t *r2);++/**+ * Computes the size of the difference (andnot) between two bitmaps.+ */+uint64_t roaring_bitmap_andnot_cardinality(const roaring_bitmap_t *r1,+                                           const roaring_bitmap_t *r2);++/**+ * Computes the size of the symmetric difference (xor) between two bitmaps.+ */+uint64_t roaring_bitmap_xor_cardinality(const roaring_bitmap_t *r1,+                                        const roaring_bitmap_t *r2);++/**+ * Inplace version of `roaring_bitmap_and()`, modifies r1+ * r1 == r2 is allowed.+ *+ * Performance hint: if you are computing the intersection between several+ * bitmaps, two-by-two, it is best to start with the smallest bitmap.+ */+void roaring_bitmap_and_inplace(roaring_bitmap_t *r1,+                                const roaring_bitmap_t *r2);++/**+ * Computes the union between two bitmaps and returns new bitmap. The caller is+ * responsible for memory management.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_or(const roaring_bitmap_t *r1,+                                    const roaring_bitmap_t *r2);++/**+ * Inplace version of `roaring_bitmap_or(), modifies r1.+ * TODO: decide whether r1 == r2 ok+ */+void roaring_bitmap_or_inplace(roaring_bitmap_t *r1,+                               const roaring_bitmap_t *r2);++/**+ * Compute the union of 'number' bitmaps.+ * Caller is responsible for freeing the result.+ * See also `roaring_bitmap_or_many_heap()`+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_or_many(size_t number,+                                         const roaring_bitmap_t **rs);++/**+ * Compute the union of 'number' bitmaps using a heap. This can sometimes be+ * faster than `roaring_bitmap_or_many() which uses a naive algorithm.+ * Caller is responsible for freeing the result.+ */+roaring_bitmap_t *roaring_bitmap_or_many_heap(uint32_t number,+                                              const roaring_bitmap_t **rs);++/**+ * Computes the symmetric difference (xor) between two bitmaps+ * and returns new bitmap. The caller is responsible for memory management.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_xor(const roaring_bitmap_t *r1,+                                     const roaring_bitmap_t *r2);++/**+ * Inplace version of roaring_bitmap_xor, modifies r1, r1 != r2.+ */+void roaring_bitmap_xor_inplace(roaring_bitmap_t *r1,+                                const roaring_bitmap_t *r2);++/**+ * Compute the xor of 'number' bitmaps.+ * Caller is responsible for freeing the result.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_xor_many(size_t number,+                                          const roaring_bitmap_t **rs);++/**+ * Computes the difference (andnot) between two bitmaps and returns new bitmap.+ * Caller is responsible for freeing the result.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_andnot(const roaring_bitmap_t *r1,+                                        const roaring_bitmap_t *r2);++/**+ * Inplace version of roaring_bitmap_andnot, modifies r1, r1 != r2.+ */+void roaring_bitmap_andnot_inplace(roaring_bitmap_t *r1,+                                   const roaring_bitmap_t *r2);++/**+ * TODO: consider implementing:+ *+ * "Compute the xor of 'number' bitmaps using a heap. This can sometimes be+ *  faster than roaring_bitmap_xor_many which uses a naive algorithm. Caller is+ *  responsible for freeing the result.""+ *+ * roaring_bitmap_t *roaring_bitmap_xor_many_heap(uint32_t number,+ *                                                const roaring_bitmap_t **rs);+ */++/**+ * Frees the memory.+ */+void roaring_bitmap_free(const roaring_bitmap_t *r);++/**+ * A bit of context usable with `roaring_bitmap_*_bulk()` functions+ *+ * Should be initialized with `{0}` (or `memset()` to all zeros).+ * Callers should treat it as an opaque type.+ *+ * A context may only be used with a single bitmap+ * (unless re-initialized to zero), and any modification to a bitmap+ * (other than modifications performed with `_bulk()` functions with the context+ * passed) will invalidate any contexts associated with that bitmap.+ */+typedef struct roaring_bulk_context_s {+    ROARING_CONTAINER_T *container;+    int idx;+    uint16_t key;+    uint8_t typecode;+} roaring_bulk_context_t;++/**+ * Add an item, using context from a previous insert for speed optimization.+ *+ * `context` will be used to store information between calls to make bulk+ * operations faster. `*context` should be zero-initialized before the first+ * call to this function.+ *+ * Modifying the bitmap in any way (other than `-bulk` suffixed functions)+ * will invalidate the stored context, calling this function with a non-zero+ * context after doing any modification invokes undefined behavior.+ *+ * In order to exploit this optimization, the caller should call this function+ * with values with the same "key" (high 16 bits of the value) consecutively.+ */+void roaring_bitmap_add_bulk(roaring_bitmap_t *r,+                             roaring_bulk_context_t *context, uint32_t val);++/**+ * Add value n_args from pointer vals, faster than repeatedly calling+ * `roaring_bitmap_add()`+ *+ * In order to exploit this optimization, the caller should attempt to keep+ * values with the same "key" (high 16 bits of the value) as consecutive+ * elements in `vals`+ */+void roaring_bitmap_add_many(roaring_bitmap_t *r, size_t n_args,+                             const uint32_t *vals);++/**+ * Add value x+ */+void roaring_bitmap_add(roaring_bitmap_t *r, uint32_t x);++/**+ * Add value x+ * Returns true if a new value was added, false if the value already existed.+ */+bool roaring_bitmap_add_checked(roaring_bitmap_t *r, uint32_t x);++/**+ * Add all values in range [min, max]+ */+void roaring_bitmap_add_range_closed(roaring_bitmap_t *r, uint32_t min,+                                     uint32_t max);++/**+ * Add all values in range [min, max)+ * Values of max beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+inline void roaring_bitmap_add_range(roaring_bitmap_t *r, uint64_t min,+                                     uint64_t max) {+    if (max > (uint64_t)UINT32_MAX + 1) {+        max = (uint64_t)UINT32_MAX + 1;+    }+    if (max <= min) {+        return;+    }+    roaring_bitmap_add_range_closed(r, (uint32_t)min, (uint32_t)(max - 1));+}++/**+ * Remove value x+ */+void roaring_bitmap_remove(roaring_bitmap_t *r, uint32_t x);++/**+ * Remove all values in range [min, max]+ */+void roaring_bitmap_remove_range_closed(roaring_bitmap_t *r, uint32_t min,+                                        uint32_t max);++/**+ * Remove all values in range [min, max)+ * Values of max beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+inline void roaring_bitmap_remove_range(roaring_bitmap_t *r, uint64_t min,+                                        uint64_t max) {+    if (max > (uint64_t)UINT32_MAX + 1) {+        max = (uint64_t)UINT32_MAX + 1;+    }+    if (max <= min) {+        return;+    }+    roaring_bitmap_remove_range_closed(r, (uint32_t)min, (uint32_t)(max - 1));+}++/**+ * Remove multiple values+ */+void roaring_bitmap_remove_many(roaring_bitmap_t *r, size_t n_args,+                                const uint32_t *vals);++/**+ * Remove value x+ * Returns true if a new value was removed, false if the value was not existing.+ */+bool roaring_bitmap_remove_checked(roaring_bitmap_t *r, uint32_t x);++/**+ * Check if value is present+ */+inline bool roaring_bitmap_contains(const roaring_bitmap_t *r, uint32_t val) {+    // For performance reasons, this function is inline and uses internal+    // functions directly.+#ifdef __cplusplus+    using namespace ::roaring::internal;+#endif+    const uint16_t hb = val >> 16;+    /*+     * the next function call involves a binary search and lots of branching.+     */+    int32_t i = ra_get_index(&r->high_low_container, hb);+    if (i < 0) return false;++    uint8_t typecode;+    // next call ought to be cheap+    container_t *container = ra_get_container_at_index(&r->high_low_container,+                                                       (uint16_t)i, &typecode);+    // rest might be a tad expensive, possibly involving another round of binary+    // search+    return container_contains(container, val & 0xFFFF, typecode);+}++/**+ * Check whether a range of values from range_start (included)+ * to range_end (excluded) is present+ * Values of range_end beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+bool roaring_bitmap_contains_range(const roaring_bitmap_t *r,+                                   uint64_t range_start, uint64_t range_end);++/**+ * Check whether a range of values from range_start (included)+ * to range_end (included) is present+ */+bool roaring_bitmap_contains_range_closed(const roaring_bitmap_t *r,+                                          uint32_t range_start,+                                          uint32_t range_end);++/**+ * Check if an items is present, using context from a previous insert or search+ * for speed optimization.+ *+ * `context` will be used to store information between calls to make bulk+ * operations faster. `*context` should be zero-initialized before the first+ * call to this function.+ *+ * Modifying the bitmap in any way (other than `-bulk` suffixed functions)+ * will invalidate the stored context, calling this function with a non-zero+ * context after doing any modification invokes undefined behavior.+ *+ * In order to exploit this optimization, the caller should call this function+ * with values with the same "key" (high 16 bits of the value) consecutively.+ */+bool roaring_bitmap_contains_bulk(const roaring_bitmap_t *r,+                                  roaring_bulk_context_t *context,+                                  uint32_t val);++/**+ * Get the cardinality of the bitmap (number of elements).+ */+uint64_t roaring_bitmap_get_cardinality(const roaring_bitmap_t *r);++/**+ * Returns the number of elements in the range [range_start, range_end).+ * Values of range_end beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+uint64_t roaring_bitmap_range_cardinality(const roaring_bitmap_t *r,+                                          uint64_t range_start,+                                          uint64_t range_end);++/**+ * Returns the number of elements in the range [range_start, range_end].+ */+uint64_t roaring_bitmap_range_cardinality_closed(const roaring_bitmap_t *r,+                                                 uint32_t range_start,+                                                 uint32_t range_end);+/**+ * Returns true if the bitmap is empty (cardinality is zero).+ */+bool roaring_bitmap_is_empty(const roaring_bitmap_t *r);++/**+ * Empties the bitmap.  It will have no auxiliary allocations (so if the bitmap+ * was initialized in client memory via roaring_bitmap_init(), then a call to+ * roaring_bitmap_clear() would be enough to "free" it)+ */+void roaring_bitmap_clear(roaring_bitmap_t *r);++/**+ * Convert the bitmap to a sorted array, output in `ans`.+ *+ * Caller is responsible to ensure that there is enough memory allocated, e.g.+ *+ *     ans = malloc(roaring_bitmap_get_cardinality(bitmap) * sizeof(uint32_t));+ */+void roaring_bitmap_to_uint32_array(const roaring_bitmap_t *r, uint32_t *ans);++/**+ * Store the bitmap to a bitset. This can be useful for people+ * who need the performance and simplicity of a standard bitset.+ * We assume that the input bitset is originally empty (does not+ * have any set bit).+ *+ *   bitset_t * out = bitset_create();+ *   // if the bitset has content in it, call "bitset_clear(out)"+ *   bool success = roaring_bitmap_to_bitset(mybitmap, out);+ *   // on failure, success will be false.+ *   // You can then query the bitset:+ *   bool is_present = bitset_get(out,  10011 );+ *   // you must free the memory:+ *   bitset_free(out);+ *+ */+bool roaring_bitmap_to_bitset(const roaring_bitmap_t *r, bitset_t *bitset);++/**+ * Writes the values of the bitmap in a range of 64-bit words, from+ * 64 * first_word (included) to 64 * (first_word + n_words) (excluded), as a+ * bitset in `words`: value 64 * (first_word + k) + i, if present, sets bit i+ * of words[k]. All n_words words are written (the caller does not need to zero+ * them). The cost is proportional to n_words plus the number of values and+ * runs in the range, not to the size of the bitmap.+ *+ * first_word + n_words must be at most 2^26 (that is, 2^32 / 64).+ *+ *   uint64_t words[16];+ *   // values 1024 to 2047, value 1024 + 64 * k + i in bit i of words[k]+ *   roaring_bitmap_word_range_to_bitset(mybitmap, 16, 16, words);+ */+void roaring_bitmap_word_range_to_bitset(const roaring_bitmap_t *r,+                                         uint32_t first_word, uint32_t n_words,+                                         uint64_t *words);++/**+ * Convert the bitmap to a sorted array from `offset` by `limit`, output in+ * `ans`.+ *+ * Caller is responsible to ensure that there is enough memory allocated, e.g.+ *+ *     ans = malloc(roaring_bitmap_get_cardinality(limit) * sizeof(uint32_t));+ *+ * This function always returns `true`+ *+ * For more control, see `roaring_uint32_iterator_skip` and+ * `roaring_uint32_iterator_read`, which can be used to e.g. tell how many+ * values were actually read.+ */+bool roaring_bitmap_range_uint32_array(const roaring_bitmap_t *r, size_t offset,+                                       size_t limit, uint32_t *ans);++/**+ * Remove run-length encoding even when it is more space efficient.+ * Return whether a change was applied.+ */+bool roaring_bitmap_remove_run_compression(roaring_bitmap_t *r);++/**+ * Convert array and bitmap containers to run containers when it is more+ * efficient; also convert from run containers when more space efficient.+ *+ * Adding or removing single values (for example with roaring_bitmap_add or+ * roaring_bitmap_remove) does not convert a run container back to an array or+ * bitmap container when its runs stop being the most compact encoding. Such a+ * container can then use more memory and serialized space than needed: in the+ * worst case, the runs of one 65536-value chunk can take about 128 KiB where a+ * bitmap container takes 8 KiB. Calling this function again after such+ * modifications recovers that space.+ *+ * Returns true if the result has at least one run container.+ * Additional savings might be possible by calling `shrinkToFit()`.+ */+bool roaring_bitmap_run_optimize(roaring_bitmap_t *r);++/**+ * If needed, reallocate memory to shrink the memory usage.+ * Returns the number of bytes saved.+ * Containers shared with other copy-on-write bitmaps are skipped.+ */+size_t roaring_bitmap_shrink_to_fit(roaring_bitmap_t *r);++/**+ * Write the bitmap to an output pointer, this output buffer should refer to+ * at least `roaring_bitmap_size_in_bytes(r)` allocated bytes.+ *+ * See `roaring_bitmap_portable_serialize()` if you want a format that's+ * compatible with Java and Go implementations.  This format can sometimes be+ * more space efficient than the portable form, e.g. when the data is sparse.+ *+ * Returns how many bytes written, should be `roaring_bitmap_size_in_bytes(r)`.+ *+ * When serializing data to a file, we recommend that you also use+ * checksums so that, at deserialization, you can be confident+ * that you are recovering the correct data.+ */+size_t roaring_bitmap_serialize(const roaring_bitmap_t *r, char *buf);++/**+ * Use with `roaring_bitmap_serialize()`.+ *+ * (See `roaring_bitmap_portable_deserialize()` if you want a format that's+ * compatible with Java and Go implementations).+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_deserialize(const void *buf);++/**+ * Load a bitmap from a serialized buffer safely (reading up to maxbytes).+ *+ * Use with `roaring_bitmap_serialize()`.+ *+ * (See `roaring_bitmap_portable_deserialize_safe()` if you want a format that's+ * compatible with Java and Go implementations).+ *+ * The difference with `roaring_bitmap_deserialize()` is that this function+ * is guaranteed to not read beyond the provided buffer. If the buffer is too+ * small, NULL is returned.+ *+ * The function itself is safe in the sense that it will not cause buffer+ * overflows: it will not read beyond the scope of the provided buffer+ * (buf,maxbytes).+ *+ * However, for correct operations, it is assumed that the bitmap+ * read was once serialized from a valid bitmap (i.e., it follows the format+ * specification). If you provided an incorrect input (garbage), then the bitmap+ * read may not be in a valid state and following operations may not lead to+ * sensible results (using it may cause crashes, or it may just give incoherent+ * answers). You can call roaring_bitmap_internal_validate to check the validity+ * of the bitmap if the source is untrusted. Only after calling+ * roaring_bitmap_internal_validate is the bitmap considered safe for use.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_deserialize_safe(const void *buf,+                                                  size_t maxbytes);++/**+ * How many bytes are required to serialize this bitmap (NOT compatible+ * with Java and Go versions)+ */+size_t roaring_bitmap_size_in_bytes(const roaring_bitmap_t *r);++/**+ * Read bitmap from a serialized buffer.+ * In case of failure, NULL is returned.+ *+ * This function is unsafe in the sense that if there is no valid serialized+ * bitmap at the pointer, then many bytes could be read, possibly causing a+ * buffer overflow. In other words, this routine assumes that `buf` points to a+ * complete, correctly formatted serialized bitmap and does not take a buffer+ * length argument that would let it enforce a read bound.+ *+ * Use this function only when the input buffer is already trusted, for example+ * because it comes from memory that was previously filled by+ * `roaring_bitmap_portable_serialize()` and whose size is known by some other+ * means. If the source is untrusted, truncated, or otherwise not guaranteed to+ * contain a valid serialized bitmap, prefer+ * `roaring_bitmap_portable_deserialize_safe()`.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_portable_deserialize(const char *buf);++/**+ * Read bitmap from a serialized buffer safely (reading up to maxbytes).+ * In case of failure, NULL is returned.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ *+ * The function itself is safe in the sense that it will not cause buffer+ * overflows: it will not read beyond the scope of the provided buffer+ * (buf,maxbytes).+ *+ * However, for correct operations, it is assumed that the bitmap+ * read was once serialized from a valid bitmap (i.e., it follows the format+ * specification). If you provided an incorrect input (garbage), then the bitmap+ * read may not be in a valid state and following operations may not lead to+ * sensible results. In particular, the serialized array containers need to be+ * in sorted order, and the run containers should be in sorted non-overlapping+ * order. This is is guaranteed to happen when serializing an existing bitmap,+ * but not for random inputs.+ *+ * If the source is untrusted, you should call+ * roaring_bitmap_internal_validate to check the validity of the+ * bitmap prior to using it. Only after calling roaring_bitmap_internal_validate+ * is the bitmap considered safe for use.+ *+ * We also recommend that you use checksums to check that serialized data+ * corresponds to the serialized bitmap. The CRoaring library does not provide+ * checksumming.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf,+                                                           size_t maxbytes);++/**+ * Read bitmap from a serialized buffer.+ * In case of failure, NULL is returned.+ *+ * Bitmap returned by this function can be used in all readonly contexts.+ * Bitmap must be freed as usual, by calling roaring_bitmap_free().+ * Underlying buffer must not be freed or modified while it backs any bitmaps.+ *+ * The function is unsafe: a buffer overflow may occur if buf does not point to+ * a valid serialized bitmap.+ *+ * Container payloads are used where they sit in the buffer, so they may be+ * unaligned whatever the alignment of buf. Every access path is either SIMD+ * with unaligned loads or marked `CROARING_ALLOW_UNALIGNED`.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ *+ * Returns NULL on a big-endian system (e.g., a mainframe IBM s390x). The+ * portable format is little-endian on every host, and this function uses the+ * container payloads where they sit rather than converting them, so there is+ * no correct in-place view of them there. Use+ * `roaring_bitmap_portable_deserialize_safe()`, which converts as it copies.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_portable_deserialize_frozen(const char *buf);++/**+ * Check how many bytes would be read (up to maxbytes) at this pointer if there+ * is a bitmap, returns zero if there is no valid bitmap.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ */+size_t roaring_bitmap_portable_deserialize_size(const char *buf,+                                                size_t maxbytes);++/**+ * How many bytes are required to serialize this bitmap.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ */+size_t roaring_bitmap_portable_size_in_bytes(const roaring_bitmap_t *r);++/**+ * Write a bitmap to a char buffer.  The output buffer should refer to at least+ * `roaring_bitmap_portable_size_in_bytes(r)` bytes of allocated memory.+ *+ * Returns how many bytes were written which should match+ * `roaring_bitmap_portable_size_in_bytes(r)`.+ *+ * This is meant to be compatible with the Java and Go versions:+ * https://github.com/RoaringBitmap/RoaringFormatSpec+ *+ * When serializing data to a file, we recommend that you also use+ * checksums so that, at deserialization, you can be confident+ * that you are recovering the correct data.+ */+size_t roaring_bitmap_portable_serialize(const roaring_bitmap_t *r, char *buf);++/*+ * "Frozen" serialization format imitates memory layout of roaring_bitmap_t.+ * Deserialized bitmap is a constant view of the underlying buffer.+ * This significantly reduces amount of allocations and copying required during+ * deserialization.+ * It can be used with memory mapped files.+ * Example can be found in benchmarks/frozen_benchmark.c+ *+ *         [#####] const roaring_bitmap_t *+ *          | | |+ *     +----+ | +-++ *     |      |   |+ * [#####################################] underlying buffer+ *+ * Note that because frozen serialization format imitates C memory layout+ * of roaring_bitmap_t, it is not fixed. It is different on big/little endian+ * platforms and can be changed in future.+ */++/**+ * Returns number of bytes required to serialize bitmap using frozen format.+ */+size_t roaring_bitmap_frozen_size_in_bytes(const roaring_bitmap_t *r);++/**+ * Serializes bitmap using frozen format.+ * Buffer size must be at least roaring_bitmap_frozen_size_in_bytes().+ *+ * This function is endian-sensitive. If you have a big-endian system (e.g., a+ * mainframe IBM s390x), the data format is going to be big-endian and not+ * compatible with little-endian systems. This is not a bug, it is by design,+ *since the format imitates C memory layout+ *+ * When serializing data to a file, we recommend that you also use+ * checksums so that, at deserialization, you can be confident+ * that you are recovering the correct data.+ */+void roaring_bitmap_frozen_serialize(const roaring_bitmap_t *r, char *buf);++/**+ * Creates constant bitmap that is a view of a given buffer.+ * Buffer data should have been written by `roaring_bitmap_frozen_serialize()`+ * Its beginning must also be aligned by 32 bytes.+ * Length must be equal exactly to `roaring_bitmap_frozen_size_in_bytes()`.+ * In case of failure, NULL is returned.+ *+ * Bitmap returned by this function can be used in all readonly contexts.+ * Bitmap must be freed as usual, by calling roaring_bitmap_free().+ * Underlying buffer must not be freed or modified while it backs any bitmaps.+ *+ * This function is endian-sensitive. If you have a big-endian system (e.g., a+ * mainframe IBM s390x), the data format is going to be big-endian and not+ * compatible with little-endian systems. This is not a bug, it is by design,+ *since the format imitates C memory layout of roaring_bitmap_t.+ */+const roaring_bitmap_t *roaring_bitmap_frozen_view(const char *buf,+                                                   size_t length);++/**+ * Iterate over the bitmap elements. The function iterator is called once for+ * all the values with ptr (can be NULL) as the second parameter of each call.+ *+ * `roaring_iterator` is simply a pointer to a function that returns bool+ * (true means that the iteration should continue while false means that it+ * should stop), and takes (uint32_t,void*) as inputs.+ *+ * Returns true if the roaring_iterator returned true throughout (so that all+ * data points were necessarily visited).+ *+ * Iteration is ordered: from the smallest to the largest elements.+ */+bool roaring_iterate(const roaring_bitmap_t *r, roaring_iterator iterator,+                     void *ptr);++/**+ * Like `roaring_iterate`, but the 32-bit values are widened to 64 bits by+ * adding `high_bits` (shifted into the upper 32 bits) before being passed to+ * the iterator. This is used to build 64-bit iteration on top of 32-bit+ * bitmaps. `ptr` (can be NULL) is forwarded as the second argument of each+ * call.+ *+ * Returns true if the iterator returned true throughout (so that all values+ * were necessarily visited).+ */+bool roaring_iterate64(const roaring_bitmap_t *r, roaring_iterator64 iterator,+                       uint64_t high_bits, void *ptr);++/**+ * Return true if the two bitmaps contain the same elements.+ */+bool roaring_bitmap_equals(const roaring_bitmap_t *r1,+                           const roaring_bitmap_t *r2);++/**+ * Return true if all the elements of r1 are also in r2.+ */+bool roaring_bitmap_is_subset(const roaring_bitmap_t *r1,+                              const roaring_bitmap_t *r2);++/**+ * Return true if all the elements of r1 are also in r2, and r2 is strictly+ * greater than r1.+ */+bool roaring_bitmap_is_strict_subset(const roaring_bitmap_t *r1,+                                     const roaring_bitmap_t *r2);++/**+ * (For expert users who seek high performance.)+ *+ * Computes the union between two bitmaps and returns new bitmap. The caller is+ * responsible for memory management.+ *+ * The lazy version defers some computations such as the maintenance of the+ * cardinality counts. Thus you must call `roaring_bitmap_repair_after_lazy()`+ * after executing "lazy" computations.+ *+ * It is safe to repeatedly call roaring_bitmap_lazy_or_inplace on the result.+ *+ * `bitsetconversion` is a flag which determines whether container-container+ * operations force a bitset conversion.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_lazy_or(const roaring_bitmap_t *r1,+                                         const roaring_bitmap_t *r2,+                                         const bool bitsetconversion);++/**+ * (For expert users who seek high performance.)+ *+ * Inplace version of roaring_bitmap_lazy_or, modifies r1.+ *+ * `bitsetconversion` is a flag which determines whether container-container+ * operations force a bitset conversion.+ */+void roaring_bitmap_lazy_or_inplace(roaring_bitmap_t *r1,+                                    const roaring_bitmap_t *r2,+                                    const bool bitsetconversion);++/**+ * (For expert users who seek high performance.)+ *+ * Execute maintenance on a bitmap created from `roaring_bitmap_lazy_or()`+ * or modified with `roaring_bitmap_lazy_or_inplace()`.+ */+void roaring_bitmap_repair_after_lazy(roaring_bitmap_t *r1);++/**+ * Computes the symmetric difference between two bitmaps and returns new bitmap.+ * The caller is responsible for memory management.+ *+ * The lazy version defers some computations such as the maintenance of the+ * cardinality counts. Thus you must call `roaring_bitmap_repair_after_lazy()`+ * after executing "lazy" computations.+ *+ * It is safe to repeatedly call `roaring_bitmap_lazy_xor_inplace()` on+ * the result.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_lazy_xor(const roaring_bitmap_t *r1,+                                          const roaring_bitmap_t *r2);++/**+ * (For expert users who seek high performance.)+ *+ * Inplace version of roaring_bitmap_lazy_xor, modifies r1. r1 != r2+ */+void roaring_bitmap_lazy_xor_inplace(roaring_bitmap_t *r1,+                                     const roaring_bitmap_t *r2);++/**+ * Compute the negation of the bitmap in the interval [range_start, range_end).+ * The number of negated values is range_end - range_start.+ * Areas outside the range are passed through unchanged.+ * Values of range_end beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_flip(const roaring_bitmap_t *r1,+                                      uint64_t range_start, uint64_t range_end);++/**+ * Compute the negation of the bitmap in the interval [range_start, range_end].+ * The number of negated values is range_end - range_start + 1.+ * Areas outside the range are passed through unchanged.+ * The returned pointer may be NULL in case of errors.+ */+roaring_bitmap_t *roaring_bitmap_flip_closed(const roaring_bitmap_t *x1,+                                             uint32_t range_start,+                                             uint32_t range_end);+/**+ * compute (in place) the negation of the roaring bitmap within a specified+ * interval: [range_start, range_end). The number of negated values is+ * range_end - range_start.+ * Areas outside the range are passed through unchanged.+ * Values of range_end beyond UINT32_MAX + 1 are treated as UINT32_MAX + 1.+ */+void roaring_bitmap_flip_inplace(roaring_bitmap_t *r1, uint64_t range_start,+                                 uint64_t range_end);++/**+ * compute (in place) the negation of the roaring bitmap within a specified+ * interval: [range_start, range_end]. The number of negated values is+ * range_end - range_start + 1.+ * Areas outside the range are passed through unchanged.+ */+void roaring_bitmap_flip_inplace_closed(roaring_bitmap_t *r1,+                                        uint32_t range_start,+                                        uint32_t range_end);++/**+ * Selects the element at index 'rank' where the smallest element is at index 0.+ * If the size of the roaring bitmap is strictly greater than rank, then this+ * function returns true and sets element to the element of given rank.+ * Otherwise, it returns false.+ */+bool roaring_bitmap_select(const roaring_bitmap_t *r, uint32_t rank,+                           uint32_t *element);++/**+ * roaring_bitmap_rank returns the number of integers that are smaller or equal+ * to x. Thus if x is the first element, this function will return 1. If+ * x is smaller than the smallest element, this function will return 0.+ *+ * The indexing convention differs between roaring_bitmap_select and+ * roaring_bitmap_rank: roaring_bitmap_select refers to the smallest value+ * as having index 0, whereas roaring_bitmap_rank returns 1 when ranking+ * the smallest value.+ */+uint64_t roaring_bitmap_rank(const roaring_bitmap_t *r, uint32_t x);++/**+ * roaring_bitmap_rank_many is an `Bulk` version of `roaring_bitmap_rank`+ * it puts rank value of each element in `[begin .. end)` to `ans[]`+ *+ * the values in `[begin .. end)` must be sorted in Ascending order;+ * Caller is responsible to ensure that there is enough memory allocated, e.g.+ *+ *     ans = malloc((end-begin) * sizeof(uint64_t));+ */+void roaring_bitmap_rank_many(const roaring_bitmap_t *r, const uint32_t *begin,+                              const uint32_t *end, uint64_t *ans);++/**+ * Returns the index of x in the given roaring bitmap.+ * If the roaring bitmap doesn't contain x , this function will return -1.+ * The difference with rank function is that this function will return -1 when x+ * is not the element of roaring bitmap, but the rank function will return a+ * non-negative number.+ */+int64_t roaring_bitmap_get_index(const roaring_bitmap_t *r, uint32_t x);++/**+ * Returns the smallest value in the set, or UINT32_MAX if the set is empty.+ */+uint32_t roaring_bitmap_minimum(const roaring_bitmap_t *r);++/**+ * Returns the greatest value in the set, or 0 if the set is empty.+ */+uint32_t roaring_bitmap_maximum(const roaring_bitmap_t *r);++/**+ * (For advanced users.)+ *+ * Collect statistics about the bitmap, see roaring_types.h for+ * a description of roaring_statistics_t+ */+void roaring_bitmap_statistics(const roaring_bitmap_t *r,+                               roaring_statistics_t *stat);++/**+ * Perform internal consistency checks. Returns true if the bitmap is+ * consistent. It may be useful to call this after deserializing bitmaps from+ * untrusted sources. If roaring_bitmap_internal_validate returns true, then the+ * bitmap should be consistent and can be trusted not to cause crashes or memory+ * corruption.+ *+ * Note that some operations intentionally leave bitmaps in an inconsistent+ * state temporarily, for example, `roaring_bitmap_lazy_*` functions, until+ * `roaring_bitmap_repair_after_lazy` is called.+ *+ * If reason is non-null, it will be set to a string describing the first+ * inconsistency found if any.+ */+bool roaring_bitmap_internal_validate(const roaring_bitmap_t *r,+                                      const char **reason);++/*********************+* What follows is code use to iterate through values in a roaring bitmap++roaring_bitmap_t *r =...+roaring_uint32_iterator_t i;+roaring_iterator_create(r, &i);+while(i.has_value) {+  printf("value = %d\n", i.current_value);+  roaring_uint32_iterator_advance(&i);+}+roaring_uint32_iterator_free(&i);++Obviously, if you modify the underlying bitmap, the iterator+becomes invalid. So don't.+*/++/**+ * A struct used to keep iterator state. Users should only access+ * `current_value` and `has_value`, the rest of the type should be treated as+ * opaque.+ */+typedef struct roaring_uint32_iterator_s {+    const roaring_bitmap_t *parent;        // Owner+    const ROARING_CONTAINER_T *container;  // Current container+    uint8_t typecode;                      // Typecode of current container+    int32_t container_index;               // Current container index+    uint32_t highbits;                     // High 16 bits of the current value+    roaring_container_iterator_t container_it;++    uint32_t current_value;+    bool has_value;+} roaring_uint32_iterator_t;++/**+ * Initialize an iterator object that can be used to iterate through the values.+ * If there is a  value, then this iterator points to the first value and+ * `it->has_value` is true. The value is in `it->current_value`.+ */+void roaring_iterator_init(const roaring_bitmap_t *r,+                           roaring_uint32_iterator_t *newit);++/** DEPRECATED, use `roaring_iterator_init`. */+CROARING_DEPRECATED static inline void roaring_init_iterator(+    const roaring_bitmap_t *r, roaring_uint32_iterator_t *newit) {+    roaring_iterator_init(r, newit);+}++/**+ * Initialize an iterator object that can be used to iterate through the values.+ * If there is a value, then this iterator points to the last value and+ * `it->has_value` is true. The value is in `it->current_value`.+ */+void roaring_iterator_init_last(const roaring_bitmap_t *r,+                                roaring_uint32_iterator_t *newit);++/** DEPRECATED, use `roaring_iterator_init_last`. */+CROARING_DEPRECATED static inline void roaring_init_iterator_last(+    const roaring_bitmap_t *r, roaring_uint32_iterator_t *newit) {+    roaring_iterator_init_last(r, newit);+}++/**+ * Create an iterator object that can be used to iterate through the values.+ * Caller is responsible for calling `roaring_uint32_iterator_free()`.+ *+ * The iterator is initialized (this function calls `roaring_iterator_init()`)+ * If there is a value, then this iterator points to the first value and+ * `it->has_value` is true.  The value is in `it->current_value`.+ */+roaring_uint32_iterator_t *roaring_iterator_create(const roaring_bitmap_t *r);++/** DEPRECATED, use `roaring_iterator_create`. */+CROARING_DEPRECATED static inline roaring_uint32_iterator_t *+roaring_create_iterator(const roaring_bitmap_t *r) {+    return roaring_iterator_create(r);+}++/**+ * Advance the iterator. If there is a new value, then `it->has_value` is true.+ * The new value is in `it->current_value`. Values are traversed in increasing+ * orders. For convenience, returns `it->has_value`.+ *+ * Once `it->has_value` is false, `roaring_uint32_iterator_advance` should not+ * be called on the iterator again. Calling `roaring_uint32_iterator_previous`+ * is allowed.+ */+bool roaring_uint32_iterator_advance(roaring_uint32_iterator_t *it);++/** DEPRECATED, use `roaring_uint32_iterator_advance`. */+CROARING_DEPRECATED static inline bool roaring_advance_uint32_iterator(+    roaring_uint32_iterator_t *it) {+    return roaring_uint32_iterator_advance(it);+}++/**+ * Decrement the iterator. If there's a new value, then `it->has_value` is true.+ * The new value is in `it->current_value`. Values are traversed in decreasing+ * order. For convenience, returns `it->has_value`.+ *+ * Once `it->has_value` is false, `roaring_uint32_iterator_previous` should not+ * be called on the iterator again. Calling `roaring_uint32_iterator_advance` is+ * allowed.+ */+bool roaring_uint32_iterator_previous(roaring_uint32_iterator_t *it);++/** DEPRECATED, use `roaring_uint32_iterator_previous`. */+CROARING_DEPRECATED static inline bool roaring_previous_uint32_iterator(+    roaring_uint32_iterator_t *it) {+    return roaring_uint32_iterator_previous(it);+}++/**+ * Move the iterator to the first value >= `val`. If there is a such a value,+ * then `it->has_value` is true. The new value is in `it->current_value`.+ * For convenience, returns `it->has_value`.+ */+bool roaring_uint32_iterator_move_equalorlarger(roaring_uint32_iterator_t *it,+                                                uint32_t val);++/** DEPRECATED, use `roaring_uint32_iterator_move_equalorlarger`. */+CROARING_DEPRECATED static inline bool+roaring_move_uint32_iterator_equalorlarger(roaring_uint32_iterator_t *it,+                                           uint32_t val) {+    return roaring_uint32_iterator_move_equalorlarger(it, val);+}++/**+ * Creates a copy of an iterator.+ * Caller must free it.+ */+roaring_uint32_iterator_t *roaring_uint32_iterator_copy(+    const roaring_uint32_iterator_t *it);++/** DEPRECATED, use `roaring_uint32_iterator_copy`. */+CROARING_DEPRECATED static inline roaring_uint32_iterator_t *+roaring_copy_uint32_iterator(const roaring_uint32_iterator_t *it) {+    return roaring_uint32_iterator_copy(it);+}++/**+ * Free memory following `roaring_iterator_create()`+ */+void roaring_uint32_iterator_free(roaring_uint32_iterator_t *it);++/** DEPRECATED, use `roaring_uint32_iterator_free`. */+CROARING_DEPRECATED static inline void roaring_free_uint32_iterator(+    roaring_uint32_iterator_t *it) {+    roaring_uint32_iterator_free(it);+}++/**+ * Reads next ${count} values from iterator into user-supplied ${buf}.+ * Returns the number of read elements.+ * This number can be smaller than ${count}, which means that iterator is+ * drained.+ *+ * This function satisfies semantics of iteration and can be used together with+ * other iterator functions.+ *  - first value is copied from ${it}->current_value+ *  - after function returns, iterator is positioned at the next element+ */+uint32_t roaring_uint32_iterator_read(roaring_uint32_iterator_t *it,+                                      uint32_t *buf, uint32_t count);++/** DEPRECATED, use `roaring_uint32_iterator_read`. */+CROARING_DEPRECATED static inline uint32_t roaring_read_uint32_iterator(+    roaring_uint32_iterator_t *it, uint32_t *buf, uint32_t count) {+    return roaring_uint32_iterator_read(it, buf, count);+}++/**+ * Reads previous ${count} values from iterator into user-supplied ${buf}.+ * Returns the number of read elements.+ * This number can be smaller than ${count}, which means that iterator is+ * drained.+ *+ * Values are written in descending order: buf[0] is the highest (current)+ * value, buf[ret-1] is the lowest value read.+ *+ * This function satisfies semantics of reverse iteration and can be used+ * together with other iterator functions.+ *  - first value is copied from ${it}->current_value+ *  - after function returns, iterator is positioned at the previous element+ */+uint32_t roaring_uint32_iterator_read_backward(roaring_uint32_iterator_t *it,+                                               uint32_t *buf, uint32_t count);++/**+ * Skip the next ${count} values from iterator.+ * Returns the number of values actually skipped.+ * The number can be smaller than ${count}, which means that iterator is+ * drained.+ *+ * This function is equivalent to calling `roaring_uint32_iterator_advance()`+ * ${count} times but is much more efficient.+ */+uint32_t roaring_uint32_iterator_skip(roaring_uint32_iterator_t *it,+                                      uint32_t count);++/**+ * Skip the previous ${count} values from iterator (move backwards).+ * Returns the number of values actually skipped backwards.+ * The number can be smaller than ${count}, which means that iterator reached+ * the beginning.+ *+ * This function is equivalent to calling `roaring_uint32_iterator_previous()`+ * ${count} times but is much more efficient.+ */+uint32_t roaring_uint32_iterator_skip_backward(roaring_uint32_iterator_t *it,+                                               uint32_t count);++typedef struct roaring_uint32_range_closed_s {+    uint32_t min;+    uint32_t max;+} roaring_uint32_range_closed_t;++/**+ * Reads next ${count} ranges from iterator into user-supplied ${buf}.+ * A range is defined as a maximal interval of consecutive values.+ * For example, the set {1,2,3,5,6} contains two ranges: [1..3] and [5..6].+ * Each range is represented as a struct {min,max}, both endpoints included.+ * Consecutive values that span internal container boundaries are merged into+ * a single range.+ *+ * Returns the number of read ranges.+ * This number can be smaller than ${count}, which means that the iterator is+ * drained.+ *+ * This function satisfies the semantics of iteration and can be used together+ * with other iterator functions.+ *  - first range will start with ${it}->current_value+ *  - after the function returns, the iterator is positioned at the next element+ *    after the end of the last returned range, or ${it}->has_value is false if+ *    the bitmap is exhausted.+ */+size_t roaring_uint32_iterator_read_ranges(roaring_uint32_iterator_t *it,+                                           roaring_uint32_range_closed_t *buf,+                                           size_t count);++/**+ * Reads previous ${count} ranges from iterator into user-supplied ${buf}.+ * A range is defined as a maximal interval of consecutive values.+ * For example, the set {1,2,3,5,6} contains two ranges: [1..3] and [5..6].+ * Each range is represented as a struct {min,max}, both endpoints included.+ * Consecutive values that span internal container boundaries are merged into+ * a single range.+ *+ * Returns the number of read ranges.+ * This number can be smaller than ${count}, which means that the iterator is+ * drained.+ *+ * Ranges are returned in reverse order, e.g. the first range returned is the+ * highest range (ending at the current value)+ *+ * This function satisfies the semantics of reverse iteration and can be used+ * together with other iterator functions.+ *  - first range will end with ${it}->current_value+ *  - after the function returns, the iterator is positioned at the element+ *    before the beginning of the last returned range, or ${it}->has_value is+ *    false if the bitmap is exhausted.+ */+size_t roaring_uint32_iterator_read_prev_ranges(+    roaring_uint32_iterator_t *it, roaring_uint32_range_closed_t *buf,+    size_t count);++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace api {+#endif++#endif /* ROARING_H */++#ifdef __cplusplus+/**+ * Best practices for C++ headers is to avoid polluting global scope.+ * But for C compatibility when just `roaring.h` is included building as+ * C++, default to global access for the C public API.+ *+ * BUT when `roaring.hh` is included instead, it sets this flag.  That way+ * explicit namespacing must be used to get the C functions.+ *+ * This is outside the include guard so that if you include BOTH headers,+ * the order won't matter; you still get the global definitions.+ */+#if !defined(ROARING_API_NOT_IN_GLOBAL_NAMESPACE)+using namespace ::roaring::api;+#endif+#endif++// roaring64 will include roaring.h, but we would+// prefer to avoid having our users include roaring64.h+// in addition to roaring.h.+#include <roaring/roaring64.h>
+ cbits/CRoaring/include/roaring/roaring64.h view
@@ -0,0 +1,966 @@+/*+ * roaring64.h+ *+ * This file declares the 64-bit Roaring bitmap API. A roaring64 bitmap stores+ * sets of 64-bit unsigned integers by partitioning the value space by high+ * bits and using Roaring containers for the lower bits inside each partition.+ * This keeps the structure compact while preserving fast membership tests,+ * insertions, iteration, and set operations over large sparse integer sets.+ */+#ifndef ROARING64_H+#define ROARING64_H++#include <stdbool.h>+#include <stddef.h>+#include <stdint.h>++#include <roaring/memory.h>+#include <roaring/portability.h>+#include <roaring/roaring.h>+#include <roaring/roaring_types.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace api {+#endif++/** An opaque 64-bit Roaring bitmap. Create one with `roaring64_bitmap_create()`+ * and release it with `roaring64_bitmap_free()`. */+typedef struct roaring64_bitmap_s roaring64_bitmap_t;+/** Internal leaf type, exposed only for use inside `roaring64_bulk_context_t`.+ * Callers should treat it as opaque. */+typedef uint64_t roaring64_leaf_t;+/** An opaque iterator over a 64-bit bitmap. See `roaring64_iterator_create()`.+ */+typedef struct roaring64_iterator_s roaring64_iterator_t;++/** The leading members of `roaring64_iterator_t`, so that+ * `roaring64_iterator_value()` and `roaring64_iterator_has_value()` can be+ * read without a call. The iterator itself stays opaque; do not declare one of+ * these, and do not rely on the layout beyond these two members. */+typedef struct roaring64_iterator_public_s {+    uint64_t value;+    bool has_value;+} roaring64_iterator_public_t;++/**+ * A bit of context usable with `roaring64_bitmap_*_bulk()` functions.+ *+ * Should be initialized with `{0}` (or `memset()` to all zeros).+ * Callers should treat it as an opaque type.+ *+ * A context may only be used with a single bitmap (unless re-initialized to+ * zero), and any modification to a bitmap (other than modifications performed+ * with `_bulk()` functions with the context passed) will invalidate any+ * contexts associated with that bitmap.+ */+typedef struct roaring64_bulk_context_s {+    uint8_t high_bytes[6];+    roaring64_leaf_t *leaf;+} roaring64_bulk_context_t;++/**+ * Dynamically allocates a new bitmap (initially empty).+ * Client is responsible for calling `roaring64_bitmap_free()`.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_create(void);+void roaring64_bitmap_free(roaring64_bitmap_t *r);++/**+ * Returns a copy of a bitmap.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_copy(const roaring64_bitmap_t *r);++/**+ * Copies a bitmap from src to dest. It is assumed that the pointer dest+ * is to an already allocated bitmap. The content of the dest bitmap is+ * freed/deleted.+ *+ * It might be preferable and simpler to call roaring64_bitmap_copy except+ * that roaring64_bitmap_overwrite can save on memory allocations.+ *+ */+void roaring64_bitmap_overwrite(roaring64_bitmap_t *dest,+                                const roaring64_bitmap_t *src);++/**+ * Creates a new bitmap of a pointer to N 64-bit integers.+ */+roaring64_bitmap_t *roaring64_bitmap_of_ptr(size_t n_args,+                                            const uint64_t *vals);++#ifdef __cplusplus+/**+ * Creates a new bitmap which contains all values passed in as arguments.+ *+ * To create a bitmap from a variable number of arguments, use the+ * `roaring64_bitmap_of_ptr` function instead.+ */+// Use an immediately invoked closure, capturing by reference+// (in case __VA_ARGS__ refers to context outside the closure)+// Include a 0 at the beginning of the array to make the array length > 0+// (zero sized arrays are not valid in standard c/c++)+#define roaring64_bitmap_from(...)                                       \+    [&]() {                                                              \+        const uint64_t roaring64_bitmap_from_array[] = {0, __VA_ARGS__}; \+        return roaring64_bitmap_of_ptr(                                  \+            (sizeof(roaring64_bitmap_from_array) /                       \+             sizeof(roaring64_bitmap_from_array[0])) -                   \+                1,                                                       \+            &roaring64_bitmap_from_array[1]);                            \+    }()+#else+/**+ * Creates a new bitmap which contains all values passed in as arguments.+ *+ * To create a bitmap from a variable number of arguments, use the+ * `roaring64_bitmap_of_ptr` function instead.+ */+// While __VA_ARGS__ occurs twice in expansion, one of the times is in a sizeof+// expression, which is an unevaluated context, so it's even safe in the case+// where expressions passed have side effects (roaring64_bitmap_from(my_func(),+// ++i))+// Include a 0 at the beginning of the array to make the array length > 0+// (zero sized arrays are not valid in standard c/c++)+#define roaring64_bitmap_from(...)                                           \+    roaring64_bitmap_of_ptr(                                                 \+        (sizeof((const uint64_t[]){0, __VA_ARGS__}) / sizeof(uint64_t)) - 1, \+        &((const uint64_t[]){0, __VA_ARGS__})[1])+#endif++/**+ * Create a new bitmap by moving containers from a 32 bit roaring bitmap.+ * Containers shared with other copy-on-write bitmaps are copied.+ *+ * After calling this function, the original bitmap will be empty, and the+ * returned bitmap will contain all the values from the original bitmap.+ */+roaring64_bitmap_t *roaring64_bitmap_move_from_roaring32(roaring_bitmap_t *r);++/**+ * Create a new bitmap containing all the values in [min, max) that are at a+ * distance k*step from min.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_from_range(uint64_t min, uint64_t max,+                                                uint64_t step);++/**+ * Adds the provided value to the bitmap.+ */+void roaring64_bitmap_add(roaring64_bitmap_t *r, uint64_t val);++/**+ * Adds the provided value to the bitmap.+ * Returns true if a new value was added, false if the value already existed.+ */+bool roaring64_bitmap_add_checked(roaring64_bitmap_t *r, uint64_t val);++/**+ * Add an item, using context from a previous insert for faster insertion.+ *+ * `context` will be used to store information between calls to make bulk+ * operations faster. `*context` should be zero-initialized before the first+ * call to this function.+ *+ * Modifying the bitmap in any way (other than `-bulk` suffixed functions)+ * will invalidate the stored context, calling this function with a non-zero+ * context after doing any modification invokes undefined behavior.+ *+ * In order to exploit this optimization, the caller should call this function+ * with values with the same high 48 bits of the value consecutively.+ */+void roaring64_bitmap_add_bulk(roaring64_bitmap_t *r,+                               roaring64_bulk_context_t *context, uint64_t val);++/**+ * Add `n_args` values from `vals`, faster than repeatedly calling+ * `roaring64_bitmap_add()`+ *+ * In order to exploit this optimization, the caller should attempt to keep+ * values with the same high 48 bits of the value as consecutive elements in+ * `vals`.+ */+void roaring64_bitmap_add_many(roaring64_bitmap_t *r, size_t n_args,+                               const uint64_t *vals);++/**+ * Add all values in range [min, max).+ */+void roaring64_bitmap_add_range(roaring64_bitmap_t *r, uint64_t min,+                                uint64_t max);++/**+ * Add all values in range [min, max].+ */+void roaring64_bitmap_add_range_closed(roaring64_bitmap_t *r, uint64_t min,+                                       uint64_t max);++/**+ * Removes a value from the bitmap if present.+ */+void roaring64_bitmap_remove(roaring64_bitmap_t *r, uint64_t val);++/**+ * Removes a value from the bitmap if present, returns true if the value was+ * removed and false if the value was not present.+ */+bool roaring64_bitmap_remove_checked(roaring64_bitmap_t *r, uint64_t val);++/**+ * Remove an item, using context from a previous insert for faster removal.+ *+ * `context` will be used to store information between calls to make bulk+ * operations faster. `*context` should be zero-initialized before the first+ * call to this function.+ *+ * Modifying the bitmap in any way (other than `-bulk` suffixed functions)+ * will invalidate the stored context, calling this function with a non-zero+ * context after doing any modification invokes undefined behavior.+ *+ * In order to exploit this optimization, the caller should call this function+ * with values with the same high 48 bits of the value consecutively.+ */+void roaring64_bitmap_remove_bulk(roaring64_bitmap_t *r,+                                  roaring64_bulk_context_t *context,+                                  uint64_t val);++/**+ * Remove `n_args` values from `vals`, faster than repeatedly calling+ * `roaring64_bitmap_remove()`+ *+ * In order to exploit this optimization, the caller should attempt to keep+ * values with the same high 48 bits of the value as consecutive elements in+ * `vals`.+ */+void roaring64_bitmap_remove_many(roaring64_bitmap_t *r, size_t n_args,+                                  const uint64_t *vals);++/**+ * Remove all values in range [min, max).+ */+void roaring64_bitmap_remove_range(roaring64_bitmap_t *r, uint64_t min,+                                   uint64_t max);++/**+ * Remove all values in range [min, max].+ */+void roaring64_bitmap_remove_range_closed(roaring64_bitmap_t *r, uint64_t min,+                                          uint64_t max);++/**+ * Empties the bitmap.+ */+void roaring64_bitmap_clear(roaring64_bitmap_t *r);++/**+ * Returns true if the provided value is present.+ */+bool roaring64_bitmap_contains(const roaring64_bitmap_t *r, uint64_t val);++/**+ * Returns true if all values in the range [min, max) are present.+ */+bool roaring64_bitmap_contains_range(const roaring64_bitmap_t *r, uint64_t min,+                                     uint64_t max);++/**+ * Returns true if all values in the range [min, max] are present.+ */+bool roaring64_bitmap_contains_range_closed(const roaring64_bitmap_t *r,+                                            uint64_t min, uint64_t max);++/**+ * Check if an item is present using context from a previous insert or search+ * for faster search.+ *+ * `context` will be used to store information between calls to make bulk+ * operations faster. `*context` should be zero-initialized before the first+ * call to this function.+ *+ * Modifying the bitmap in any way (other than `-bulk` suffixed functions)+ * will invalidate the stored context, calling this function with a non-zero+ * context after doing any modification invokes undefined behavior.+ *+ * In order to exploit this optimization, the caller should call this function+ * with values with the same high 48 bits of the value consecutively.+ */+bool roaring64_bitmap_contains_bulk(const roaring64_bitmap_t *r,+                                    roaring64_bulk_context_t *context,+                                    uint64_t val);++/**+ * Selects the element at index 'rank' where the smallest element is at index 0.+ * If the size of the bitmap is strictly greater than rank, then this function+ * returns true and sets element to the element of given rank. Otherwise, it+ * returns false.+ */+bool roaring64_bitmap_select(const roaring64_bitmap_t *r, uint64_t rank,+                             uint64_t *element);++/**+ * Returns the number of integers that are smaller or equal to x. Thus if x is+ * the first element, this function will return 1. If x is smaller than the+ * smallest element, this function will return 0.+ *+ * The indexing convention differs between roaring64_bitmap_select and+ * roaring64_bitmap_rank: roaring_bitmap64_select refers to the smallest value+ * as having index 0, whereas roaring64_bitmap_rank returns 1 when ranking+ * the smallest value.+ */+uint64_t roaring64_bitmap_rank(const roaring64_bitmap_t *r, uint64_t val);++/**+ * Returns true if the given value is in the bitmap, and sets `out_index` to the+ * (0-based) index of the value in the bitmap. Returns false if the value is not+ * in the bitmap.+ */+bool roaring64_bitmap_get_index(const roaring64_bitmap_t *r, uint64_t val,+                                uint64_t *out_index);++/**+ * Returns the number of values in the bitmap.+ */+uint64_t roaring64_bitmap_get_cardinality(const roaring64_bitmap_t *r);++/**+ * Returns the number of elements in the range [min, max).+ */+uint64_t roaring64_bitmap_range_cardinality(const roaring64_bitmap_t *r,+                                            uint64_t min, uint64_t max);++/**+ * Returns the number of elements in the range [min, max]+ */+uint64_t roaring64_bitmap_range_closed_cardinality(const roaring64_bitmap_t *r,+                                                   uint64_t min, uint64_t max);++/**+ * Returns true if the bitmap is empty (cardinality is zero).+ */+bool roaring64_bitmap_is_empty(const roaring64_bitmap_t *r);++/**+ * Returns the smallest value in the set, or UINT64_MAX if the set is empty.+ */+uint64_t roaring64_bitmap_minimum(const roaring64_bitmap_t *r);++/**+ * Returns the largest value in the set, or 0 if empty.+ */+uint64_t roaring64_bitmap_maximum(const roaring64_bitmap_t *r);++/**+ * Remove run-length encoding even when it is more space efficient.+ * Return whether a change was applied.+ */+bool roaring64_bitmap_remove_run_compression(roaring64_bitmap_t *r);++/**+ * Convert array and bitmap containers to run containers when it is more+ * efficient; also convert from run containers when more space efficient.+ *+ * Adding or removing single values (for example with roaring64_bitmap_add or+ * roaring64_bitmap_remove) does not convert a run container back to an array or+ * bitmap container when its runs stop being the most compact encoding. Such a+ * container can then use more memory and serialized space than needed: in the+ * worst case, the runs of one 65536-value chunk can take about 128 KiB where a+ * bitmap container takes 8 KiB. Calling this function again after such+ * modifications recovers that space.+ *+ * Returns true if the result has at least one run container.+ */+bool roaring64_bitmap_run_optimize(roaring64_bitmap_t *r);++/**+ * Shrinks internal arrays to eliminate any unused capacity. Returns the number+ * of bytes freed.+ */+size_t roaring64_bitmap_shrink_to_fit(roaring64_bitmap_t *r);++/**+ *  (For advanced users.)+ * Collect statistics about the bitmap+ */+void roaring64_bitmap_statistics(const roaring64_bitmap_t *r,+                                 roaring64_statistics_t *stat);++/**+ * Perform internal consistency checks.+ *+ * Returns true if the bitmap is consistent. It may be useful to call this+ * after deserializing bitmaps from untrusted sources. If+ * roaring64_bitmap_internal_validate returns true, then the bitmap is+ * consistent and can be trusted not to cause crashes or memory corruption.+ *+ * If reason is non-null, it will be set to a string describing the first+ * inconsistency found if any.+ */+bool roaring64_bitmap_internal_validate(const roaring64_bitmap_t *r,+                                        const char **reason);++/**+ * Return true if the two bitmaps contain the same elements.+ */+bool roaring64_bitmap_equals(const roaring64_bitmap_t *r1,+                             const roaring64_bitmap_t *r2);++/**+ * Return true if all the elements of r1 are also in r2.+ */+bool roaring64_bitmap_is_subset(const roaring64_bitmap_t *r1,+                                const roaring64_bitmap_t *r2);++/**+ * Return true if all the elements of r1 are also in r2, and r2 is strictly+ * greater than r1.+ */+bool roaring64_bitmap_is_strict_subset(const roaring64_bitmap_t *r1,+                                       const roaring64_bitmap_t *r2);++/**+ * Computes the intersection between two bitmaps and returns new bitmap. The+ * caller is responsible for free-ing the result.+ *+ * Performance hint: if you are computing the intersection between several+ * bitmaps, two-by-two, it is best to start with the smallest bitmaps. You may+ * also rely on roaring64_bitmap_and_inplace to avoid creating many temporary+ * bitmaps.+ *+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_and(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2);++/**+ * Computes the size of the intersection between two bitmaps.+ */+uint64_t roaring64_bitmap_and_cardinality(const roaring64_bitmap_t *r1,+                                          const roaring64_bitmap_t *r2);++/**+ * In-place version of `roaring64_bitmap_and()`, modifies `r1`. `r1` and `r2`+ * are allowed to be equal.+ *+ * Performance hint: if you are computing the intersection between several+ * bitmaps, two-by-two, it is best to start with the smallest bitmaps.+ */+void roaring64_bitmap_and_inplace(roaring64_bitmap_t *r1,+                                  const roaring64_bitmap_t *r2);++/**+ * Check whether two bitmaps intersect.+ */+bool roaring64_bitmap_intersect(const roaring64_bitmap_t *r1,+                                const roaring64_bitmap_t *r2);++/**+ * Check whether a bitmap intersects the range [min, max).+ */+bool roaring64_bitmap_intersect_with_range(const roaring64_bitmap_t *r,+                                           uint64_t min, uint64_t max);++/**+ * Computes the Jaccard index between two bitmaps. (Also known as the Tanimoto+ * distance, or the Jaccard similarity coefficient)+ *+ * The Jaccard index is undefined if both bitmaps are empty.+ */+double roaring64_bitmap_jaccard_index(const roaring64_bitmap_t *r1,+                                      const roaring64_bitmap_t *r2);++/**+ * Computes the union between two bitmaps and returns new bitmap. The caller is+ * responsible for free-ing the result.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_or(const roaring64_bitmap_t *r1,+                                        const roaring64_bitmap_t *r2);++/**+ * Computes the size of the union between two bitmaps.+ */+uint64_t roaring64_bitmap_or_cardinality(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2);++/**+ * In-place version of `roaring64_bitmap_or(), modifies `r1`.+ */+void roaring64_bitmap_or_inplace(roaring64_bitmap_t *r1,+                                 const roaring64_bitmap_t *r2);++/**+ * Computes the symmetric difference (xor) between two bitmaps and returns a new+ * bitmap. The caller is responsible for free-ing the result.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_xor(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2);++/**+ * Computes the size of the symmetric difference (xor) between two bitmaps.+ */+uint64_t roaring64_bitmap_xor_cardinality(const roaring64_bitmap_t *r1,+                                          const roaring64_bitmap_t *r2);++/**+ * In-place version of `roaring64_bitmap_xor()`, modifies `r1`. `r1` and `r2`+ * are not allowed to be equal (that would result in an empty bitmap).+ */+void roaring64_bitmap_xor_inplace(roaring64_bitmap_t *r1,+                                  const roaring64_bitmap_t *r2);++/**+ * Computes the difference (andnot) between two bitmaps and returns a new+ * bitmap. The caller is responsible for free-ing the result.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_andnot(const roaring64_bitmap_t *r1,+                                            const roaring64_bitmap_t *r2);++/**+ * Computes the size of the difference (andnot) between two bitmaps.+ */+uint64_t roaring64_bitmap_andnot_cardinality(const roaring64_bitmap_t *r1,+                                             const roaring64_bitmap_t *r2);++/**+ * In-place version of `roaring64_bitmap_andnot()`, modifies `r1`. `r1` and `r2`+ * are not allowed to be equal (that would result in an empty bitmap).+ */+void roaring64_bitmap_andnot_inplace(roaring64_bitmap_t *r1,+                                     const roaring64_bitmap_t *r2);++/**+ * Compute the negation of the bitmap in the interval [min, max).+ * The number of negated values is `max - min`. Areas outside the range are+ * passed through unchanged.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_flip(const roaring64_bitmap_t *r,+                                          uint64_t min, uint64_t max);++/**+ * Compute the negation of the bitmap in the interval [min, max].+ * The number of negated values is `max - min + 1`. Areas outside the range are+ * passed through unchanged.+ * The returned pointer may be NULL in case of errors.+ */+roaring64_bitmap_t *roaring64_bitmap_flip_closed(const roaring64_bitmap_t *r,+                                                 uint64_t min, uint64_t max);++/**+ * In-place version of `roaring64_bitmap_flip`. Compute the negation of the+ * bitmap in the interval [min, max). The number of negated values is `max -+ * min`. Areas outside the range are passed through unchanged.+ */+void roaring64_bitmap_flip_inplace(roaring64_bitmap_t *r, uint64_t min,+                                   uint64_t max);+/**+ * In-place version of `roaring64_bitmap_flip_closed`. Compute the negation of+ * the bitmap in the interval [min, max]. The number of negated values is `max -+ * min + 1`. Areas outside the range are passed through unchanged.+ */+void roaring64_bitmap_flip_closed_inplace(roaring64_bitmap_t *r, uint64_t min,+                                          uint64_t max);+/**+ * Return a copy of the bitmap with all values shifted by offset.+ *+ * If `positive` is true, the shift is added, otherwise subtracted. Values that+ * overflow or underflow uint64_t are dropped. The caller is responsible for+ * freeing the returned bitmap.+ */+roaring64_bitmap_t *roaring64_bitmap_add_offset_signed(+    const roaring64_bitmap_t *r, bool positive, uint64_t offset);++/**+ * Return a copy of the bitmap with all values shifted up by offset.+ *+ * Values that overflow or underflow uint64_t are dropped. The caller is+ * responsible for freeing the returned bitmap.+ */+static inline roaring64_bitmap_t *roaring64_bitmap_add_offset(+    const roaring64_bitmap_t *r, uint64_t offset) {+    return roaring64_bitmap_add_offset_signed(r, true, offset);+}++/**+ * Return a copy of the bitmap with all values shifted down by offset.+ *+ * Values that overflow or underflow uint64_t are dropped. The caller is+ * responsible for freeing the returned bitmap.+ */+static inline roaring64_bitmap_t *roaring64_bitmap_sub_offset(+    const roaring64_bitmap_t *r, uint64_t offset) {+    return roaring64_bitmap_add_offset_signed(r, false, offset);+}++/**+ * How many bytes are required to serialize this bitmap.+ *+ * This is meant to be compatible with other languages:+ * https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+ */+size_t roaring64_bitmap_portable_size_in_bytes(const roaring64_bitmap_t *r);++/**+ * Write a bitmap to a buffer. The output buffer should refer to at least+ * `roaring64_bitmap_portable_size_in_bytes(r)` bytes of allocated memory.+ *+ * Returns how many bytes were written, which should match+ * `roaring64_bitmap_portable_size_in_bytes(r)`.+ *+ * This is meant to be compatible with other languages:+ * https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+ *+ * When serializing data to a file, we recommend that you also use+ * checksums so that, at deserialization, you can be confident+ * that you are recovering the correct data.+ */+size_t roaring64_bitmap_portable_serialize(const roaring64_bitmap_t *r,+                                           char *buf);+/**+ * Check how many bytes would be read (up to maxbytes) at this pointer if there+ * is a valid bitmap, returns zero if there is no valid bitmap.+ *+ * This is meant to be compatible with other languages+ * https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+ */+size_t roaring64_bitmap_portable_deserialize_size(const char *buf,+                                                  size_t maxbytes);++/**+ * Read a bitmap from a serialized buffer (reading up to maxbytes).+ * In case of failure, NULL is returned.+ *+ * This is meant to be compatible with other languages+ * https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+ *+ * The function itself is safe in the sense that it will not cause buffer+ * overflows: it will not read beyond the scope of the provided buffer+ * (buf,maxbytes).+ *+ * However, for correct operations, it is assumed that the bitmap+ * read was once serialized from a valid bitmap (i.e., it follows the format+ * specification). If you provided an incorrect input (garbage), then the bitmap+ * read may not be in a valid state and following operations may not lead to+ * sensible results. In particular, the serialized array containers need to be+ * in sorted order, and the run containers should be in sorted non-overlapping+ * order. This is is guaranteed to happen when serializing an existing bitmap,+ * but not for random inputs.+ *+ * If the source is untrusted, you should call+ * roaring64_bitmap_internal_validate to check the validity of the+ * bitmap prior to using it. Only after calling+ * roaring64_bitmap_internal_validate is the bitmap considered safe for use.+ *+ * We also recommend that you use checksums to check that serialized data+ * corresponds to the serialized bitmap. The CRoaring library does not provide+ * checksumming.+ */+roaring64_bitmap_t *roaring64_bitmap_portable_deserialize_safe(const char *buf,+                                                               size_t maxbytes);++/**+ * Read a bitmap from a portable serialized buffer as a read-only view of the+ * container payloads. Headers and the ART index are allocated; bitset/array/run+ * payloads alias `buf` and are not copied.+ *+ * In case of failure, NULL is returned. The function will not read beyond+ * `maxbytes`.+ *+ * The returned bitmap must only be used in a readonly manner. It must be+ * freed with `roaring64_bitmap_free()`. The backing buffer must outlive the+ * bitmap and must not be freed or modified while it backs it. Calling any+ * mutating function on the result is undefined behavior: its container array+ * and headers live in a single allocation, so growing it would reallocate an+ * interior pointer.+ *+ * The function itself is safe in the sense that it will not read beyond+ * (buf, maxbytes). However, as with+ * `roaring64_bitmap_portable_deserialize_safe()`, a bitmap read from garbage+ * may not be in a valid state, and subsequent operations on it may not lead+ * to sensible results: array containers must be sorted, and run containers+ * sorted and non-overlapping, which is guaranteed only when the input came+ * from a real serialized bitmap.+ *+ * If the source is untrusted, you should call+ * `roaring64_bitmap_internal_validate` on the result before using it. Only+ * after that is the bitmap considered safe for use. We also recommend+ * checksumming the serialized data; CRoaring does not provide checksumming.+ *+ * Returns NULL on a big-endian system (e.g., a mainframe IBM s390x). The+ * portable format is little-endian and this function uses the payload bytes+ * where they sit, so there is no correct in-place view of them there; use+ * `roaring64_bitmap_portable_deserialize_safe()`, which converts as it copies.+ *+ * Container payloads are used where they sit in the buffer, so they may be+ * unaligned. Every access path is either SIMD with unaligned loads or marked+ * `CROARING_ALLOW_UNALIGNED`.+ */+roaring64_bitmap_t *roaring64_bitmap_portable_deserialize_frozen(+    const char *buf, size_t maxbytes);++/**+ * Returns the number of bytes required to serialize this bitmap in a "frozen"+ * format. This is not compatible with any other serialization formats.+ *+ * `roaring64_bitmap_shrink_to_fit()` must be called before this method.+ */+size_t roaring64_bitmap_frozen_size_in_bytes(const roaring64_bitmap_t *r);++/**+ * Serializes the bitmap in a "frozen" format. The given buffer must be at least+ * `roaring64_bitmap_frozen_size_in_bytes()` in size. Returns the number of+ * bytes used for serialization.+ *+ * `roaring64_bitmap_shrink_to_fit()` must be called before this method.+ *+ * The frozen format is optimized for speed of (de)serialization, as well as+ * allowing the user to create a bitmap based on a memory mapped file, which is+ * possible because the format mimics the memory layout of the bitmap.+ *+ * Because the format mimics the memory layout of the bitmap, the format is not+ * fixed across releases of Roaring Bitmaps, and may change in future releases.+ *+ * This function is endian-sensitive. If you have a big-endian system (e.g., a+ * mainframe IBM s390x), the data format is going to be big-endian and not+ * compatible with little-endian systems. This is not a bug, it is by design,+ * since the format imitates C memory layout of roaring64_bitmap_t.+ */+size_t roaring64_bitmap_frozen_serialize(const roaring64_bitmap_t *r,+                                         char *buf);++/**+ * Creates a readonly bitmap that is a view of the given buffer. The buffer+ * must be created with `roaring64_bitmap_frozen_serialize()`, and must be+ * aligned by 64 bytes.+ *+ * Returns NULL if deserialization fails.+ *+ * The returned bitmap must only be used in a readonly manner. The bitmap must+ * be freed using `roaring64_bitmap_free()` as normal. The backing buffer must+ * only be freed after the bitmap.+ *+ * This function is endian-sensitive. If you have a big-endian system (e.g., a+ * mainframe IBM s390x), the data format is going to be big-endian and not+ * compatible with little-endian systems. This is not a bug, it is by design,+ * since the format imitates C memory layout of roaring64_bitmap_t.+ */+roaring64_bitmap_t *roaring64_bitmap_frozen_view(const char *buf,+                                                 size_t maxbytes);++/**+ * Iterate over the bitmap elements. The function `iterator` is called once for+ * all the values with `ptr` (can be NULL) as the second parameter of each call.+ *+ * `roaring_iterator64` is simply a pointer to a function that returns a bool+ * and takes `(uint64_t, void*)` as inputs. True means that the iteration should+ * continue, while false means that it should stop.+ *+ * Returns true if the `roaring64_iterator` returned true throughout (so that+ * all data points were necessarily visited).+ *+ * Iteration is ordered from the smallest to the largest elements.+ */+bool roaring64_bitmap_iterate(const roaring64_bitmap_t *r,+                              roaring_iterator64 iterator, void *ptr);++/**+ * Convert the bitmap to a sorted array `out`.+ *+ * Caller is responsible to ensure that there is enough memory allocated, e.g.+ * ```+ * out = malloc(roaring64_bitmap_get_cardinality(bitmap) * sizeof(uint64_t));+ * ```+ */+void roaring64_bitmap_to_uint64_array(const roaring64_bitmap_t *r,+                                      uint64_t *out);++/**+ * Create an iterator object that can be used to iterate through the values.+ * Caller is responsible for calling `roaring64_iterator_free()`.+ *+ * The iterator is initialized. If there is a value, then this iterator points+ * to the first value and `roaring64_iterator_has_value()` returns true. The+ * value can be retrieved with `roaring64_iterator_value()`.+ */+roaring64_iterator_t *roaring64_iterator_create(const roaring64_bitmap_t *r);++/**+ * Create an iterator object that can be used to iterate through the values.+ * Caller is responsible for calling `roaring64_iterator_free()`.+ *+ * The iterator is initialized. If there is a value, then this iterator points+ * to the last value and `roaring64_iterator_has_value()` returns true. The+ * value can be retrieved with `roaring64_iterator_value()`.+ */+roaring64_iterator_t *roaring64_iterator_create_last(+    const roaring64_bitmap_t *r);++/**+ * Re-initializes an existing iterator. Functionally the same as+ * `roaring64_iterator_create` without a allocation.+ */+void roaring64_iterator_reinit(const roaring64_bitmap_t *r,+                               roaring64_iterator_t *it);++/**+ * Re-initializes an existing iterator. Functionally the same as+ * `roaring64_iterator_create_last` without a allocation.+ */+void roaring64_iterator_reinit_last(const roaring64_bitmap_t *r,+                                    roaring64_iterator_t *it);++/**+ * Creates a copy of the iterator. Caller is responsible for calling+ * `roaring64_iterator_free()` on the resulting iterator.+ */+roaring64_iterator_t *roaring64_iterator_copy(const roaring64_iterator_t *it);++/**+ * Free the iterator.+ */+void roaring64_iterator_free(roaring64_iterator_t *it);++/**+ * Returns true if the iterator currently points to a value. If so, calling+ * `roaring64_iterator_value()` returns the value.+ *+ * A pointer to a structure, suitably converted, points to its initial member+ * (C17 6.7.2.1p15), and `roaring64_iterator_public_t` is the initial member of+ * `roaring64_iterator_t`, so this reads the field directly.+ */+inline bool roaring64_iterator_has_value(const roaring64_iterator_t *it) {+    return ((const roaring64_iterator_public_t *)it)->has_value;+}++/**+ * Returns the value the iterator currently points to. Should only be called if+ * `roaring64_iterator_has_value()` returns true.+ */+inline uint64_t roaring64_iterator_value(const roaring64_iterator_t *it) {+    return ((const roaring64_iterator_public_t *)it)->value;+}++/**+ * Advance the iterator. If there is a new value, then+ * `roaring64_iterator_has_value()` returns true. Values are traversed in+ * increasing order. For convenience, returns the result of+ * `roaring64_iterator_has_value()`.+ *+ * Once this returns false, `roaring64_iterator_advance` should not be called on+ * the iterator again. Calling `roaring64_iterator_previous` is allowed.+ */+bool roaring64_iterator_advance(roaring64_iterator_t *it);++/**+ * Decrement the iterator. If there is a new value, then+ * `roaring64_iterator_has_value()` returns true. Values are traversed in+ * decreasing order. For convenience, returns the result of+ * `roaring64_iterator_has_value()`.+ *+ * Once this returns false, `roaring64_iterator_previous` should not be called+ * on the iterator again. Calling `roaring64_iterator_advance` is allowed.+ */+bool roaring64_iterator_previous(roaring64_iterator_t *it);++/**+ * Move the iterator to the first value greater than or equal to `val`, if it+ * exists at or after the current position of the iterator. If there is a new+ * value, then `roaring64_iterator_has_value()` returns true. Values are+ * traversed in increasing order. For convenience, returns the result of+ * `roaring64_iterator_has_value()`.+ */+bool roaring64_iterator_move_equalorlarger(roaring64_iterator_t *it,+                                           uint64_t val);++/**+ * Reads up to `count` values from the iterator into the given `buf`. Returns+ * the number of elements read. The number of elements read can be smaller than+ * `count`, which means that there are no more elements in the bitmap.+ *+ * This function can be used together with other iterator functions.+ */+uint64_t roaring64_iterator_read(roaring64_iterator_t *it, uint64_t *buf,+                                 uint64_t count);++/**+ * Reads previous ${count} values from iterator into user-supplied ${buf}.+ * Returns the number of read elements.+ * This number can be smaller than ${count}, which means that iterator is+ * drained.+ *+ * Values are written in descending order: buf[0] is the highest (current)+ * value, buf[ret-1] is the lowest value read.+ *+ * This function satisfies semantics of reverse iteration and can be used+ * together with other iterator functions.+ *  - first value is copied from the current iterator value+ *  - after function returns, iterator is positioned at the previous element+ */+uint64_t roaring64_iterator_read_backward(roaring64_iterator_t *it,+                                          uint64_t *buf, uint64_t count);++typedef struct roaring64_range_closed_s {+    uint64_t min;+    uint64_t max;+} roaring64_range_closed_t;++/**+ * Reads next ${count} ranges from iterator into user-supplied ${buf}.+ * A range is defined as a maximal interval of consecutive values.+ * For example, the set {1,2,3,5,6} contains two ranges: [1..3] and [5..6].+ * Each range is represented as a struct {min,max}, both endpoints included.+ * Consecutive values that span internal container boundaries are merged into+ * a single range.+ *+ * Returns the number of read ranges.+ * This number can be smaller than ${count}, which means that the iterator is+ * drained.+ *+ * This function can be used together with other iterator functions.+ *  - first range will start with the current iterator value+ *  - after the function returns, the iterator is positioned at the next element+ *    after the end of the last returned range, or has_value is false if+ *    the bitmap is exhausted.+ */+size_t roaring64_iterator_read_ranges(roaring64_iterator_t *it,+                                      roaring64_range_closed_t *buf,+                                      size_t count);++/**+ * Reads previous ${count} ranges from iterator into user-supplied ${buf}.+ * A range is defined as a maximal interval of consecutive values.+ * For example, the set {1,2,3,5,6} contains two ranges: [1..3] and [5..6].+ * Each range is represented as a struct {min,max}, both endpoints included.+ * Consecutive values that span internal container boundaries are merged into+ * a single range.+ *+ * Returns the number of read ranges.+ * This number can be smaller than ${count}, which means that the iterator is+ * drained.+ *+ * Ranges are returned in reverse order, e.g. the first range returned is the+ * highest range (ending at the current value).+ *+ * This function can be used together with other iterator functions.+ *  - first range will end with the current iterator value+ *  - after the function returns, the iterator is positioned at the element+ *    before the beginning of the last returned range, or has_value is false if+ *    the bitmap is exhausted.+ */+size_t roaring64_iterator_read_prev_ranges(roaring64_iterator_t *it,+                                           roaring64_range_closed_t *buf,+                                           size_t count);++#ifdef __cplusplus+}  // extern "C"+}  // namespace roaring+}  // namespace api+#endif++#endif /* ROARING64_H */
+ cbits/CRoaring/include/roaring/roaring_array.h view
@@ -0,0 +1,323 @@+/*+ * roaring_array.h+ *+ * This file declares the roaring_array helper structure and the operations+ * used to manage it. A roaring array is the top-level index used by a 32-bit+ * Roaring bitmap: it stores sorted 16-bit high keys alongside the container+ * pointers and type codes associated with each key.+ *+ * In effect, it is the directory that maps each populated 16-bit chunk of the+ * 32-bit value space to the container holding that chunk's low 16-bit values.+ * The functions in this header handle allocation, lookup, insertion,+ * replacement, copying, serialization support, and structural updates on that+ * directory.+ */+#ifndef INCLUDE_ROARING_ARRAY_H+#define INCLUDE_ROARING_ARRAY_H++#include <assert.h>+#include <stdbool.h>+#include <stdint.h>++#include <roaring/array_util.h>+#include <roaring/containers/containers.h>  // get_writable_copy_if_shared()++#ifdef __cplusplus+extern "C" {+namespace roaring {++// Note: in pure C++ code, you should avoid putting `using` in header files+using api::roaring_array_t;++namespace internal {+#endif++enum {+    SERIAL_COOKIE_NO_RUNCONTAINER = 12346,+    SERIAL_COOKIE = 12347,+    FROZEN_COOKIE = 13766,+    NO_OFFSET_THRESHOLD = 4+};++/**+ * Create a new roaring array+ */+roaring_array_t *ra_create(void);++/**+ * Initialize an existing roaring array with the specified capacity (in number+ * of containers)+ */+bool ra_init_with_capacity(roaring_array_t *new_ra, uint32_t cap);++/**+ * Initialize with zero capacity+ */+void ra_init(roaring_array_t *t);++/**+ * Copies this roaring array, we assume that dest is not initialized+ */+bool ra_copy(const roaring_array_t *source, roaring_array_t *dest,+             bool copy_on_write);++/*+ * Shrinks the capacity, returns the number of bytes saved.+ */+int ra_shrink_to_fit(roaring_array_t *ra);++/**+ * Copies this roaring array, we assume that dest is initialized+ */+bool ra_overwrite(const roaring_array_t *source, roaring_array_t *dest,+                  bool copy_on_write);++/**+ * Frees the memory used by a roaring array+ */+void ra_clear(roaring_array_t *r);++/**+ * Frees the memory used by a roaring array, but does not free the containers+ */+void ra_clear_without_containers(roaring_array_t *r);++/**+ * Frees just the containers+ */+void ra_clear_containers(roaring_array_t *ra);++/**+ * Get the index corresponding to a 16-bit key+ */+inline int32_t ra_get_index(const roaring_array_t *ra, uint16_t x) {+    if ((ra->size == 0) || ra->keys[ra->size - 1] == x) return ra->size - 1;+    return binarySearch(ra->keys, (int32_t)ra->size, x);+}++/**+ * Retrieves the container at index i, filling in the typecode+ */+inline container_t *ra_get_container_at_index(const roaring_array_t *ra,+                                              uint16_t i, uint8_t *typecode) {+    *typecode = ra->typecodes[i];+    return ra->containers[i];+}++/**+ * Retrieves the key at index i+ */+inline uint16_t ra_get_key_at_index(const roaring_array_t *ra, uint16_t i) {+    return ra->keys[i];+}++/**+ * Add a new key-value pair at index i+ */+void ra_insert_new_key_value_at(roaring_array_t *ra, int32_t i, uint16_t key,+                                container_t *c, uint8_t typecode);++/**+ * Append a new key-value pair+ */+void ra_append(roaring_array_t *ra, uint16_t key, container_t *c,+               uint8_t typecode);++/**+ * Append a new key-value pair to ra, cloning (in COW sense) a value from sa+ * at index index+ */+void ra_append_copy(roaring_array_t *ra, const roaring_array_t *sa,+                    uint16_t index, bool copy_on_write);++/**+ * Append new key-value pairs to ra, cloning (in COW sense)  values from sa+ * at indexes+ * [start_index, end_index)+ */+void ra_append_copy_range(roaring_array_t *ra, const roaring_array_t *sa,+                          int32_t start_index, int32_t end_index,+                          bool copy_on_write);++/** appends from sa to ra, ending with the greatest key that is+ * is less or equal stopping_key+ */+void ra_append_copies_until(roaring_array_t *ra, const roaring_array_t *sa,+                            uint16_t stopping_key, bool copy_on_write);++/** appends from sa to ra, starting with the smallest key that is+ * is strictly greater than before_start+ */++void ra_append_copies_after(roaring_array_t *ra, const roaring_array_t *sa,+                            uint16_t before_start, bool copy_on_write);++/**+ * Move the key-value pairs to ra from sa at indexes+ * [start_index, end_index), old array should not be freed+ * (use ra_clear_without_containers)+ **/+void ra_append_move_range(roaring_array_t *ra, roaring_array_t *sa,+                          int32_t start_index, int32_t end_index);+/**+ * Append new key-value pairs to ra,  from sa at indexes+ * [start_index, end_index)+ */+void ra_append_range(roaring_array_t *ra, roaring_array_t *sa,+                     int32_t start_index, int32_t end_index,+                     bool copy_on_write);++/**+ * Set the container at the corresponding index using the specified+ * typecode.+ */+inline void ra_set_container_at_index(const roaring_array_t *ra, int32_t i,+                                      container_t *c, uint8_t typecode) {+    assert(i < ra->size);+    ra->containers[i] = c;+    ra->typecodes[i] = typecode;+}++container_t *ra_get_container(roaring_array_t *ra, uint16_t x,+                              uint8_t *typecode);++/**+ * If needed, increase the capacity of the array so that it can fit k values+ * (at+ * least);+ */+bool extend_array(roaring_array_t *ra, int32_t k);++inline int32_t ra_get_size(const roaring_array_t *ra) { return ra->size; }++static inline int32_t ra_advance_until(const roaring_array_t *ra, uint16_t x,+                                       int32_t pos) {+    return advanceUntil(ra->keys, pos, ra->size, x);+}++int32_t ra_advance_until_freeing(roaring_array_t *ra, uint16_t x, int32_t pos);++void ra_downsize(roaring_array_t *ra, int32_t new_length);++inline void ra_replace_key_and_container_at_index(roaring_array_t *ra,+                                                  int32_t i, uint16_t key,+                                                  container_t *c,+                                                  uint8_t typecode) {+    assert(i < ra->size);++    ra->keys[i] = key;+    ra->containers[i] = c;+    ra->typecodes[i] = typecode;+}++// write set bits to an array+void ra_to_uint32_array(const roaring_array_t *ra, uint32_t *ans);++/**+ * write a bitmap to a buffer. This is meant to be compatible with+ * the+ * Java and Go versions. Return the size in bytes of the serialized+ * output (which should be ra_portable_size_in_bytes(ra)).+ */+size_t ra_portable_serialize(const roaring_array_t *ra, char *buf);++/**+ * read a bitmap from a serialized version. This is meant to be compatible+ * with the Java and Go versions.+ * maxbytes  indicates how many bytes available from buf.+ * When the function returns true, roaring_array_t is populated with the data+ * and *readbytes indicates how many bytes were read. In all cases, if the+ * function returns true, then maxbytes >= *readbytes.+ */+bool ra_portable_deserialize(roaring_array_t *ra, const char *buf,+                             const size_t maxbytes, size_t *readbytes);++/**+ * Quickly checks whether there is a serialized bitmap at the pointer,+ * not exceeding size "maxbytes" in bytes. This function does not allocate+ * memory dynamically.+ *+ * This function returns 0 if and only if no valid bitmap is found.+ * Otherwise, it returns how many bytes are occupied by the bitmap data.+ */+size_t ra_portable_deserialize_size(const char *buf, const size_t maxbytes);++/**+ * How many bytes are required to serialize this bitmap (meant to be+ * compatible+ * with Java and Go versions)+ */+size_t ra_portable_size_in_bytes(const roaring_array_t *ra);++/**+ * return true if it contains at least one run container.+ */+bool ra_has_run_container(const roaring_array_t *ra);++/**+ * Size of the header when serializing (meant to be compatible+ * with Java and Go versions)+ */+uint32_t ra_portable_header_size(const roaring_array_t *ra);++/**+ * Size of the header when serializing "num_containers" containers.+ * "hasrun" should be true if at least one of them is a run container.+ */+uint32_t ra_portable_header_size_from_counts(int32_t num_containers,+                                             bool hasrun);++/**+ * If the container at the index i is share, unshare it (creating a local+ * copy if needed).+ */+static inline void ra_unshare_container_at_index(roaring_array_t *ra,+                                                 uint16_t i) {+    assert(i < ra->size);+    ra->containers[i] =+        get_writable_copy_if_shared(ra->containers[i], &ra->typecodes[i]);+}++/**+ * remove at index i, sliding over all entries after i+ */+void ra_remove_at_index(roaring_array_t *ra, int32_t i);++/**+ * clears all containers, sets the size at 0 and shrinks the memory usage.+ */+void ra_reset(roaring_array_t *ra);++/**+ * remove at index i, sliding over all entries after i. Free removed container.+ */+void ra_remove_at_index_and_free(roaring_array_t *ra, int32_t i);++/**+ * remove a chunk of indices, sliding over entries after it+ */+// void ra_remove_index_range(roaring_array_t *ra, int32_t begin, int32_t end);++// used in inplace andNot only, to slide left the containers from+// the mutated RoaringBitmap that are after the largest container of+// the argument RoaringBitmap.  It is followed by a call to resize.+//+void ra_copy_range(roaring_array_t *ra, uint32_t begin, uint32_t end,+                   uint32_t new_begin);++/**+ * Shifts rightmost $count containers to the left (distance < 0) or+ * to the right (distance > 0).+ * Allocates memory if necessary.+ * This function doesn't free or create new containers.+ * Caller is responsible for that.+ */+void ra_shift_tail(roaring_array_t *ra, int32_t count, int32_t distance);++#ifdef __cplusplus+}  // namespace internal+}+}  // extern "C" { namespace roaring {+#endif++#endif
+ cbits/CRoaring/include/roaring/roaring_types.h view
@@ -0,0 +1,161 @@+/*+    Shared type definitions used across the CRoaring public API and internal+    components.++    This file centralizes common typedefs and small structs that are referenced+    by multiple headers, including iterator callback signatures, roaring array+    metadata, statistics structures, and compatibility types used to bridge the+    C and C++ builds.+*/++#ifndef ROARING_TYPES_H+#define ROARING_TYPES_H++#include <stdbool.h>+#include <stdint.h>++#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace api {+#endif++/**+ * When building .c files as C++, there's added compile-time checking if the+ * container types are derived from a `container_t` base class.  So long as+ * such a base class is empty, the struct will behave compatibly with C structs+ * despite the derivation.  This is due to the Empty Base Class Optimization:+ *+ * https://en.cppreference.com/w/cpp/language/ebo+ *+ * But since C isn't namespaced, taking `container_t` globally might collide+ * with other projects.  So roaring.h uses ROARING_CONTAINER_T, while internal+ * code #undefs that after declaring `typedef ROARING_CONTAINER_T container_t;`+ */+#if defined(__cplusplus)+extern "C++" {+struct container_s {};+}+#define ROARING_CONTAINER_T ::roaring::api::container_s+#else+#define ROARING_CONTAINER_T void  // no compile-time checking+#endif++#define ROARING_FLAG_COW UINT8_C(0x1)+#define ROARING_FLAG_FROZEN UINT8_C(0x2)+// 64-bit only: ART node arrays alias a frozen buffer (do not art_free).+#define ROARING_FLAG_FROZEN_ART UINT8_C(0x4)++/**+ * Roaring arrays are array-based key-value pairs having containers as values+ * and 16-bit integer keys. A roaring bitmap  might be implemented as such.+ */++// parallel arrays.  Element sizes quite different.+// Alternative is array+// of structs.  Which would have better+// cache performance through binary searches?++typedef struct roaring_array_s {+    int32_t size;+    int32_t allocation_size;+    ROARING_CONTAINER_T **containers;  // Use container_t in non-API files!+    uint16_t *keys;+    uint8_t *typecodes;+    uint8_t flags;+} roaring_array_t;++typedef bool (*roaring_iterator)(uint32_t value, void *param);+typedef bool (*roaring_iterator64)(uint64_t value, void *param);++/**+ *  (For advanced users.)+ * The roaring_statistics_t can be used to collect detailed statistics about+ * the composition of a roaring bitmap.+ */+typedef struct roaring_statistics_s {+    uint32_t n_containers; /* number of containers */++    uint32_t n_array_containers;  /* number of array containers */+    uint32_t n_run_containers;    /* number of run containers */+    uint32_t n_bitset_containers; /* number of bitmap containers */++    uint32_t+        n_values_array_containers;    /* number of values in array containers */+    uint32_t n_values_run_containers; /* number of values in run containers */+    uint32_t+        n_values_bitset_containers; /* number of values in  bitmap containers */++    uint32_t n_bytes_array_containers;  /* number of allocated bytes in array+                                           containers */+    uint32_t n_bytes_run_containers;    /* number of allocated bytes in run+                                           containers */+    uint32_t n_bytes_bitset_containers; /* number of allocated bytes in  bitmap+                                           containers */++    uint32_t+        max_value; /* the maximal value, undefined if cardinality is zero */+    uint32_t+        min_value; /* the minimal value, undefined if cardinality is zero */++    CROARING_DEPRECATED+    uint64_t sum_value; /* deprecated always zero */++    uint64_t cardinality; /* total number of values stored in the bitmap */++    // and n_values_arrays, n_values_rle, n_values_bitmap+} roaring_statistics_t;++/**+ *  (For advanced users.)+ * The roaring64_statistics_t can be used to collect detailed statistics about+ * the composition of a roaring64 bitmap.+ */+typedef struct roaring64_statistics_s {+    uint64_t n_containers; /* number of containers */++    uint64_t n_array_containers;  /* number of array containers */+    uint64_t n_run_containers;    /* number of run containers */+    uint64_t n_bitset_containers; /* number of bitmap containers */++    uint64_t+        n_values_array_containers;    /* number of values in array containers */+    uint64_t n_values_run_containers; /* number of values in run containers */+    uint64_t+        n_values_bitset_containers; /* number of values in  bitmap containers */++    uint64_t n_bytes_array_containers;  /* number of allocated bytes in array+                                           containers */+    uint64_t n_bytes_run_containers;    /* number of allocated bytes in run+                                           containers */+    uint64_t n_bytes_bitset_containers; /* number of allocated bytes in  bitmap+                                           containers */++    uint64_t+        max_value; /* the maximal value, undefined if cardinality is zero */+    uint64_t+        min_value; /* the minimal value, undefined if cardinality is zero */++    uint64_t cardinality; /* total number of values stored in the bitmap */++    // and n_values_arrays, n_values_rle, n_values_bitmap+} roaring64_statistics_t;++/**+ * Roaring-internal type used to iterate within a roaring container.+ */+typedef struct roaring_container_iterator_s {+    // For bitset and array containers this is the index of the bit / entry.+    // For run containers this points at the run.+    int32_t index;+} roaring_container_iterator_t;++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace api {+#endif++#endif /* ROARING_TYPES_H */
+ cbits/CRoaring/include/roaring/roaring_version.h view
@@ -0,0 +1,12 @@+// clang-format off+// /include/roaring/roaring_version.h automatically generated by release.py, do not change by hand+#ifndef ROARING_INCLUDE_ROARING_VERSION+#define ROARING_INCLUDE_ROARING_VERSION+#define ROARING_VERSION "5.5.5"+enum {+    ROARING_VERSION_MAJOR = 5,+    ROARING_VERSION_MINOR = 5,+    ROARING_VERSION_REVISION = 5+};+#endif // ROARING_INCLUDE_ROARING_VERSION+// clang-format on
+ cbits/CRoaring/include/roaring/utilasm.h view
@@ -0,0 +1,89 @@+/*+ * utilasm.h+ *+ * This file provides optional inline-assembly helpers for low-level bit+ * manipulation on supported x86/x64 targets. These macros are used to map a+ * few performance-sensitive operations, such as shifting, testing, setting,+ * and clearing bits, to specific machine instructions when inline assembly is+ * enabled.+ *+ * The intent is to centralize these architecture-specific primitives behind a+ * small interface so the rest of the codebase can use them conditionally while+ * keeping the generic implementation paths separate.+ */++#ifndef INCLUDE_UTILASM_H_+#define INCLUDE_UTILASM_H_++#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+#endif++#if defined(CROARING_INLINE_ASM)+#define CROARING_ASMBITMANIPOPTIMIZATION  // optimization flag++#define CROARING_ASM_SHIFT_RIGHT(srcReg, bitsReg, destReg) \+    __asm volatile("shrx %1, %2, %0"                       \+                   : "=r"(destReg)                         \+                   :             /* write */               \+                   "r"(bitsReg), /* read only */           \+                   "r"(srcReg)   /* read only */           \+    )++#define CROARING_ASM_INPLACESHIFT_RIGHT(srcReg, bitsReg) \+    __asm volatile("shrx %1, %0, %0"                     \+                   : "+r"(srcReg)                        \+                   :            /* read/write */         \+                   "r"(bitsReg) /* read only */          \+    )++#define CROARING_ASM_SHIFT_LEFT(srcReg, bitsReg, destReg) \+    __asm volatile("shlx %1, %2, %0"                      \+                   : "=r"(destReg)                        \+                   :             /* write */              \+                   "r"(bitsReg), /* read only */          \+                   "r"(srcReg)   /* read only */          \+    )+// set bit at position testBit within testByte to 1 and+// copy cmovDst to cmovSrc if that bit was previously clear+#define CROARING_ASM_SET_BIT_INC_WAS_CLEAR(testByte, testBit, count) \+    __asm volatile(                                                  \+        "bts %2, %0\n"                                               \+        "sbb $-1, %1\n"                                              \+        : "+r"(testByte), /* read/write */                           \+          "+r"(count)                                                \+        :            /* read/write */                                \+        "r"(testBit) /* read only */                                 \+    )++#define CROARING_ASM_CLEAR_BIT_DEC_WAS_SET(testByte, testBit, count) \+    __asm volatile(                                                  \+        "btr %2, %0\n"                                               \+        "sbb $0, %1\n"                                               \+        : "+r"(testByte), /* read/write */                           \+          "+r"(count)                                                \+        :            /* read/write */                                \+        "r"(testBit) /* read only */                                 \+    )++#define CROARING_ASM_BT64(testByte, testBit, count) \+    __asm volatile(                                 \+        "bt %2,%1\n"                                \+        "sbb %0,%0" /*could use setb */             \+        : "=r"(count)                               \+        :              /* write */                  \+        "r"(testByte), /* read only */              \+        "r"(testBit)   /* read only */              \+    )++#endif++#ifdef __cplusplus+}+}  // extern "C" { namespace roaring {+#endif++#endif /* INCLUDE_UTILASM_H_ */
+ cbits/CRoaring/src/array_util.c view
@@ -0,0 +1,2654 @@+#include <assert.h>+#include <stdbool.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/array_util.h>+#include <roaring/portability.h>+#include <roaring/utilasm.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif++#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+using namespace ::roaring::internal;+extern "C" {+namespace roaring {+namespace internal {+#endif++extern inline int32_t binarySearch(const uint16_t *array, int32_t lenarray,+                                   uint16_t ikey);++#if CROARING_IS_X64+// used by intersect_vector16+ALIGNED(0x1000)+static const uint8_t shuffle_mask16[] = {+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    6,    7,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    8,    9,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    8,    9,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    6,    7,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,    8,    9,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,+    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    6,    7,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,    8,    9,    0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    6,    7,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    6,    7,    8,    9,    0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    6,    7,+    8,    9,    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 10,   11,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    10,   11,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    6,    7,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    6,    7,    10,   11,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    6,    7,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    6,    7,    10,   11,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    6,    7,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,    10,   11,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    8,    9,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,    10,   11,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    8,    9,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    8,    9,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    8,    9,    10,   11,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    6,    7,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    8,    9,+    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    8,    9,    10,   11,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    8,    9,    10,   11,+    0xFF, 0xFF, 0xFF, 0xFF, 12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    6,    7,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    6,    7,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    6,    7,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    6,    7,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    6,    7,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 8,    9,    12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    8,    9,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    8,    9,    12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    8,    9,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    8,    9,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,+    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    6,    7,    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    6,    7,    8,    9,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,+    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    6,    7,    8,    9,    12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    6,    7,    8,    9,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    6,    7,    8,    9,    12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    10,   11,   12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    10,   11,   12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    10,   11,   12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    6,    7,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    10,   11,   12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 8,    9,    10,   11,   12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,+    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    8,    9,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,+    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    8,    9,    10,   11,   12,   13,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    8,    9,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    6,    7,    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,    8,    9,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,+    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    6,    7,    8,    9,    10,   11,   12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,    8,    9,    10,   11,+    12,   13,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    6,    7,    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    6,    7,    8,    9,    10,   11,   12,   13,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    6,    7,+    8,    9,    10,   11,   12,   13,   0xFF, 0xFF, 14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    6,    7,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    6,    7,    14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    6,    7,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    6,    7,    14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    6,    7,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,    14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    8,    9,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,    14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    8,    9,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    8,    9,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    8,    9,    14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    6,    7,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    8,    9,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    8,    9,    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    8,    9,    14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    10,   11,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    10,   11,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    6,    7,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,+    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    6,    7,    10,   11,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    6,    7,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    6,    7,    10,   11,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    6,    7,+    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 8,    9,    10,   11,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    8,    9,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    8,    9,    10,   11,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    8,    9,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    8,    9,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,+    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    6,    7,    8,    9,    10,   11,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    6,    7,    8,    9,+    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,+    8,    9,    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    6,    7,    8,    9,    10,   11,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    6,    7,    8,    9,+    10,   11,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    6,    7,    8,    9,    10,   11,   14,   15,   0xFF, 0xFF,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    12,   13,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    12,   13,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    6,    7,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 8,    9,    12,   13,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    8,    9,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    8,    9,    12,   13,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    8,    9,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    6,    7,    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,    8,    9,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,+    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    6,    7,    8,    9,    12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,    8,    9,    12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    6,    7,    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    6,    7,    8,    9,    12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    6,    7,+    8,    9,    12,   13,   14,   15,   0xFF, 0xFF, 10,   11,   12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    4,    5,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    10,   11,   12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 6,    7,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    6,    7,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    6,    7,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    6,    7,    10,   11,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    6,    7,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    4,    5,    6,    7,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    4,    5,    6,    7,    10,   11,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    4,    5,    6,    7,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF,+    8,    9,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    8,    9,    10,   11,   12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    8,    9,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    2,    3,    8,    9,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 4,    5,    8,    9,    10,   11,   12,   13,+    14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,+    8,    9,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF,+    2,    3,    4,    5,    8,    9,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,    4,    5,    8,    9,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 6,    7,    8,    9,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0,    1,    6,    7,    8,    9,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 2,    3,    6,    7,    8,    9,    10,   11,+    12,   13,   14,   15,   0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    2,    3,+    6,    7,    8,    9,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF,+    4,    5,    6,    7,    8,    9,    10,   11,   12,   13,   14,   15,+    0xFF, 0xFF, 0xFF, 0xFF, 0,    1,    4,    5,    6,    7,    8,    9,+    10,   11,   12,   13,   14,   15,   0xFF, 0xFF, 2,    3,    4,    5,+    6,    7,    8,    9,    10,   11,   12,   13,   14,   15,   0xFF, 0xFF,+    0,    1,    2,    3,    4,    5,    6,    7,    8,    9,    10,   11,+    12,   13,   14,   15};++/**+ * From Schlegel et al., Fast Sorted-Set Intersection using SIMD Instructions+ * Optimized by D. Lemire on May 3rd 2013+ */+CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+int32_t intersect_vector16(const uint16_t *A, size_t s_a, const uint16_t *B,+                           size_t s_b, uint16_t *C) {+    size_t count = 0;+    size_t i_a = 0, i_b = 0;+    const int vectorlength = sizeof(__m128i) / sizeof(uint16_t);+    const size_t st_a = (s_a / vectorlength) * vectorlength;+    const size_t st_b = (s_b / vectorlength) * vectorlength;+    __m128i v_a, v_b;+    if ((i_a < st_a) && (i_b < st_b)) {+        v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+        v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+        while ((A[i_a] == 0) || (B[i_b] == 0)) {+            const __m128i res_v = _mm_cmpestrm(+                v_b, vectorlength, v_a, vectorlength,+                _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+            const int r = _mm_extract_epi32(res_v, 0);+            __m128i sm16 = _mm_loadu_si128((const __m128i *)shuffle_mask16 + r);+            __m128i p = _mm_shuffle_epi8(v_a, sm16);+            _mm_storeu_si128((__m128i *)&C[count], p);  // can overflow+            count += _mm_popcnt_u32(r);+            const uint16_t a_max = A[i_a + vectorlength - 1];+            const uint16_t b_max = B[i_b + vectorlength - 1];+            if (a_max <= b_max) {+                i_a += vectorlength;+                if (i_a == st_a) break;+                v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+            }+            if (b_max <= a_max) {+                i_b += vectorlength;+                if (i_b == st_b) break;+                v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+            }+        }+        if ((i_a < st_a) && (i_b < st_b))+            while (true) {+                const __m128i res_v = _mm_cmpistrm(+                    v_b, v_a,+                    _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+                const int r = _mm_extract_epi32(res_v, 0);+                __m128i sm16 =+                    _mm_loadu_si128((const __m128i *)shuffle_mask16 + r);+                __m128i p = _mm_shuffle_epi8(v_a, sm16);+                _mm_storeu_si128((__m128i *)&C[count], p);  // can overflow+                count += _mm_popcnt_u32(r);+                const uint16_t a_max = A[i_a + vectorlength - 1];+                const uint16_t b_max = B[i_b + vectorlength - 1];+                if (a_max <= b_max) {+                    i_a += vectorlength;+                    if (i_a == st_a) break;+                    v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+                }+                if (b_max <= a_max) {+                    i_b += vectorlength;+                    if (i_b == st_b) break;+                    v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+                }+            }+    }+    // intersect the tail using scalar intersection+    while (i_a < s_a && i_b < s_b) {+        uint16_t a = A[i_a];+        uint16_t b = B[i_b];+        if (a < b) {+            i_a++;+        } else if (b < a) {+            i_b++;+        } else {+            C[count] = a;  //==b;+            count++;+            i_a++;+            i_b++;+        }+    }+    return (int32_t)count;+}++CROARING_ALLOW_UNALIGNED+int array_container_to_uint32_array_vector16(void *vout, const uint16_t *array,+                                             size_t cardinality,+                                             uint32_t base) {+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;+    size_t i = 0;+    for (; i + sizeof(__m128i) / sizeof(uint16_t) <= cardinality;+         i += sizeof(__m128i) / sizeof(uint16_t)) {+        __m128i vinput = _mm_loadu_si128((const __m128i *)(array + i));+        __m256i voutput = _mm256_add_epi32(_mm256_cvtepu16_epi32(vinput),+                                           _mm256_set1_epi32(base));+        _mm256_storeu_si256((__m256i *)(out + outpos), voutput);+        outpos += sizeof(__m256i) / sizeof(uint32_t);+    }+    for (; i < cardinality; ++i) {+        const uint32_t val = base + array[i];+        memcpy(out + outpos, &val,+               sizeof(uint32_t));  // should be compiled as a MOV on x64+        outpos++;+    }+    return outpos;+}++CROARING_ALLOW_UNALIGNED+int32_t intersect_vector16_inplace(uint16_t *A, size_t s_a, const uint16_t *B,+                                   size_t s_b) {+    size_t count = 0;+    size_t i_a = 0, i_b = 0;+    const int vectorlength = sizeof(__m128i) / sizeof(uint16_t);+    const size_t st_a = (s_a / vectorlength) * vectorlength;+    const size_t st_b = (s_b / vectorlength) * vectorlength;+    __m128i v_a, v_b;+    if ((i_a < st_a) && (i_b < st_b)) {+        v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+        v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+        __m128i tmp[2] = {_mm_setzero_si128()};+        size_t tmp_count = 0;+        while ((A[i_a] == 0) || (B[i_b] == 0)) {+            const __m128i res_v = _mm_cmpestrm(+                v_b, vectorlength, v_a, vectorlength,+                _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+            const int r = _mm_extract_epi32(res_v, 0);+            __m128i sm16 = _mm_loadu_si128((const __m128i *)shuffle_mask16 + r);+            __m128i p = _mm_shuffle_epi8(v_a, sm16);+            _mm_storeu_si128((__m128i *)&((uint16_t *)tmp)[tmp_count], p);+            tmp_count += _mm_popcnt_u32(r);+            const uint16_t a_max = A[i_a + vectorlength - 1];+            const uint16_t b_max = B[i_b + vectorlength - 1];+            if (a_max <= b_max) {+                _mm_storeu_si128((__m128i *)&A[count], tmp[0]);+                _mm_storeu_si128(tmp, _mm_setzero_si128());+                count += tmp_count;+                tmp_count = 0;+                i_a += vectorlength;+                if (i_a == st_a) break;+                v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+            }+            if (b_max <= a_max) {+                i_b += vectorlength;+                if (i_b == st_b) break;+                v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+            }+        }+        if ((i_a < st_a) && (i_b < st_b)) {+            while (true) {+                const __m128i res_v = _mm_cmpistrm(+                    v_b, v_a,+                    _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+                const int r = _mm_extract_epi32(res_v, 0);+                __m128i sm16 =+                    _mm_loadu_si128((const __m128i *)shuffle_mask16 + r);+                __m128i p = _mm_shuffle_epi8(v_a, sm16);+                _mm_storeu_si128((__m128i *)&((uint16_t *)tmp)[tmp_count], p);+                tmp_count += _mm_popcnt_u32(r);+                const uint16_t a_max = A[i_a + vectorlength - 1];+                const uint16_t b_max = B[i_b + vectorlength - 1];+                if (a_max <= b_max) {+                    _mm_storeu_si128((__m128i *)&A[count], tmp[0]);+                    _mm_storeu_si128(tmp, _mm_setzero_si128());+                    count += tmp_count;+                    tmp_count = 0;+                    i_a += vectorlength;+                    if (i_a == st_a) break;+                    v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+                }+                if (b_max <= a_max) {+                    i_b += vectorlength;+                    if (i_b == st_b) break;+                    v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+                }+            }+        }+        // tmp_count <= 8, so this does not affect efficiency so much+        for (size_t i = 0; i < tmp_count; i++) {+            A[count] = ((uint16_t *)tmp)[i];+            count++;+        }+        i_a += tmp_count;  // We can at least jump pass $tmp_count elements in A+    }+    // intersect the tail using scalar intersection+    while (i_a < s_a && i_b < s_b) {+        uint16_t a = A[i_a];+        uint16_t b = B[i_b];+        if (a < b) {+            i_a++;+        } else if (b < a) {+            i_b++;+        } else {+            A[count] = a;  //==b;+            count++;+            i_a++;+            i_b++;+        }+    }+    return (int32_t)count;+}+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+int32_t intersect_vector16_cardinality(const uint16_t *A, size_t s_a,+                                       const uint16_t *B, size_t s_b) {+    size_t count = 0;+    size_t i_a = 0, i_b = 0;+    const int vectorlength = sizeof(__m128i) / sizeof(uint16_t);+    const size_t st_a = (s_a / vectorlength) * vectorlength;+    const size_t st_b = (s_b / vectorlength) * vectorlength;+    __m128i v_a, v_b;+    if ((i_a < st_a) && (i_b < st_b)) {+        v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+        v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+        while ((A[i_a] == 0) || (B[i_b] == 0)) {+            const __m128i res_v = _mm_cmpestrm(+                v_b, vectorlength, v_a, vectorlength,+                _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+            const int r = _mm_extract_epi32(res_v, 0);+            count += _mm_popcnt_u32(r);+            const uint16_t a_max = A[i_a + vectorlength - 1];+            const uint16_t b_max = B[i_b + vectorlength - 1];+            if (a_max <= b_max) {+                i_a += vectorlength;+                if (i_a == st_a) break;+                v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+            }+            if (b_max <= a_max) {+                i_b += vectorlength;+                if (i_b == st_b) break;+                v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+            }+        }+        if ((i_a < st_a) && (i_b < st_b))+            while (true) {+                const __m128i res_v = _mm_cmpistrm(+                    v_b, v_a,+                    _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+                const int r = _mm_extract_epi32(res_v, 0);+                count += _mm_popcnt_u32(r);+                const uint16_t a_max = A[i_a + vectorlength - 1];+                const uint16_t b_max = B[i_b + vectorlength - 1];+                if (a_max <= b_max) {+                    i_a += vectorlength;+                    if (i_a == st_a) break;+                    v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+                }+                if (b_max <= a_max) {+                    i_b += vectorlength;+                    if (i_b == st_b) break;+                    v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+                }+            }+    }+    // intersect the tail using scalar intersection+    while (i_a < s_a && i_b < s_b) {+        uint16_t a = A[i_a];+        uint16_t b = B[i_b];+        if (a < b) {+            i_a++;+        } else if (b < a) {+            i_b++;+        } else {+            count++;+            i_a++;+            i_b++;+        }+    }+    return (int32_t)count;+}+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+/////////+// Warning:+// This function may not be safe if A == C or B == C.+/////////+CROARING_ALLOW_UNALIGNED+int32_t difference_vector16(const uint16_t *A, size_t s_a, const uint16_t *B,+                            size_t s_b, uint16_t *C) {+    // we handle the degenerate case+    if (s_a == 0) return 0;+    if (s_b == 0) {+        if (A != C) memcpy(C, A, sizeof(uint16_t) * s_a);+        return (int32_t)s_a;+    }+    // handle the leading zeroes, it is messy but it allows us to use the fast+    // _mm_cmpistrm instrinsic safely+    int32_t count = 0;+    if ((A[0] == 0) || (B[0] == 0)) {+        if ((A[0] == 0) && (B[0] == 0)) {+            A++;+            s_a--;+            B++;+            s_b--;+        } else if (A[0] == 0) {+            C[count++] = 0;+            A++;+            s_a--;+        } else {+            B++;+            s_b--;+        }+    }+    // at this point, we have two non-empty arrays, made of non-zero+    // increasing values.+    size_t i_a = 0, i_b = 0;+    const size_t vectorlength = sizeof(__m128i) / sizeof(uint16_t);+    const size_t st_a = (s_a / vectorlength) * vectorlength;+    const size_t st_b = (s_b / vectorlength) * vectorlength;+    if ((i_a < st_a) && (i_b < st_b)) {  // this is the vectorized code path+        __m128i v_a, v_b;                //, v_bmax;+        // we load a vector from A and a vector from B+        v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+        v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+        // we have a runningmask which indicates which values from A have been+        // spotted in B, these don't get written out.+        __m128i runningmask_a_found_in_b = _mm_setzero_si128();+        /****+         * start of the main vectorized loop+         *****/+        while (true) {+            // afoundinb will contain a mask indicate for each entry in A+            // whether it is seen+            // in B+            const __m128i a_found_in_b = _mm_cmpistrm(+                v_b, v_a,+                _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+            runningmask_a_found_in_b =+                _mm_or_si128(runningmask_a_found_in_b, a_found_in_b);+            // we always compare the last values of A and B+            const uint16_t a_max = A[i_a + vectorlength - 1];+            const uint16_t b_max = B[i_b + vectorlength - 1];+            if (a_max <= b_max) {+                // Ok. In this code path, we are ready to write our v_a+                // because there is no need to read more from B, they will+                // all be large values.+                const int bitmask_belongs_to_difference =+                    _mm_extract_epi32(runningmask_a_found_in_b, 0) ^ 0xFF;+                /*** next few lines are probably expensive *****/+                __m128i sm16 = _mm_loadu_si128((const __m128i *)shuffle_mask16 ++                                               bitmask_belongs_to_difference);+                __m128i p = _mm_shuffle_epi8(v_a, sm16);+                _mm_storeu_si128((__m128i *)&C[count], p);  // can overflow+                count += _mm_popcnt_u32(bitmask_belongs_to_difference);+                // we advance a+                i_a += vectorlength;+                if (i_a == st_a)  // no more+                    break;+                runningmask_a_found_in_b = _mm_setzero_si128();+                v_a = _mm_lddqu_si128((__m128i *)&A[i_a]);+            }+            if (b_max <= a_max) {+                // in this code path, the current v_b has become useless+                i_b += vectorlength;+                if (i_b == st_b) break;+                v_b = _mm_lddqu_si128((__m128i *)&B[i_b]);+            }+        }+        // at this point, either we have i_a == st_a, which is the end of the+        // vectorized processing,+        // or we have i_b == st_b,  and we are not done processing the vector...+        // so we need to finish it off.+        if (i_a < st_a) {        // we have unfinished business...+            uint16_t buffer[8];  // buffer to do a masked load+            memset(buffer, 0, 8 * sizeof(uint16_t));+            memcpy(buffer, B + i_b, (s_b - i_b) * sizeof(uint16_t));+            v_b = _mm_lddqu_si128((__m128i *)buffer);+            const __m128i a_found_in_b = _mm_cmpistrm(+                v_b, v_a,+                _SIDD_UWORD_OPS | _SIDD_CMP_EQUAL_ANY | _SIDD_BIT_MASK);+            runningmask_a_found_in_b =+                _mm_or_si128(runningmask_a_found_in_b, a_found_in_b);+            const int bitmask_belongs_to_difference =+                _mm_extract_epi32(runningmask_a_found_in_b, 0) ^ 0xFF;+            __m128i sm16 = _mm_loadu_si128((const __m128i *)shuffle_mask16 ++                                           bitmask_belongs_to_difference);+            __m128i p = _mm_shuffle_epi8(v_a, sm16);+            _mm_storeu_si128((__m128i *)&C[count], p);  // can overflow+            count += _mm_popcnt_u32(bitmask_belongs_to_difference);+            i_a += vectorlength;+        }+        // at this point we should have i_a == st_a and i_b == st_b+    }+    // do the tail using scalar code+    while (i_a < s_a && i_b < s_b) {+        uint16_t a = A[i_a];+        uint16_t b = B[i_b];+        if (b < a) {+            i_b++;+        } else if (a < b) {+            C[count] = a;+            count++;+            i_a++;+        } else {  //==+            i_a++;+            i_b++;+        }+    }+    if (i_a < s_a) {+        if (C == A) {+            assert((size_t)count <= i_a);+            if ((size_t)count < i_a) {+                memmove(C + count, A + i_a, sizeof(uint16_t) * (s_a - i_a));+            }+        } else {+            for (size_t i = 0; i < (s_a - i_a); i++) {+                C[count + i] = A[i + i_a];+            }+        }+        count += (int32_t)(s_a - i_a);+    }+    return count;+}+CROARING_UNTARGET_AVX2+#endif  // CROARING_IS_X64++/**+ * Branchless binary search going after 4 values at once.+ * Assumes that array is sorted.+ * You have that array[*index1] >= target1, array[*index12] >= target2, ...+ * except when *index1 = n, in which case you know that all values in array are+ * smaller than target1, and so forth.+ * It has logarithmic complexity.+ */+CROARING_ALLOW_UNALIGNED+static void binarySearch4(const uint16_t *array, int32_t n, uint16_t target1,+                          uint16_t target2, uint16_t target3, uint16_t target4,+                          int32_t *index1, int32_t *index2, int32_t *index3,+                          int32_t *index4) {+    const uint16_t *base1 = array;+    const uint16_t *base2 = array;+    const uint16_t *base3 = array;+    const uint16_t *base4 = array;+    if (n == 0) return;+    while (n > 1) {+        int32_t half = n >> 1;+        base1 = (base1[half] < target1) ? &base1[half] : base1;+        base2 = (base2[half] < target2) ? &base2[half] : base2;+        base3 = (base3[half] < target3) ? &base3[half] : base3;+        base4 = (base4[half] < target4) ? &base4[half] : base4;+        n -= half;+    }+    *index1 = (int32_t)((*base1 < target1) + base1 - array);+    *index2 = (int32_t)((*base2 < target2) + base2 - array);+    *index3 = (int32_t)((*base3 < target3) + base3 - array);+    *index4 = (int32_t)((*base4 < target4) + base4 - array);+}++/**+ * Branchless binary search going after 2 values at once.+ * Assumes that array is sorted.+ * You have that array[*index1] >= target1, array[*index12] >= target2.+ * except when *index1 = n, in which case you know that all values in array are+ * smaller than target1, and so forth.+ * It has logarithmic complexity.+ */+CROARING_ALLOW_UNALIGNED+static void binarySearch2(const uint16_t *array, int32_t n, uint16_t target1,+                          uint16_t target2, int32_t *index1, int32_t *index2) {+    const uint16_t *base1 = array;+    const uint16_t *base2 = array;+    if (n == 0) return;+    while (n > 1) {+        int32_t half = n >> 1;+        base1 = (base1[half] < target1) ? &base1[half] : base1;+        base2 = (base2[half] < target2) ? &base2[half] : base2;+        n -= half;+    }+    *index1 = (int32_t)((*base1 < target1) + base1 - array);+    *index2 = (int32_t)((*base2 < target2) + base2 - array);+}++/* Computes the intersection between one small and one large set of uint16_t.+ * Stores the result into buffer and return the number of elements.+ * Processes the small set in blocks of 4 values calling binarySearch4+ * and binarySearch2. This approach can be slightly superior to a conventional+ * galloping search in some instances.+ */+int32_t intersect_skewed_uint16(const uint16_t *small, size_t size_s,+                                const uint16_t *large, size_t size_l,+                                uint16_t *buffer) {+    size_t pos = 0, idx_l = 0, idx_s = 0;++    if (0 == size_s) {+        return 0;+    }+    int32_t index1 = 0, index2 = 0, index3 = 0, index4 = 0;+    while ((idx_s + 4 <= size_s) && (idx_l < size_l)) {+        uint16_t target1 = small[idx_s];+        uint16_t target2 = small[idx_s + 1];+        uint16_t target3 = small[idx_s + 2];+        uint16_t target4 = small[idx_s + 3];+        binarySearch4(large + idx_l, (int32_t)(size_l - idx_l), target1,+                      target2, target3, target4, &index1, &index2, &index3,+                      &index4);+        if ((index1 + idx_l < size_l) && (large[idx_l + index1] == target1)) {+            buffer[pos++] = target1;+        }+        if ((index2 + idx_l < size_l) && (large[idx_l + index2] == target2)) {+            buffer[pos++] = target2;+        }+        if ((index3 + idx_l < size_l) && (large[idx_l + index3] == target3)) {+            buffer[pos++] = target3;+        }+        if ((index4 + idx_l < size_l) && (large[idx_l + index4] == target4)) {+            buffer[pos++] = target4;+        }+        idx_s += 4;+        idx_l += index4;+    }+    if ((idx_s + 2 <= size_s) && (idx_l < size_l)) {+        uint16_t target1 = small[idx_s];+        uint16_t target2 = small[idx_s + 1];+        binarySearch2(large + idx_l, (int32_t)(size_l - idx_l), target1,+                      target2, &index1, &index2);+        if ((index1 + idx_l < size_l) && (large[idx_l + index1] == target1)) {+            buffer[pos++] = target1;+        }+        if ((index2 + idx_l < size_l) && (large[idx_l + index2] == target2)) {+            buffer[pos++] = target2;+        }+        idx_s += 2;+        idx_l += index2;+    }+    if ((idx_s < size_s) && (idx_l < size_l)) {+        uint16_t val_s = small[idx_s];+        int32_t index =+            binarySearch(large + idx_l, (int32_t)(size_l - idx_l), val_s);+        if (index >= 0) buffer[pos++] = val_s;+    }+    return (int32_t)pos;+}++// TODO: this could be accelerated, possibly, by using binarySearch4 as above.+int32_t intersect_skewed_uint16_cardinality(const uint16_t *small,+                                            size_t size_s,+                                            const uint16_t *large,+                                            size_t size_l) {+    size_t pos = 0, idx_l = 0, idx_s = 0;++    if (0 == size_s) {+        return 0;+    }++    uint16_t val_l = large[idx_l], val_s = small[idx_s];++    while (true) {+        if (val_l < val_s) {+            idx_l = advanceUntil(large, (int32_t)idx_l, (int32_t)size_l, val_s);+            if (idx_l == size_l) break;+            val_l = large[idx_l];+        } else if (val_s < val_l) {+            idx_s++;+            if (idx_s == size_s) break;+            val_s = small[idx_s];+        } else {+            pos++;+            idx_s++;+            if (idx_s == size_s) break;+            val_s = small[idx_s];+            idx_l = advanceUntil(large, (int32_t)idx_l, (int32_t)size_l, val_s);+            if (idx_l == size_l) break;+            val_l = large[idx_l];+        }+    }++    return (int32_t)pos;+}++bool intersect_skewed_uint16_nonempty(const uint16_t *small, size_t size_s,+                                      const uint16_t *large, size_t size_l) {+    size_t idx_l = 0, idx_s = 0;++    if (0 == size_s) {+        return false;+    }++    uint16_t val_l = large[idx_l], val_s = small[idx_s];++    while (true) {+        if (val_l < val_s) {+            idx_l = advanceUntil(large, (int32_t)idx_l, (int32_t)size_l, val_s);+            if (idx_l == size_l) break;+            val_l = large[idx_l];+        } else if (val_s < val_l) {+            idx_s++;+            if (idx_s == size_s) break;+            val_s = small[idx_s];+        } else {+            return true;+        }+    }++    return false;+}++/**+ * Generic intersection function.+ */+// The scalar merges below avoid a data-dependent branch per value: on sets+// that interleave unpredictably, a branchy merge mispredicts about once per+// value. While four values remain on each side, they first look for a block+// of four values to handle at once (all below the other side's current value,+// or equal to the other side's next four values), which is common and well+// predicted on clustered data. Otherwise they take four branchless steps.+CROARING_ALLOW_UNALIGNED+static inline bool croaring_equal4_uint16(const uint16_t *a,+                                          const uint16_t *b) {+    uint64_t x, y;+    memcpy(&x, a, sizeof(x));+    memcpy(&y, b, sizeof(y));+    return x == y;+}++// The branchless steps carry a dependency from one value to the next,+// so they cost several cycles per value. When one side is much shorter,+// or once fewer than four values remain on one side, the merges below+// instead locate each value of the short side in the long side by+// galloping and copy the values in between in bulk.++// Returns the first index i in [lo, n) with array[i] >= key, or n.+CROARING_ALLOW_UNALIGNED+static inline size_t croaring_gallop_uint16(const uint16_t *array, size_t lo,+                                            size_t n, uint16_t key) {+    if (lo >= n || array[lo] >= key) return lo;+    // Invariant: array[lo] < key, and the answer is in (lo, hi], where+    // hi == n stands for a value >= key past the end.+    size_t step = 1, hi = lo + 1;+    while (hi < n && array[hi] < key) {+        lo = hi;+        step <<= 1;+        hi = lo + step;+    }+    if (hi > n) hi = n;+    size_t len = hi - lo;+    while (len > 1) {+        const size_t half = len >> 1;+        if (array[lo + half] < key) lo += half;+        len -= half;+    }+    return lo + 1;+}++// Union of a short and a long sorted array. Every write goes to+// out[pos] with pos <= i + j, where i and j count the values read so+// far.+CROARING_ALLOW_UNALIGNED+static size_t union_skewed_uint16(const uint16_t *small, size_t size_s,+                                  const uint16_t *large, size_t size_l,+                                  uint16_t *out) {+    size_t pos = 0, j = 0;+    for (size_t i = 0; i < size_s; i++) {+        const uint16_t v = small[i];+        const size_t p = croaring_gallop_uint16(large, j, size_l, v);+        memmove(out + pos, large + j, (p - j) * sizeof(uint16_t));+        pos += p - j;+        j = p;+        out[pos++] = v;+        j += (j < size_l && large[j] == v);+    }+    memmove(out + pos, large + j, (size_l - j) * sizeof(uint16_t));+    return pos + size_l - j;+}++// Symmetric difference of a short and a long sorted array.+CROARING_ALLOW_UNALIGNED+static int32_t xor_skewed_uint16(const uint16_t *small, size_t size_s,+                                 const uint16_t *large, size_t size_l,+                                 uint16_t *out) {+    size_t pos = 0, j = 0;+    for (size_t i = 0; i < size_s; i++) {+        const uint16_t v = small[i];+        const size_t p = croaring_gallop_uint16(large, j, size_l, v);+        memcpy(out + pos, large + j, (p - j) * sizeof(uint16_t));+        pos += p - j;+        j = p;+        if (j < size_l && large[j] == v) {+            j++;+        } else {+            out[pos++] = v;+        }+    }+    memcpy(out + pos, large + j, (size_l - j) * sizeof(uint16_t));+    return (int32_t)(pos + size_l - j);+}++// a1 minus a short a2. Every write goes to a_out[k..] with k <= i, so+// a_out may be a1.+CROARING_ALLOW_UNALIGNED+static int difference_skewed_small_a2_uint16(const uint16_t *a1, int length1,+                                             const uint16_t *a2, int length2,+                                             uint16_t *a_out) {+    size_t i = 0, k = 0;+    for (int j = 0; j < length2 && i < (size_t)length1; j++) {+        const uint16_t v = a2[j];+        const size_t p = croaring_gallop_uint16(a1, i, (size_t)length1, v);+        memmove(a_out + k, a1 + i, (p - i) * sizeof(uint16_t));+        k += p - i;+        i = p;+        i += (i < (size_t)length1 && a1[i] == v);+    }+    memmove(a_out + k, a1 + i, ((size_t)length1 - i) * sizeof(uint16_t));+    return (int)(k + (size_t)length1 - i);+}++// A short a1 minus a2. Every write goes to a_out[k] with k <= i, so+// a_out may be a1.+CROARING_ALLOW_UNALIGNED+static int difference_skewed_small_a1_uint16(const uint16_t *a1, int length1,+                                             const uint16_t *a2, int length2,+                                             uint16_t *a_out) {+    size_t j = 0, k = 0;+    for (int i = 0; i < length1; i++) {+        const uint16_t v = a1[i];+        j = croaring_gallop_uint16(a2, j, (size_t)length2, v);+        a_out[k] = v;+        k += (j == (size_t)length2 || a2[j] != v);+    }+    return (int)k;+}++// Intersection of a short and a long sorted array, writing only on a+// match: every write goes to out[k] with k no larger than the index of+// the matched value on either side, so out may be either input.+CROARING_ALLOW_UNALIGNED+static size_t intersect_tail_uint16(const uint16_t *small, size_t size_s,+                                    const uint16_t *large, size_t size_l,+                                    uint16_t *out) {+    size_t j = 0, k = 0;+    for (size_t i = 0; i < size_s && j < size_l; i++) {+        const uint16_t v = small[i];+        j = croaring_gallop_uint16(large, j, size_l, v);+        if (j < size_l && large[j] == v) {+            out[k++] = v;+            j++;+        }+    }+    return k;+}++CROARING_ALLOW_UNALIGNED+static size_t intersect_tail_uint16_cardinality(const uint16_t *small,+                                                size_t size_s,+                                                const uint16_t *large,+                                                size_t size_l) {+    size_t j = 0, k = 0;+    for (size_t i = 0; i < size_s && j < size_l; i++) {+        const uint16_t v = small[i];+        j = croaring_gallop_uint16(large, j, size_l, v);+        if (j < size_l && large[j] == v) {+            k++;+            j++;+        }+    }+    return k;+}++CROARING_ALLOW_UNALIGNED+int32_t intersect_uint16(const uint16_t *A, size_t lenA, const uint16_t *B,+                         size_t lenB, uint16_t *out) {+    if (lenA == 0 || lenB == 0) return 0;+    // Every write goes to out[k..] with k <= i and k <= j, so "out" may be A.+    // If it is B, swap the inputs, since the values written come from A.+    if (out == B) {+        const uint16_t *tmp = A;+        A = B;+        B = tmp;+        size_t tmplen = lenA;+        lenA = lenB;+        lenB = tmplen;+    }+    size_t i = 0, j = 0, k = 0;+    while (i + 4 <= lenA && j + 4 <= lenB) {+        if (A[i + 3] < B[j]) {+            i += 4;+        } else if (B[j + 3] < A[i]) {+            j += 4;+        } else if (croaring_equal4_uint16(A + i, B + j)) {+            memmove(out + k, A + i, 4 * sizeof(uint16_t));+            k += 4;+            i += 4;+            j += 4;+        } else {+            for (int step = 0; step < 4; step++) {+                const uint16_t a = A[i], b = B[j];+                out[k] = a;+                k += (a == b);+                i += (a <= b);+                j += (a >= b);+            }+        }+    }+    // Fewer than four values remain on one side. If many remain on the+    // other, gallop rather than step through them one at a time.+    const size_t remA = lenA - i, remB = lenB - j;+    if (croaring_skewed_sizes(remA, remB)) {+        return (int32_t)(k + intersect_tail_uint16(A + i, remA, B + j, remB,+                                                   out + k));+    }+    if (croaring_skewed_sizes(remB, remA)) {+        return (int32_t)(k + intersect_tail_uint16(B + j, remB, A + i, remA,+                                                   out + k));+    }+    while (i < lenA && j < lenB) {+        const uint16_t a = A[i], b = B[j];+        out[k] = a;+        k += (a == b);+        i += (a <= b);+        j += (a >= b);+    }+    return (int32_t)k;+}++CROARING_ALLOW_UNALIGNED+int32_t intersect_uint16_cardinality(const uint16_t *A, const size_t lenA,+                                     const uint16_t *B, const size_t lenB) {+    size_t i = 0, j = 0;+    int32_t answer = 0;+    while (i + 4 <= lenA && j + 4 <= lenB) {+        if (A[i + 3] < B[j]) {+            i += 4;+        } else if (B[j + 3] < A[i]) {+            j += 4;+        } else if (croaring_equal4_uint16(A + i, B + j)) {+            answer += 4;+            i += 4;+            j += 4;+        } else {+            for (int step = 0; step < 4; step++) {+                const uint16_t a = A[i], b = B[j];+                answer += (a == b);+                i += (a <= b);+                j += (a >= b);+            }+        }+    }+    // Fewer than four values remain on one side. If many remain on the+    // other, gallop rather than step through them one at a time.+    const size_t remA = lenA - i, remB = lenB - j;+    if (croaring_skewed_sizes(remA, remB)) {+        return answer + (int32_t)intersect_tail_uint16_cardinality(A + i, remA,+                                                                   B + j, remB);+    }+    if (croaring_skewed_sizes(remB, remA)) {+        return answer + (int32_t)intersect_tail_uint16_cardinality(B + j, remB,+                                                                   A + i, remA);+    }+    while (i < lenA && j < lenB) {+        const uint16_t a = A[i], b = B[j];+        answer += (a == b);+        i += (a <= b);+        j += (a >= b);+    }+    return answer;+}++CROARING_ALLOW_UNALIGNED+bool intersect_uint16_nonempty(const uint16_t *A, const size_t lenA,+                               const uint16_t *B, const size_t lenB) {+    if (lenA == 0 || lenB == 0) return 0;+    const uint16_t *endA = A + lenA;+    const uint16_t *endB = B + lenB;++    while (1) {+        while (*A < *B) {+        SKIP_FIRST_COMPARE:+            if (++A == endA) return false;+        }+        while (*A > *B) {+            if (++B == endB) return false;+        }+        if (*A == *B) {+            return true;+        } else {+            goto SKIP_FIRST_COMPARE;+        }+    }+    return false;  // NOTREACHED+}++/**+ * Generic intersection function.+ */+size_t intersection_uint32(const uint32_t *A, const size_t lenA,+                           const uint32_t *B, const size_t lenB,+                           uint32_t *out) {+    const uint32_t *initout = out;+    if (lenA == 0 || lenB == 0) return 0;+    const uint32_t *endA = A + lenA;+    const uint32_t *endB = B + lenB;++    while (1) {+        while (*A < *B) {+        SKIP_FIRST_COMPARE:+            if (++A == endA) return (out - initout);+        }+        while (*A > *B) {+            if (++B == endB) return (out - initout);+        }+        if (*A == *B) {+            *out++ = *A;+            if (++A == endA || ++B == endB) return (out - initout);+        } else {+            goto SKIP_FIRST_COMPARE;+        }+    }+    // return (out - initout);  // NOTREACHED+}++size_t intersection_uint32_card(const uint32_t *A, const size_t lenA,+                                const uint32_t *B, const size_t lenB) {+    if (lenA == 0 || lenB == 0) return 0;+    size_t card = 0;+    const uint32_t *endA = A + lenA;+    const uint32_t *endB = B + lenB;++    while (1) {+        while (*A < *B) {+        SKIP_FIRST_COMPARE:+            if (++A == endA) return card;+        }+        while (*A > *B) {+            if (++B == endB) return card;+        }+        if (*A == *B) {+            card++;+            if (++A == endA || ++B == endB) return card;+        } else {+            goto SKIP_FIRST_COMPARE;+        }+    }+    // return card;  // NOTREACHED+}++// can one vectorize the computation of the union? (Update: Yes! See+// union_vector16).++CROARING_ALLOW_UNALIGNED+size_t union_uint16(const uint16_t *set_1, size_t size_1, const uint16_t *set_2,+                    size_t size_2, uint16_t *buffer) {+    // Every write goes before buffer + idx_1 + size_2, so buffer may start+    // size_2 values before set_1 (array_array_container_inplace_union).+    if (croaring_skewed_sizes(size_1, size_2)) {+        return union_skewed_uint16(set_1, size_1, set_2, size_2, buffer);+    }+    if (croaring_skewed_sizes(size_2, size_1)) {+        return union_skewed_uint16(set_2, size_2, set_1, size_1, buffer);+    }+    size_t pos = 0, idx_1 = 0, idx_2 = 0;+    while (idx_1 + 4 <= size_1 && idx_2 + 4 <= size_2) {+        if (set_1[idx_1 + 3] < set_2[idx_2]) {+            memmove(buffer + pos, set_1 + idx_1, 4 * sizeof(uint16_t));+            pos += 4;+            idx_1 += 4;+        } else if (set_2[idx_2 + 3] < set_1[idx_1]) {+            memmove(buffer + pos, set_2 + idx_2, 4 * sizeof(uint16_t));+            pos += 4;+            idx_2 += 4;+        } else if (croaring_equal4_uint16(set_1 + idx_1, set_2 + idx_2)) {+            memmove(buffer + pos, set_1 + idx_1, 4 * sizeof(uint16_t));+            pos += 4;+            idx_1 += 4;+            idx_2 += 4;+        } else {+            for (int step = 0; step < 4; step++) {+                const uint16_t val_1 = set_1[idx_1], val_2 = set_2[idx_2];+                buffer[pos++] = val_1 < val_2 ? val_1 : val_2;+                idx_1 += (val_1 <= val_2);+                idx_2 += (val_1 >= val_2);+            }+        }+    }+    // Fewer than four values remain on one side. If many remain on the+    // other, gallop rather than step through them one at a time.+    const size_t rem_1 = size_1 - idx_1, rem_2 = size_2 - idx_2;+    if (croaring_skewed_sizes(rem_1, rem_2)) {+        return pos + union_skewed_uint16(set_1 + idx_1, rem_1, set_2 + idx_2,+                                         rem_2, buffer + pos);+    }+    if (croaring_skewed_sizes(rem_2, rem_1)) {+        return pos + union_skewed_uint16(set_2 + idx_2, rem_2, set_1 + idx_1,+                                         rem_1, buffer + pos);+    }+    while (idx_1 < size_1 && idx_2 < size_2) {+        const uint16_t val_1 = set_1[idx_1], val_2 = set_2[idx_2];+        buffer[pos++] = val_1 < val_2 ? val_1 : val_2;+        idx_1 += (val_1 <= val_2);+        idx_2 += (val_1 >= val_2);+    }+    if (idx_1 < size_1) {+        const size_t n_elems = size_1 - idx_1;+        memmove(buffer + pos, set_1 + idx_1, n_elems * sizeof(uint16_t));+        pos += n_elems;+    } else if (idx_2 < size_2) {+        const size_t n_elems = size_2 - idx_2;+        memmove(buffer + pos, set_2 + idx_2, n_elems * sizeof(uint16_t));+        pos += n_elems;+    }+    return pos;+}++CROARING_ALLOW_UNALIGNED+int difference_uint16(const uint16_t *a1, int length1, const uint16_t *a2,+                      int length2, uint16_t *a_out) {+    // Every write goes to a_out[k..] with k <= i, so a_out may be a1.+    if (croaring_skewed_sizes((size_t)length2, (size_t)length1)) {+        return difference_skewed_small_a2_uint16(a1, length1, a2, length2,+                                                 a_out);+    }+    if (croaring_skewed_sizes((size_t)length1, (size_t)length2)) {+        return difference_skewed_small_a1_uint16(a1, length1, a2, length2,+                                                 a_out);+    }+    int i = 0, j = 0, k = 0;+    while (i + 4 <= length1 && j + 4 <= length2) {+        if (a1[i + 3] < a2[j]) {+            memmove(a_out + k, a1 + i, 4 * sizeof(uint16_t));+            k += 4;+            i += 4;+        } else if (a2[j + 3] < a1[i]) {+            j += 4;+        } else if (croaring_equal4_uint16(a1 + i, a2 + j)) {+            i += 4;+            j += 4;+        } else {+            for (int step = 0; step < 4; step++) {+                const uint16_t a = a1[i], b = a2[j];+                a_out[k] = a;+                k += (a < b);+                i += (a <= b);+                j += (a >= b);+            }+        }+    }+    // Fewer than four values remain on one side. If many remain on the+    // other, gallop rather than step through them one at a time.+    const int rem_1 = length1 - i, rem_2 = length2 - j;+    if (croaring_skewed_sizes((size_t)rem_2, (size_t)rem_1)) {+        return k + difference_skewed_small_a2_uint16(a1 + i, rem_1, a2 + j,+                                                     rem_2, a_out + k);+    }+    if (croaring_skewed_sizes((size_t)rem_1, (size_t)rem_2)) {+        return k + difference_skewed_small_a1_uint16(a1 + i, rem_1, a2 + j,+                                                     rem_2, a_out + k);+    }+    while (i < length1 && j < length2) {+        const uint16_t a = a1[i], b = a2[j];+        a_out[k] = a;+        k += (a < b);+        i += (a <= b);+        j += (a >= b);+    }+    if (i < length1) {+        memmove(a_out + k, a1 + i, sizeof(uint16_t) * (length1 - i));+        k += length1 - i;+    }+    return k;+}++CROARING_ALLOW_UNALIGNED+int32_t xor_uint16(const uint16_t *array_1, int32_t card_1,+                   const uint16_t *array_2, int32_t card_2, uint16_t *out) {+    if (croaring_skewed_sizes((size_t)card_1, (size_t)card_2)) {+        return xor_skewed_uint16(array_1, (size_t)card_1, array_2,+                                 (size_t)card_2, out);+    }+    if (croaring_skewed_sizes((size_t)card_2, (size_t)card_1)) {+        return xor_skewed_uint16(array_2, (size_t)card_2, array_1,+                                 (size_t)card_1, out);+    }+    int32_t pos1 = 0, pos2 = 0, pos_out = 0;+    while (pos1 + 4 <= card_1 && pos2 + 4 <= card_2) {+        if (array_1[pos1 + 3] < array_2[pos2]) {+            memcpy(out + pos_out, array_1 + pos1, 4 * sizeof(uint16_t));+            pos_out += 4;+            pos1 += 4;+        } else if (array_2[pos2 + 3] < array_1[pos1]) {+            memcpy(out + pos_out, array_2 + pos2, 4 * sizeof(uint16_t));+            pos_out += 4;+            pos2 += 4;+        } else if (croaring_equal4_uint16(array_1 + pos1, array_2 + pos2)) {+            pos1 += 4;+            pos2 += 4;+        } else {+            for (int step = 0; step < 4; step++) {+                const uint16_t v1 = array_1[pos1], v2 = array_2[pos2];+                out[pos_out] = v1 < v2 ? v1 : v2;+                pos_out += (v1 != v2);+                pos1 += (v1 <= v2);+                pos2 += (v1 >= v2);+            }+        }+    }+    // Fewer than four values remain on one side. If many remain on the+    // other, gallop rather than step through them one at a time.+    const size_t rem_1 = (size_t)(card_1 - pos1),+                 rem_2 = (size_t)(card_2 - pos2);+    if (croaring_skewed_sizes(rem_1, rem_2)) {+        return pos_out + xor_skewed_uint16(array_1 + pos1, rem_1,+                                           array_2 + pos2, rem_2,+                                           out + pos_out);+    }+    if (croaring_skewed_sizes(rem_2, rem_1)) {+        return pos_out + xor_skewed_uint16(array_2 + pos2, rem_2,+                                           array_1 + pos1, rem_1,+                                           out + pos_out);+    }+    while (pos1 < card_1 && pos2 < card_2) {+        const uint16_t v1 = array_1[pos1], v2 = array_2[pos2];+        out[pos_out] = v1 < v2 ? v1 : v2;+        pos_out += (v1 != v2);+        pos1 += (v1 <= v2);+        pos2 += (v1 >= v2);+    }+    if (pos1 < card_1) {+        const size_t n_elems = card_1 - pos1;+        memcpy(out + pos_out, array_1 + pos1, n_elems * sizeof(uint16_t));+        pos_out += (int32_t)n_elems;+    } else if (pos2 < card_2) {+        const size_t n_elems = card_2 - pos2;+        memcpy(out + pos_out, array_2 + pos2, n_elems * sizeof(uint16_t));+        pos_out += (int32_t)n_elems;+    }+    return pos_out;+}++#if CROARING_IS_X64++/***+ * start of the SIMD 16-bit union code+ *+ */+CROARING_TARGET_AVX2++// Merge two ascending vectors: *vecMin receives the 8 smallest values in+// ascending order, *vecMax the 8 largest, also ascending. `b` is reversed, so+// pass the operand that is not on a loop-carried chain there.+//+// This is the perfect-shuffle form of the bitonic merge (Stone, "Parallel+// processing with the perfect shuffle", 1971): a lane-wise min/max between the+// two registers followed by a 16-bit interleave, four times. A bitonic merge of+// 16 values takes log2(16) = 4 compare-exchange stages, at distances 8, 4, 2+// and 1. The interleave lines up the next stage's pairs in the same lanes, so+// no stage needs a blend, and after the fourth interleave the 16 values are+// back in sorted lane order. Only two operations per stage sit on the chain+// through *vecMax.+static inline void sse_merge(__m128i a, __m128i b, __m128i *vecMin,+                             __m128i *vecMax) {+    const __m128i rev =+        _mm_setr_epi8(14, 15, 12, 13, 10, 11, 8, 9, 6, 7, 4, 5, 2, 3, 0, 1);+    __m128i x = a;+    __m128i y = _mm_shuffle_epi8(b, rev);+    for (int i = 0; i < 4; i++) {+        __m128i mn = _mm_min_epu16(x, y);+        __m128i mx = _mm_max_epu16(x, y);+        x = _mm_unpacklo_epi16(mn, mx);+        y = _mm_unpackhi_epi16(mn, mx);+    }+    *vecMin = x;+    *vecMax = y;+}+CROARING_UNTARGET_AVX2+// used by store_unique, generated by simdunion.py+static uint8_t uniqshuf[] = {+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,+    0xc,  0xd,  0xe,  0xf,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xa,  0xb,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0xa,  0xb,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xa,  0xb,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xa,  0xb,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,+    0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,+    0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x8,  0x9,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x8,  0x9,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0xc,  0xd,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0xc,  0xd,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xc,  0xd,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,+    0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x8,  0x9,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x8,  0x9,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x8,  0x9,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,  0xa,  0xb,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x6,  0x7,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0xa,  0xb,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0xa,  0xb,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xa,  0xb,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x8,  0x9,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0xe,  0xf,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xe,  0xf,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xe,  0xf,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x8,  0x9,+    0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x8,  0x9,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0xa,  0xb,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0xa,  0xb,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xa,  0xb,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,+    0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,  0x8,  0x9,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0x8,  0x9,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x8,  0x9,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x8,  0x9,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x8,  0x9,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x6,  0x7,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,  0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x6,  0x7,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0xc,  0xd,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0xc,  0xd,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xc,  0xd,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xa,  0xb,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x8,  0x9,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xa,  0xb,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0xa,  0xb,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xa,  0xb,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xa,  0xb,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0x6,  0x7,  0x8,  0x9,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x6,  0x7,+    0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,  0x8,  0x9,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x6,  0x7,+    0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x6,  0x7,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x8,  0x9,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,+    0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x4,  0x5,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x8,  0x9,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x8,  0x9,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x4,  0x5,  0x6,  0x7,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,+    0x6,  0x7,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x4,  0x5,  0x6,  0x7,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,  0x6,  0x7,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0x6,  0x7,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x6,  0x7,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x6,  0x7,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x2,  0x3,+    0x4,  0x5,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x2,  0x3,  0x4,  0x5,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0x4,  0x5,  0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x4,  0x5,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0x0,  0x1,  0x2,  0x3,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0x2,  0x3,  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x0,  0x1,  0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,+    0xFF, 0xFF, 0xFF, 0xFF};+CROARING_TARGET_AVX2+// write vector new, while omitting repeated values assuming that previously+// written vector was "old"+static inline int store_unique(__m128i old, __m128i newval, uint16_t *output) {+    __m128i vecTmp = _mm_alignr_epi8(newval, old, 16 - 2);+    // lots of high latency instructions follow (optimize?)+    int M = _mm_movemask_epi8(+        _mm_packs_epi16(_mm_cmpeq_epi16(vecTmp, newval), _mm_setzero_si128()));+    int numberofnewvalues = 8 - _mm_popcnt_u32(M);+    __m128i key = _mm_lddqu_si128((const __m128i *)uniqshuf + M);+    __m128i val = _mm_shuffle_epi8(newval, key);+    _mm_storeu_si128((__m128i *)output, val);+    return numberofnewvalues;+}+CROARING_UNTARGET_AVX2++// working in-place, this function overwrites the repeated values+// could be avoided?+static inline uint32_t unique(uint16_t *out, uint32_t len) {+    uint32_t pos = 1;+    for (uint32_t i = 1; i < len; ++i) {+        if (out[i] != out[i - 1]) {+            out[pos++] = out[i];+        }+    }+    return pos;+}++// Sort a very short run of uint16 values in place. The callers below feed this+// at most 16 values; calling qsort() for that costs more in glibc merge-sort+// setup, function-pointer compares and memmove traffic than the sort itself.+static inline void sort_uint16_short(uint16_t *z, uint32_t n) {+    for (uint32_t i = 1; i < n; i++) {+        uint16_t v = z[i];+        uint32_t j = i;+        while (j > 0 && z[j - 1] > v) {+            z[j] = z[j - 1];+            j--;+        }+        z[j] = v;+    }+}++CROARING_TARGET_AVX2+// a one-pass SSE union algorithm+// This function may not be safe if array1 == output or array2 == output.+CROARING_ALLOW_UNALIGNED+uint32_t union_vector16(const uint16_t *array1, uint32_t length1,+                        const uint16_t *array2, uint32_t length2,+                        uint16_t *output) {+    if ((length1 < 8) || (length2 < 8)) {+        return (uint32_t)union_uint16(array1, length1, array2, length2, output);+    }+    __m128i vA, vB, V, vecMin, vecMax;+    __m128i laststore;+    uint16_t *initoutput = output;+    uint32_t len1 = length1 / 8;+    uint32_t len2 = length2 / 8;+    uint32_t pos1 = 0;+    uint32_t pos2 = 0;+    // we start the machine+    vA = _mm_lddqu_si128((const __m128i *)array1 + pos1);+    pos1++;+    vB = _mm_lddqu_si128((const __m128i *)array2 + pos2);+    pos2++;+    sse_merge(vA, vB, &vecMin, &vecMax);+    laststore = _mm_set1_epi16(-1);+    output += store_unique(laststore, vecMin, output);+    laststore = vecMin;+    if ((pos1 < len1) && (pos2 < len2)) {+        uint16_t curA, curB;+        curA = array1[8 * pos1];+        curB = array2[8 * pos2];+        while (true) {+            if (curA <= curB) {+                V = _mm_lddqu_si128((const __m128i *)array1 + pos1);+                pos1++;+                if (pos1 < len1) {+                    curA = array1[8 * pos1];+                } else {+                    break;+                }+            } else {+                V = _mm_lddqu_si128((const __m128i *)array2 + pos2);+                pos2++;+                if (pos2 < len2) {+                    curB = array2[8 * pos2];+                } else {+                    break;+                }+            }+            sse_merge(vecMax, V, &vecMin, &vecMax);+            output += store_unique(laststore, vecMin, output);+            laststore = vecMin;+        }+        sse_merge(vecMax, V, &vecMin, &vecMax);+        output += store_unique(laststore, vecMin, output);+        laststore = vecMin;+    }+    // we finish the rest off using a scalar algorithm+    // could be improved?+    //+    // copy the small end on a tmp buffer+    uint32_t len = (uint32_t)(output - initoutput);+    uint16_t buffer[16];+    uint32_t leftoversize = store_unique(laststore, vecMax, buffer);+    if (pos1 == len1) {+        memcpy(buffer + leftoversize, array1 + 8 * pos1,+               (length1 - 8 * len1) * sizeof(uint16_t));+        leftoversize += length1 - 8 * len1;+        sort_uint16_short(buffer, leftoversize);++        leftoversize = unique(buffer, leftoversize);+        len += (uint32_t)union_uint16(buffer, leftoversize, array2 + 8 * pos2,+                                      length2 - 8 * pos2, output);+    } else {+        memcpy(buffer + leftoversize, array2 + 8 * pos2,+               (length2 - 8 * len2) * sizeof(uint16_t));+        leftoversize += length2 - 8 * len2;+        sort_uint16_short(buffer, leftoversize);+        leftoversize = unique(buffer, leftoversize);+        len += (uint32_t)union_uint16(buffer, leftoversize, array1 + 8 * pos1,+                                      length1 - 8 * pos1, output);+    }+    return len;+}+CROARING_UNTARGET_AVX2++/**+ * End of the SIMD 16-bit union code+ *+ */++/**+ * Start of the AVX-512 16-bit union code.+ *+ * union_vector16 above merges 8 lanes at a time: 4 compare-exchange stages per+ * 8 output values. With AVX-512 we can merge 32+32 lanes with a Batcher bitonic+ * network: one reverse plus 5 compare-exchange stages per sorted half, i.e.+ * fewer operations per output value. This needs cross-lane 16-bit permutes+ * (vpermw, vpermt2w) and a 16-bit compress store (vpcompressw); AVX2 has none+ * of these, so the algorithm is genuinely AVX-512-only rather than a wider+ * rerun of the SSE code.+ */+#if CROARING_COMPILER_SUPPORTS_AVX512++CROARING_TARGET_AVX512++// A compare-exchange at distance d pairs lane i with lane i^d and keeps the+// smaller value in whichever lane has its d-bit clear. `hi` selects the lanes+// whose d-bit is set.+static inline __m512i avx512_cx16(__m512i v, __m512i t, __mmask32 hi) {+    return _mm512_mask_min_epu16(_mm512_max_epu16(v, t), ~hi, v, t);+}++// Sort a bitonic 32-lane sequence into ascending order. Each distance has a+// dedicated cheap shuffle, so no stage needs a full vpermw.+static inline __m512i avx512_bitonic_sort32(__m512i v) {+    v = avx512_cx16(v, _mm512_shuffle_i64x2(v, v, 0x4E), 0xFFFF0000u);  // d=16+    v = avx512_cx16(v, _mm512_shuffle_i64x2(v, v, 0xB1), 0xFF00FF00u);  // d=8+    v = avx512_cx16(v, _mm512_shuffle_epi32(v, 0x4E), 0xF0F0F0F0u);     // d=4+    v = avx512_cx16(v, _mm512_shuffle_epi32(v, 0xB1), 0xCCCCCCCCu);     // d=2+    v = avx512_cx16(v, _mm512_rol_epi32(v, 16), 0xAAAAAAAAu);           // d=1+    return v;+}++// Merge two ascending 32-lane vectors: *lo receives the 32 smallest values in+// ascending order, *hi the 32 largest, also ascending.+static inline void avx512_bitonic_merge32(__m512i a, __m512i b, __m512i *lo,+                                          __m512i *hi) {+    static const uint16_t revtab[32] = {+        31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16,+        15, 14, 13, 12, 11, 10, 9,  8,  7,  6,  5,  4,  3,  2,  1,  0};+    __m512i br = _mm512_permutexvar_epi16(+        _mm512_loadu_si512((const __m512i *)revtab), b);+    *lo = avx512_bitonic_sort32(_mm512_min_epu16(a, br));+    *hi = avx512_bitonic_sort32(_mm512_max_epu16(a, br));+}++// Write the values of the ascending vector `v` that differ from their+// predecessor, where the predecessor of lane 0 is *last. Updates *last to the+// largest value emitted and returns how many values were written.+static inline int avx512_emit_unique16(__m512i v, uint16_t *out,+                                       uint16_t *last) {+    static const uint16_t shift1[32] = {+        32, 0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14,+        15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30};+    __m512i prev = _mm512_permutex2var_epi16(+        v, _mm512_loadu_si512((const __m512i *)shift1),+        _mm512_set1_epi16((short)*last));+    __mmask32 keep = _mm512_cmpneq_epi16_mask(v, prev);+    _mm512_mask_compressstoreu_epi16(out, keep, v);+    *last = (uint16_t)_mm_extract_epi16(_mm512_extracti32x4_epi32(v, 3), 7);+    return (int)roaring_hamming(keep);+}++/**+ * A one-pass AVX-512 union of two sorted uint16 arrays.+ *+ * As with union_vector16, the caller must guarantee that `output` has room for+ * size_1 + size_2 values. Partial overlap of `output` with the inputs is+ * tolerated in exactly the same way union_vector16 tolerates it (see the long+ * comment in array_run_container_inplace_union): after consuming p1 + p2 whole+ * 32-lane blocks this routine has written at most 32*(p1+p2) - 32 values,+ * because one merged block is always held back in `vmax`. The output pointer+ * therefore stays strictly behind the read pointers.+ *+ * That bound is exactly tight rather than merely satisfied: in the worst case+ * the final merge below writes its last value to precisely the slot the last+ * input value was read from. Anything that reduces how much the tail holds back+ * -- shrinking `pending`, emitting `vmax` sooner -- silently breaks aliased+ * callers, so re-derive the bound before changing the buffering here.+ */+CROARING_ALLOW_UNALIGNED+uint32_t avx512_union_uint16(const uint16_t *array1, uint32_t length1,+                             const uint16_t *array2, uint32_t length2,+                             uint16_t *output) {+    const uint32_t W = 32;  // lanes per 512-bit register+    if (length1 < W || length2 < W) {+        // Not enough on one side to fill a block. union_vector16 keeps+        // vectorizing at 8-lane granularity, so it still beats the scalar+        // merge once there is a worthwhile amount of data on the other side;+        // below that its own fixed overhead dominates.+        if (length1 + length2 >= 64) {+            return union_vector16(array1, length1, array2, length2, output);+        }+        return (uint32_t)union_uint16(array1, length1, array2, length2, output);+    }+    const uint32_t blocks1 = length1 / W, blocks2 = length2 / W;+    uint32_t p1 = 0, p2 = 0;+    uint16_t *out = output;+    __m512i vmin, vmax;++    avx512_bitonic_merge32(_mm512_loadu_si512((const __m512i *)array1),+                           _mm512_loadu_si512((const __m512i *)array2), &vmin,+                           &vmax);+    p1 = 1;+    p2 = 1;+    // Lane 0 of the very first block has no predecessor. Seeding `last` with a+    // value that differs from it in one bit keeps the first value.+    uint16_t last =+        (uint16_t)(_mm_extract_epi16(_mm512_castsi512_si128(vmin), 0) ^ 1);+    out += avx512_emit_unique16(vmin, out, &last);++    while (p1 < blocks1 && p2 < blocks2) {+        // Which side advances is essentially unpredictable, so select the+        // source pointer without branching.+        const uint16_t *pa = array1 + W * p1;+        const uint16_t *pb = array2 + W * p2;+        const uint32_t take1 = (*pa <= *pb) ? 1 : 0;+        const __m512i v =+            _mm512_loadu_si512((const __m512i *)(take1 ? pa : pb));+        p1 += take1;+        p2 += 1 - take1;+        avx512_bitonic_merge32(vmax, v, &vmin, &vmax);+        out += avx512_emit_unique16(vmin, out, &last);+    }++    // The ragged tail: the values still held in vmax, plus at most W-1 leftover+    // values from the exhausted side, plus the rest of the other side. Two+    // two-way merges branch-predict far better than one three-way merge.+    //+    // vmax can hold the same value twice (once from each input), so it goes+    // through avx512_emit_unique16 rather than a raw store: that dedups within+    // the block and against `last` at the same time. Every remaining tail value+    // is >= last and the tail is sorted, so for the two scalar inputs only the+    // very first value can still duplicate `last`.+    uint16_t pending[32];+    uint16_t pending_last = last;+    size_t npending =+        (size_t)avx512_emit_unique16(vmax, pending, &pending_last);++    const uint16_t *rest1 = array1 + W * p1, *rest2 = array2 + W * p2;+    size_t nrest1 = length1 - W * p1, nrest2 = length2 - W * p2;+    const uint16_t *shortrest, *longrest;+    size_t nshort, nlong;+    if (p1 == blocks1) {+        shortrest = rest1;+        nshort = nrest1;+        longrest = rest2;+        nlong = nrest2;+    } else {+        shortrest = rest2;+        nshort = nrest2;+        longrest = rest1;+        nlong = nrest1;+    }+    if (nshort > 0 && shortrest[0] == last) {+        shortrest++;+        nshort--;+    }+    if (nlong > 0 && longrest[0] == last) {+        longrest++;+        nlong--;+    }++    uint16_t merged[2 * 32];  // <= W pending + (W-1) leftover+    size_t nmerged = union_uint16(pending, npending, shortrest, nshort, merged);+    // When the small side held fewer than two blocks the loop above ran at most+    // once, so `longrest` can still be most of the larger input. Finishing that+    // scalar would give away more than the block loop won, hence the same+    // 8-lane handoff as in the short-input case.+    if (nlong >= 64) {+        out += union_vector16(merged, (uint32_t)nmerged, longrest,+                              (uint32_t)nlong, out);+    } else {+        out += union_uint16(merged, nmerged, longrest, nlong, out);+    }+    return (uint32_t)(out - output);+}+CROARING_UNTARGET_AVX512+#endif  // CROARING_COMPILER_SUPPORTS_AVX512++/**+ * End of the AVX-512 16-bit union code.+ */++/**+ * Start of SIMD 16-bit XOR code+ */++CROARING_TARGET_AVX2+// write vector new, while omitting repeated values assuming that previously+// written vector was "old"+static inline int store_unique_xor(__m128i old, __m128i newval,+                                   uint16_t *output) {+    __m128i vecTmp1 = _mm_alignr_epi8(newval, old, 16 - 4);+    __m128i vecTmp2 = _mm_alignr_epi8(newval, old, 16 - 2);+    __m128i equalleft = _mm_cmpeq_epi16(vecTmp2, vecTmp1);+    __m128i equalright = _mm_cmpeq_epi16(vecTmp2, newval);+    __m128i equalleftoright = _mm_or_si128(equalleft, equalright);+    int M = _mm_movemask_epi8(+        _mm_packs_epi16(equalleftoright, _mm_setzero_si128()));+    int numberofnewvalues = 8 - _mm_popcnt_u32(M);+    __m128i key = _mm_lddqu_si128((const __m128i *)uniqshuf + M);+    __m128i val = _mm_shuffle_epi8(vecTmp2, key);+    _mm_storeu_si128((__m128i *)output, val);+    return numberofnewvalues;+}+CROARING_UNTARGET_AVX2++// working in-place, this function overwrites the repeated values+// could be avoided? Warning: assumes len > 0+static inline uint32_t unique_xor(uint16_t *out, uint32_t len) {+    uint32_t pos = 1;+    for (uint32_t i = 1; i < len; ++i) {+        if (out[i] != out[i - 1]) {+            out[pos++] = out[i];+        } else+            pos--;  // if it is identical to previous, delete it+    }+    return pos;+}+CROARING_TARGET_AVX2+// a one-pass SSE xor algorithm+CROARING_ALLOW_UNALIGNED+uint32_t xor_vector16(const uint16_t *array1, uint32_t length1,+                      const uint16_t *array2, uint32_t length2,+                      uint16_t *output) {+    if ((length1 < 8) || (length2 < 8)) {+        return xor_uint16(array1, length1, array2, length2, output);+    }+    __m128i vA, vB, V, vecMin, vecMax;+    __m128i laststore;+    uint16_t *initoutput = output;+    uint32_t len1 = length1 / 8;+    uint32_t len2 = length2 / 8;+    uint32_t pos1 = 0;+    uint32_t pos2 = 0;+    // we start the machine+    vA = _mm_lddqu_si128((const __m128i *)array1 + pos1);+    pos1++;+    vB = _mm_lddqu_si128((const __m128i *)array2 + pos2);+    pos2++;+    sse_merge(vA, vB, &vecMin, &vecMax);+    laststore = _mm_set1_epi16(-1);+    uint16_t buffer[17];+    output += store_unique_xor(laststore, vecMin, output);++    laststore = vecMin;+    if ((pos1 < len1) && (pos2 < len2)) {+        uint16_t curA, curB;+        curA = array1[8 * pos1];+        curB = array2[8 * pos2];+        while (true) {+            if (curA <= curB) {+                V = _mm_lddqu_si128((const __m128i *)array1 + pos1);+                pos1++;+                if (pos1 < len1) {+                    curA = array1[8 * pos1];+                } else {+                    break;+                }+            } else {+                V = _mm_lddqu_si128((const __m128i *)array2 + pos2);+                pos2++;+                if (pos2 < len2) {+                    curB = array2[8 * pos2];+                } else {+                    break;+                }+            }+            sse_merge(vecMax, V, &vecMin, &vecMax);+            // conditionally stores the last value of laststore as well as all+            // but the+            // last value of vecMin+            output += store_unique_xor(laststore, vecMin, output);+            laststore = vecMin;+        }+        sse_merge(vecMax, V, &vecMin, &vecMax);+        // conditionally stores the last value of laststore as well as all but+        // the+        // last value of vecMin+        output += store_unique_xor(laststore, vecMin, output);+        laststore = vecMin;+    }+    uint32_t len = (uint32_t)(output - initoutput);++    // we finish the rest off using a scalar algorithm+    // could be improved?+    // conditionally stores the last value of laststore as well as all but the+    // last value of vecMax,+    // we store to "buffer"+    int leftoversize = store_unique_xor(laststore, vecMax, buffer);+    uint16_t vec7 = (uint16_t)_mm_extract_epi16(vecMax, 7);+    uint16_t vec6 = (uint16_t)_mm_extract_epi16(vecMax, 6);+    if (vec7 != vec6) buffer[leftoversize++] = vec7;+    if (pos1 == len1) {+        memcpy(buffer + leftoversize, array1 + 8 * pos1,+               (length1 - 8 * len1) * sizeof(uint16_t));+        leftoversize += length1 - 8 * len1;+        if (leftoversize == 0) {  // trivial case+            memcpy(output, array2 + 8 * pos2,+                   (length2 - 8 * pos2) * sizeof(uint16_t));+            len += (length2 - 8 * pos2);+        } else {+            sort_uint16_short(buffer, leftoversize);+            leftoversize = unique_xor(buffer, leftoversize);+            len += xor_uint16(buffer, leftoversize, array2 + 8 * pos2,+                              length2 - 8 * pos2, output);+        }+    } else {+        memcpy(buffer + leftoversize, array2 + 8 * pos2,+               (length2 - 8 * len2) * sizeof(uint16_t));+        leftoversize += length2 - 8 * len2;+        if (leftoversize == 0) {  // trivial case+            memcpy(output, array1 + 8 * pos1,+                   (length1 - 8 * pos1) * sizeof(uint16_t));+            len += (length1 - 8 * pos1);+        } else {+            sort_uint16_short(buffer, leftoversize);+            leftoversize = unique_xor(buffer, leftoversize);+            len += xor_uint16(buffer, leftoversize, array1 + 8 * pos1,+                              length1 - 8 * pos1, output);+        }+    }+    return len;+}+CROARING_UNTARGET_AVX2+/**+ * End of SIMD 16-bit XOR code+ */++#endif  // CROARING_IS_X64++size_t union_uint32(const uint32_t *set_1, size_t size_1, const uint32_t *set_2,+                    size_t size_2, uint32_t *buffer) {+    size_t pos = 0, idx_1 = 0, idx_2 = 0;++    if (0 == size_2) {+        memmove(buffer, set_1, size_1 * sizeof(uint32_t));+        return size_1;+    }+    if (0 == size_1) {+        memmove(buffer, set_2, size_2 * sizeof(uint32_t));+        return size_2;+    }++    uint32_t val_1 = set_1[idx_1], val_2 = set_2[idx_2];++    while (true) {+        if (val_1 < val_2) {+            buffer[pos++] = val_1;+            ++idx_1;+            if (idx_1 >= size_1) break;+            val_1 = set_1[idx_1];+        } else if (val_2 < val_1) {+            buffer[pos++] = val_2;+            ++idx_2;+            if (idx_2 >= size_2) break;+            val_2 = set_2[idx_2];+        } else {+            buffer[pos++] = val_1;+            ++idx_1;+            ++idx_2;+            if (idx_1 >= size_1 || idx_2 >= size_2) break;+            val_1 = set_1[idx_1];+            val_2 = set_2[idx_2];+        }+    }++    if (idx_1 < size_1) {+        const size_t n_elems = size_1 - idx_1;+        memmove(buffer + pos, set_1 + idx_1, n_elems * sizeof(uint32_t));+        pos += n_elems;+    } else if (idx_2 < size_2) {+        const size_t n_elems = size_2 - idx_2;+        memmove(buffer + pos, set_2 + idx_2, n_elems * sizeof(uint32_t));+        pos += n_elems;+    }++    return pos;+}++size_t union_uint32_card(const uint32_t *set_1, size_t size_1,+                         const uint32_t *set_2, size_t size_2) {+    size_t pos = 0, idx_1 = 0, idx_2 = 0;++    if (0 == size_2) {+        return size_1;+    }+    if (0 == size_1) {+        return size_2;+    }++    uint32_t val_1 = set_1[idx_1], val_2 = set_2[idx_2];++    while (true) {+        if (val_1 < val_2) {+            ++idx_1;+            ++pos;+            if (idx_1 >= size_1) break;+            val_1 = set_1[idx_1];+        } else if (val_2 < val_1) {+            ++idx_2;+            ++pos;+            if (idx_2 >= size_2) break;+            val_2 = set_2[idx_2];+        } else {+            ++idx_1;+            ++idx_2;+            ++pos;+            if (idx_1 >= size_1 || idx_2 >= size_2) break;+            val_1 = set_1[idx_1];+            val_2 = set_2[idx_2];+        }+    }++    if (idx_1 < size_1) {+        const size_t n_elems = size_1 - idx_1;+        pos += n_elems;+    } else if (idx_2 < size_2) {+        const size_t n_elems = size_2 - idx_2;+        pos += n_elems;+    }+    return pos;+}++size_t fast_union_uint16(const uint16_t *set_1, size_t size_1,+                         const uint16_t *set_2, size_t size_2,+                         uint16_t *buffer) {+#if CROARING_IS_X64+    if (size_1 * CROARING_SKEWED_VECTOR_MERGE_RATIO < size_2 ||+        size_2 * CROARING_SKEWED_VECTOR_MERGE_RATIO < size_1) {+        return union_uint16(set_1, size_1, set_2, size_2, buffer);+    }+    const unsigned support = (unsigned)croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX512) {+        // compute union with smallest array first+        if (size_1 < size_2) {+            return avx512_union_uint16(set_1, (uint32_t)size_1, set_2,+                                       (uint32_t)size_2, buffer);+        } else {+            return avx512_union_uint16(set_2, (uint32_t)size_2, set_1,+                                       (uint32_t)size_1, buffer);+        }+    }+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX2) {+        // compute union with smallest array first+        if (size_1 < size_2) {+            return union_vector16(set_1, (uint32_t)size_1, set_2,+                                  (uint32_t)size_2, buffer);+        } else {+            return union_vector16(set_2, (uint32_t)size_2, set_1,+                                  (uint32_t)size_1, buffer);+        }+    } else {+        // compute union with smallest array first+        if (size_1 < size_2) {+            return union_uint16(set_1, size_1, set_2, size_2, buffer);+        } else {+            return union_uint16(set_2, size_2, set_1, size_1, buffer);+        }+    }+#else+    // compute union with smallest array first+    if (size_1 < size_2) {+        return union_uint16(set_1, size_1, set_2, size_2, buffer);+    } else {+        return union_uint16(set_2, size_2, set_1, size_1, buffer);+    }+#endif+}+#if CROARING_IS_X64+#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+CROARING_ALLOW_UNALIGNED+static inline bool _avx512_memequals(const void *s1, const void *s2, size_t n) {+    const uint8_t *ptr1 = (const uint8_t *)s1;+    const uint8_t *ptr2 = (const uint8_t *)s2;+    const uint8_t *end1 = ptr1 + n;+    const uint8_t *end8 = ptr1 + ((n >> 3) << 3);+    const uint8_t *end32 = ptr1 + ((n >> 5) << 5);+    const uint8_t *end64 = ptr1 + ((n >> 6) << 6);++    while (ptr1 < end64) {+        __m512i r1 = _mm512_loadu_si512((const __m512i *)ptr1);+        __m512i r2 = _mm512_loadu_si512((const __m512i *)ptr2);++        uint64_t mask = _mm512_cmpeq_epi8_mask(r1, r2);++        if (mask != UINT64_MAX) {+            return false;+        }++        ptr1 += 64;+        ptr2 += 64;+    }++    while (ptr1 < end32) {+        __m256i r1 = _mm256_loadu_si256((const __m256i *)ptr1);+        __m256i r2 = _mm256_loadu_si256((const __m256i *)ptr2);+        int mask = _mm256_movemask_epi8(_mm256_cmpeq_epi8(r1, r2));+        if ((uint32_t)mask != UINT32_MAX) {+            return false;+        }+        ptr1 += 32;+        ptr2 += 32;+    }++    while (ptr1 < end8) {+        uint64_t v1, v2;+        memcpy(&v1, ptr1, sizeof(uint64_t));+        memcpy(&v2, ptr2, sizeof(uint64_t));+        if (v1 != v2) {+            return false;+        }+        ptr1 += 8;+        ptr2 += 8;+    }++    while (ptr1 < end1) {+        if (*ptr1 != *ptr2) {+            return false;+        }+        ptr1++;+        ptr2++;+    }++    return true;+}+CROARING_UNTARGET_AVX512+#endif  // CROARING_COMPILER_SUPPORTS_AVX512++CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+static inline bool _avx2_memequals(const void *s1, const void *s2, size_t n) {+    const uint8_t *ptr1 = (const uint8_t *)s1;+    const uint8_t *ptr2 = (const uint8_t *)s2;+    const uint8_t *end1 = ptr1 + n;+    const uint8_t *end8 = ptr1 + n / 8 * 8;+    const uint8_t *end32 = ptr1 + n / 32 * 32;++    while (ptr1 < end32) {+        __m256i r1 = _mm256_loadu_si256((const __m256i *)ptr1);+        __m256i r2 = _mm256_loadu_si256((const __m256i *)ptr2);+        int mask = _mm256_movemask_epi8(_mm256_cmpeq_epi8(r1, r2));+        if ((uint32_t)mask != UINT32_MAX) {+            return false;+        }+        ptr1 += 32;+        ptr2 += 32;+    }++    while (ptr1 < end8) {+        uint64_t v1, v2;+        memcpy(&v1, ptr1, sizeof(uint64_t));+        memcpy(&v2, ptr2, sizeof(uint64_t));+        if (v1 != v2) {+            return false;+        }+        ptr1 += 8;+        ptr2 += 8;+    }++    while (ptr1 < end1) {+        if (*ptr1 != *ptr2) {+            return false;+        }+        ptr1++;+        ptr2++;+    }++    return true;+}+CROARING_UNTARGET_AVX2+#endif++bool memequals(const void *s1, const void *s2, size_t n) {+    if (n == 0) {+        return true;+    }+#if CROARING_IS_X64+    int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX512) {+        return _avx512_memequals(s1, s2, n);+    } else+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+        if (support & ROARING_SUPPORTS_AVX2) {+            return _avx2_memequals(s1, s2, n);+        } else {+            return memcmp(s1, s2, n) == 0;+        }+#else+    return memcmp(s1, s2, n) == 0;+#endif+}++#if CROARING_IS_X64+#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+CROARING_ALLOW_UNALIGNED+int avx512_array_container_to_uint32_array(void *vout, const uint16_t *array,+                                           size_t cardinality, uint32_t base) {+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;+    size_t i = 0;+    for (; i + sizeof(__m256i) / sizeof(uint16_t) <= cardinality;+         i += sizeof(__m256i) / sizeof(uint16_t)) {+        __m256i vinput = _mm256_loadu_si256((const __m256i *)(array + i));+        __m512i voutput = _mm512_add_epi32(_mm512_cvtepu16_epi32(vinput),+                                           _mm512_set1_epi32(base));+        _mm512_storeu_si512((__m512i *)(out + outpos), voutput);+        outpos += sizeof(__m512i) / sizeof(uint32_t);+    }+    for (; i < cardinality; ++i) {+        const uint32_t val = base + array[i];+        memcpy(out + outpos, &val,+               sizeof(uint32_t));  // should be compiled as a MOV on x64+        outpos++;+    }+    return outpos;+}+CROARING_UNTARGET_AVX512+#endif  // #if CROARING_COMPILER_SUPPORTS_AVX512+#endif  // #if CROARING_IS_X64++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif
+ cbits/CRoaring/src/art/art.c view
@@ -0,0 +1,2455 @@+#include <assert.h>+#include <stdio.h>+#include <string.h>++#include <roaring/art/art.h>+#include <roaring/memory.h>+#include <roaring/portability.h>++#define CROARING_ART_NULL_REF 0++#define CROARING_ART_LEAF_TYPE 1+#define CROARING_ART_NODE4_TYPE 2+#define CROARING_ART_NODE16_TYPE 3+#define CROARING_ART_NODE48_TYPE 4+#define CROARING_ART_NODE256_TYPE 5++#define CROARING_ART_MIN_TYPE CROARING_ART_LEAF_TYPE+#define CROARING_ART_MAX_TYPE CROARING_ART_NODE256_TYPE++// Node48 placeholder value to indicate no child is present at this key index.+#define CROARING_ART_NODE48_EMPTY_VAL 48+#define CROARING_NODE48_AVAILABLE_CHILDREN_MASK ((UINT64_C(1) << 48) - 1)++#define CROARING_ART_ALIGN_BUF(buf, alignment)      \+    (char *)(((uintptr_t)(buf) + ((alignment)-1)) & \+             (ptrdiff_t)(~((alignment)-1)))++// Gives the byte difference needed to align the current buffer to the+// alignment, relative to the start of the buffer.+#define CROARING_ART_ALIGN_SIZE_RELATIVE(buf_cur, buf_start, alignment) \+    ((((ptrdiff_t)((buf_cur) - (buf_start)) + ((alignment)-1)) &        \+      (ptrdiff_t)(~((alignment)-1))) -                                  \+     (ptrdiff_t)((buf_cur) - (buf_start)))++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++typedef uint8_t art_typecode_t;++typedef struct art_leaf_s {+    union {+        struct {+            art_key_chunk_t key[ART_KEY_BYTES];+            art_val_t val;+        };+        uint64_t next_free;+    };+} art_leaf_t;++// Inner node, with prefix.+//+// We use a fixed-length array as a pointer would be larger than the array.+typedef struct art_inner_node_s {+    uint8_t prefix_size;+    uint8_t prefix[ART_KEY_BYTES - 1];+} art_inner_node_t;++// Inner node types.++// Node4: key[i] corresponds with children[i]. Keys are sorted.+typedef struct art_node4_s {+    union {+        struct {+            art_inner_node_t base;+            uint8_t count;+            uint8_t keys[4];+            art_ref_t children[4];+        };+        uint64_t next_free;+    };+} art_node4_t;++// Node16: key[i] corresponds with children[i]. Keys are sorted.+typedef struct art_node16_s {+    union {+        struct {+            art_inner_node_t base;+            uint8_t count;+            uint8_t keys[16];+            art_ref_t children[16];+        };+        uint64_t next_free;+    };+} art_node16_t;++// Node48: key[i] corresponds with children[key[i]] if key[i] !=+// CROARING_ART_NODE48_EMPTY_VAL. Keys are naturally sorted due to direct+// indexing.+typedef struct art_node48_s {+    union {+        struct {+            art_inner_node_t base;+            uint8_t count;+            // Bitset where the ith bit is set if children[i] is available+            // Because there are at most 48 children, only the bottom 48 bits+            // are used.+            uint64_t available_children;+            uint8_t keys[256];+            art_ref_t children[48];+        };+        uint64_t next_free;+    };+} art_node48_t;++// Node256: children[i] is directly indexed by key chunk. A child is present if+// children[i] != NULL.+typedef struct art_node256_s {+    union {+        struct {+            art_inner_node_t base;+            uint16_t count;+            art_ref_t children[256];+        };+        uint64_t next_free;+    };+} art_node256_t;++// Size of each node type, indexed by typecode for convenience.+static const size_t ART_NODE_SIZES[] = {+    0,+    sizeof(art_leaf_t),+    sizeof(art_node4_t),+    sizeof(art_node16_t),+    sizeof(art_node48_t),+    sizeof(art_node256_t),+};++// Helper struct to refer to a child within a node at a specific index.+typedef struct art_indexed_child_s {+    art_ref_t child;+    uint8_t index;+    art_key_chunk_t key_chunk;+} art_indexed_child_t;++typedef struct art_internal_validate_s {+    const char **reason;+    art_validate_cb_t validate_cb;+    void *context;++    int depth;+    art_key_chunk_t current_key[ART_KEY_BYTES];+} art_internal_validate_t;++// Set the reason message, and return false for convenience.+static inline bool art_validate_fail(const art_internal_validate_t *validate,+                                     const char *msg) {+    *validate->reason = msg;+    return false;+}++static inline art_ref_t art_to_ref(uint64_t index, art_typecode_t typecode) {+    return ((art_ref_t)index) << 16 | typecode;+}++static inline uint64_t art_ref_index(art_ref_t ref) {+    return ((uint64_t)ref) >> 16;+}++static inline art_typecode_t art_ref_typecode(art_ref_t ref) {+    return (art_typecode_t)ref;+}++/**+ * Gets a pointer to a node from its reference. The pointer only remains valid+ * under non-mutating operations. If any mutating operations occur, this+ * function should be called again to get a valid pointer to the node.+ */+static art_node_t *art_deref(const art_t *art, art_ref_t ref) {+    assert(ref != CROARING_ART_NULL_REF);+    art_typecode_t typecode = art_ref_typecode(ref);+    return (art_node_t *)((char *)art->nodes[typecode] ++                          art_ref_index(ref) * ART_NODE_SIZES[typecode]);+}++static inline art_node_t *art_get_node(const art_t *art, uint64_t index,+                                       art_typecode_t typecode) {+    return art_deref(art, art_to_ref(index, typecode));+}++static inline uint64_t art_get_index(const art_t *art, const art_node_t *node,+                                     art_typecode_t typecode) {+    art_node_t *nodes = art->nodes[typecode];+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return (art_leaf_t *)node - (art_leaf_t *)nodes;+        case CROARING_ART_NODE4_TYPE:+            return (art_node4_t *)node - (art_node4_t *)nodes;+        case CROARING_ART_NODE16_TYPE:+            return (art_node16_t *)node - (art_node16_t *)nodes;+        case CROARING_ART_NODE48_TYPE:+            return (art_node48_t *)node - (art_node48_t *)nodes;+        case CROARING_ART_NODE256_TYPE:+            return (art_node256_t *)node - (art_node256_t *)nodes;+        default:+            assert(false);+            return 0;+    }+}++/**+ * Creates a reference from a pointer.+ */+static inline art_ref_t art_get_ref(const art_t *art, const art_node_t *node,+                                    art_typecode_t typecode) {+    return art_to_ref(art_get_index(art, node, typecode), typecode);+}++static inline bool art_is_leaf(art_ref_t ref) {+    return art_ref_typecode(ref) == CROARING_ART_LEAF_TYPE;+}++static inline void art_init_inner_node(art_inner_node_t *node,+                                       const art_key_chunk_t prefix[],+                                       uint8_t prefix_size) {+    node->prefix_size = prefix_size;+    memcpy(node->prefix, prefix, prefix_size * sizeof(art_key_chunk_t));+}++static void art_node_free(art_t *art, art_node_t *node,+                          art_typecode_t typecode);++static uint64_t art_allocate_index(art_t *art, art_typecode_t typecode);++// ===================== Start of node-specific functions ======================++static art_ref_t art_leaf_create(art_t *art, const art_key_chunk_t key[],+                                 art_val_t val) {+    uint64_t index = art_allocate_index(art, CROARING_ART_LEAF_TYPE);+    art_leaf_t *leaf =+        ((art_leaf_t *)art->nodes[CROARING_ART_LEAF_TYPE]) + index;+    memcpy(leaf->key, key, ART_KEY_BYTES);+    leaf->val = val;+    return art_to_ref(index, CROARING_ART_LEAF_TYPE);+}++static art_node4_t *art_node4_create(art_t *art, const art_key_chunk_t prefix[],+                                     uint8_t prefix_size);+static art_node16_t *art_node16_create(art_t *art,+                                       const art_key_chunk_t prefix[],+                                       uint8_t prefix_size);+static art_node48_t *art_node48_create(art_t *art,+                                       const art_key_chunk_t prefix[],+                                       uint8_t prefix_size);+static art_node256_t *art_node256_create(art_t *art,+                                         const art_key_chunk_t prefix[],+                                         uint8_t prefix_size);++static art_ref_t art_node4_insert(art_t *art, art_node4_t *node,+                                  art_ref_t child, uint8_t key);+static art_ref_t art_node16_insert(art_t *art, art_node16_t *node,+                                   art_ref_t child, uint8_t key);+static art_ref_t art_node48_insert(art_t *art, art_node48_t *node,+                                   art_ref_t child, uint8_t key);+static art_ref_t art_node256_insert(art_t *art, art_node256_t *node,+                                    art_ref_t child, uint8_t key);++static art_node4_t *art_node4_create(art_t *art, const art_key_chunk_t prefix[],+                                     uint8_t prefix_size) {+    uint64_t index = art_allocate_index(art, CROARING_ART_NODE4_TYPE);+    art_node4_t *node =+        ((art_node4_t *)art->nodes[CROARING_ART_NODE4_TYPE]) + index;+    art_init_inner_node(&node->base, prefix, prefix_size);+    node->count = 0;+    return node;+}++static inline art_ref_t art_node4_find_child(const art_node4_t *node,+                                             art_key_chunk_t key) {+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key) {+            return node->children[i];+        }+    }+    return CROARING_ART_NULL_REF;+}++static art_ref_t art_node4_insert(art_t *art, art_node4_t *node,+                                  art_ref_t child, uint8_t key) {+    if (node->count < 4) {+        size_t idx = 0;+        for (; idx < node->count; ++idx) {+            if (node->keys[idx] > key) {+                break;+            }+        }+        size_t after = node->count - idx;+        // Shift other keys to maintain sorted order.+        memmove(node->keys + idx + 1, node->keys + idx,+                after * sizeof(art_key_chunk_t));+        memmove(node->children + idx + 1, node->children + idx,+                after * sizeof(art_ref_t));++        node->children[idx] = child;+        node->keys[idx] = key;+        node->count++;+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE4_TYPE);+    }+    art_node16_t *new_node =+        art_node16_create(art, node->base.prefix, node->base.prefix_size);+    // Instead of calling insert, this could be specialized to 2x memcpy and+    // setting the count.+    for (size_t i = 0; i < 4; ++i) {+        art_node16_insert(art, new_node, node->children[i], node->keys[i]);+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE4_TYPE);+    return art_node16_insert(art, new_node, child, key);+}++static inline art_ref_t art_node4_erase(art_t *art, art_node4_t *node,+                                        art_key_chunk_t key_chunk) {+    int idx = -1;+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key_chunk) {+            idx = i;+        }+    }+    if (idx == -1) {+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE4_TYPE);+    }+    if (node->count == 2) {+        // Only one child remains after erasing, so compress the path by+        // removing this node.+        uint8_t other_idx = idx ^ 1;+        art_ref_t remaining_child = node->children[other_idx];+        art_key_chunk_t remaining_child_key = node->keys[other_idx];+        if (!art_is_leaf(remaining_child)) {+            // Correct the prefix of the child node.+            art_inner_node_t *inner_node =+                (art_inner_node_t *)art_deref(art, remaining_child);+            memmove(inner_node->prefix + node->base.prefix_size + 1,+                    inner_node->prefix, inner_node->prefix_size);+            memcpy(inner_node->prefix, node->base.prefix,+                   node->base.prefix_size);+            inner_node->prefix[node->base.prefix_size] = remaining_child_key;+            inner_node->prefix_size += node->base.prefix_size + 1;+        }+        art_node_free(art, (art_node_t *)node, CROARING_ART_NODE4_TYPE);+        return remaining_child;+    }+    // Shift other keys to maintain sorted order.+    size_t after_next = node->count - idx - 1;+    memmove(node->keys + idx, node->keys + idx + 1,+            after_next * sizeof(art_key_chunk_t));+    memmove(node->children + idx, node->children + idx + 1,+            after_next * sizeof(art_ref_t));+    node->count--;+    return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE4_TYPE);+}++static inline void art_node4_replace(art_node4_t *node,+                                     art_key_chunk_t key_chunk,+                                     art_ref_t new_child) {+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key_chunk) {+            node->children[i] = new_child;+            return;+        }+    }+}++static inline art_indexed_child_t art_node4_next_child(const art_node4_t *node,+                                                       int index) {+    art_indexed_child_t indexed_child;+    index++;+    if (index >= node->count) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node4_prev_child(const art_node4_t *node,+                                                       int index) {+    if (index > node->count) {+        index = node->count;+    }+    index--;+    art_indexed_child_t indexed_child;+    if (index < 0) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node4_child_at(const art_node4_t *node,+                                                     int index) {+    art_indexed_child_t indexed_child;+    if (index < 0 || index >= node->count) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node4_lower_bound(+    art_node4_t *node, art_key_chunk_t key_chunk) {+    art_indexed_child_t indexed_child;+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] >= key_chunk) {+            indexed_child.index = i;+            indexed_child.child = node->children[i];+            indexed_child.key_chunk = node->keys[i];+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static bool art_internal_validate_at(const art_t *art, art_ref_t ref,+                                     art_internal_validate_t validator);++static bool art_node4_internal_validate(const art_t *art,+                                        const art_node4_t *node,+                                        art_internal_validate_t validator) {+    if (node->count == 0) {+        return art_validate_fail(&validator, "Node4 has no children");+    }+    if (node->count > 4) {+        return art_validate_fail(&validator, "Node4 has too many children");+    }+    if (node->count == 1) {+        return art_validate_fail(+            &validator, "Node4 and child node should have been combined");+    }+    validator.depth++;+    for (int i = 0; i < node->count; ++i) {+        if (i > 0) {+            if (node->keys[i - 1] >= node->keys[i]) {+                return art_validate_fail(+                    &validator, "Node4 keys are not strictly increasing");+            }+        }+        for (int j = i + 1; j < node->count; ++j) {+            if (node->children[i] == node->children[j]) {+                return art_validate_fail(&validator,+                                         "Node4 has duplicate children");+            }+        }+        validator.current_key[validator.depth - 1] = node->keys[i];+        if (!art_internal_validate_at(art, node->children[i], validator)) {+            return false;+        }+    }+    return true;+}++static art_node16_t *art_node16_create(art_t *art,+                                       const art_key_chunk_t prefix[],+                                       uint8_t prefix_size) {+    uint64_t index = art_allocate_index(art, CROARING_ART_NODE16_TYPE);+    art_node16_t *node =+        ((art_node16_t *)art->nodes[CROARING_ART_NODE16_TYPE]) + index;+    art_init_inner_node(&node->base, prefix, prefix_size);+    node->count = 0;+    return node;+}++static inline art_ref_t art_node16_find_child(const art_node16_t *node,+                                              art_key_chunk_t key) {+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key) {+            return node->children[i];+        }+    }+    return CROARING_ART_NULL_REF;+}++static art_ref_t art_node16_insert(art_t *art, art_node16_t *node,+                                   art_ref_t child, uint8_t key) {+    if (node->count < 16) {+        size_t idx = 0;+        for (; idx < node->count; ++idx) {+            if (node->keys[idx] > key) {+                break;+            }+        }+        size_t after = node->count - idx;+        // Shift other keys to maintain sorted order.+        memmove(node->keys + idx + 1, node->keys + idx,+                after * sizeof(art_key_chunk_t));+        memmove(node->children + idx + 1, node->children + idx,+                after * sizeof(art_ref_t));++        node->children[idx] = child;+        node->keys[idx] = key;+        node->count++;+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE16_TYPE);+    }+    art_node48_t *new_node =+        art_node48_create(art, node->base.prefix, node->base.prefix_size);+    for (size_t i = 0; i < 16; ++i) {+        art_node48_insert(art, new_node, node->children[i], node->keys[i]);+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE16_TYPE);+    return art_node48_insert(art, new_node, child, key);+}++static inline art_ref_t art_node16_erase(art_t *art, art_node16_t *node,+                                         uint8_t key_chunk) {+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key_chunk) {+            // Shift other keys to maintain sorted order.+            size_t after_next = node->count - i - 1;+            memmove(node->keys + i, node->keys + i + 1,+                    after_next * sizeof(key_chunk));+            memmove(node->children + i, node->children + i + 1,+                    after_next * sizeof(art_ref_t));+            node->count--;+            break;+        }+    }+    if (node->count > 4) {+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE16_TYPE);+    }+    art_node4_t *new_node =+        art_node4_create(art, node->base.prefix, node->base.prefix_size);+    // Instead of calling insert, this could be specialized to 2x memcpy and+    // setting the count.+    for (size_t i = 0; i < 4; ++i) {+        art_node4_insert(art, new_node, node->children[i], node->keys[i]);+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE16_TYPE);+    return art_get_ref(art, (art_node_t *)new_node, CROARING_ART_NODE4_TYPE);+}++static inline void art_node16_replace(art_node16_t *node,+                                      art_key_chunk_t key_chunk,+                                      art_ref_t new_child) {+    for (uint8_t i = 0; i < node->count; ++i) {+        if (node->keys[i] == key_chunk) {+            node->children[i] = new_child;+            return;+        }+    }+}++static inline art_indexed_child_t art_node16_next_child(+    const art_node16_t *node, int index) {+    art_indexed_child_t indexed_child;+    index++;+    if (index >= node->count) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node16_prev_child(+    const art_node16_t *node, int index) {+    if (index > node->count) {+        index = node->count;+    }+    index--;+    art_indexed_child_t indexed_child;+    if (index < 0) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node16_child_at(const art_node16_t *node,+                                                      int index) {+    art_indexed_child_t indexed_child;+    if (index < 0 || index >= node->count) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = node->keys[index];+    return indexed_child;+}++static inline art_indexed_child_t art_node16_lower_bound(+    art_node16_t *node, art_key_chunk_t key_chunk) {+    art_indexed_child_t indexed_child;+    for (size_t i = 0; i < node->count; ++i) {+        if (node->keys[i] >= key_chunk) {+            indexed_child.index = i;+            indexed_child.child = node->children[i];+            indexed_child.key_chunk = node->keys[i];+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static bool art_node16_internal_validate(const art_t *art,+                                         const art_node16_t *node,+                                         art_internal_validate_t validator) {+    if (node->count <= 4) {+        return art_validate_fail(&validator, "Node16 has too few children");+    }+    if (node->count > 16) {+        return art_validate_fail(&validator, "Node16 has too many children");+    }+    validator.depth++;+    for (int i = 0; i < node->count; ++i) {+        if (i > 0) {+            if (node->keys[i - 1] >= node->keys[i]) {+                return art_validate_fail(+                    &validator, "Node16 keys are not strictly increasing");+            }+        }+        for (int j = i + 1; j < node->count; ++j) {+            if (node->children[i] == node->children[j]) {+                return art_validate_fail(&validator,+                                         "Node16 has duplicate children");+            }+        }+        validator.current_key[validator.depth - 1] = node->keys[i];+        if (!art_internal_validate_at(art, node->children[i], validator)) {+            return false;+        }+    }+    return true;+}++static art_node48_t *art_node48_create(art_t *art,+                                       const art_key_chunk_t prefix[],+                                       uint8_t prefix_size) {+    uint64_t index = art_allocate_index(art, CROARING_ART_NODE48_TYPE);+    art_node48_t *node =+        ((art_node48_t *)art->nodes[CROARING_ART_NODE48_TYPE]) + index;+    art_init_inner_node(&node->base, prefix, prefix_size);+    node->count = 0;+    node->available_children = CROARING_NODE48_AVAILABLE_CHILDREN_MASK;+    for (size_t i = 0; i < 256; ++i) {+        node->keys[i] = CROARING_ART_NODE48_EMPTY_VAL;+    }+    return node;+}++static inline art_ref_t art_node48_find_child(const art_node48_t *node,+                                              art_key_chunk_t key) {+    uint8_t val_idx = node->keys[key];+    if (val_idx != CROARING_ART_NODE48_EMPTY_VAL) {+        return node->children[val_idx];+    }+    return CROARING_ART_NULL_REF;+}++static art_ref_t art_node48_insert(art_t *art, art_node48_t *node,+                                   art_ref_t child, uint8_t key) {+    if (node->count < 48) {+        // node->available_children is only zero when the node is full (count ==+        // 48), we just checked count < 48+        uint8_t val_idx = roaring_trailing_zeroes(node->available_children);+        node->keys[key] = val_idx;+        node->children[val_idx] = child;+        node->count++;+        node->available_children &= ~(UINT64_C(1) << val_idx);+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE48_TYPE);+    }+    art_node256_t *new_node =+        art_node256_create(art, node->base.prefix, node->base.prefix_size);+    for (size_t i = 0; i < 256; ++i) {+        uint8_t val_idx = node->keys[i];+        if (val_idx != CROARING_ART_NODE48_EMPTY_VAL) {+            art_node256_insert(art, new_node, node->children[val_idx], i);+        }+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE48_TYPE);+    return art_node256_insert(art, new_node, child, key);+}++static inline art_ref_t art_node48_erase(art_t *art, art_node48_t *node,+                                         uint8_t key_chunk) {+    uint8_t val_idx = node->keys[key_chunk];+    if (val_idx == CROARING_ART_NODE48_EMPTY_VAL) {+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE48_TYPE);+    }+    node->keys[key_chunk] = CROARING_ART_NODE48_EMPTY_VAL;+    node->available_children |= UINT64_C(1) << val_idx;+    node->count--;+    if (node->count > 16) {+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE48_TYPE);+    }++    art_node16_t *new_node =+        art_node16_create(art, node->base.prefix, node->base.prefix_size);+    for (size_t i = 0; i < 256; ++i) {+        val_idx = node->keys[i];+        if (val_idx != CROARING_ART_NODE48_EMPTY_VAL) {+            art_node16_insert(art, new_node, node->children[val_idx], i);+        }+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE48_TYPE);+    return art_get_ref(art, (art_node_t *)new_node, CROARING_ART_NODE16_TYPE);+}++static inline void art_node48_replace(art_node48_t *node,+                                      art_key_chunk_t key_chunk,+                                      art_ref_t new_child) {+    uint8_t val_idx = node->keys[key_chunk];+    assert(val_idx != CROARING_ART_NODE48_EMPTY_VAL);+    node->children[val_idx] = new_child;+}++static inline art_indexed_child_t art_node48_next_child(+    const art_node48_t *node, int index) {+    art_indexed_child_t indexed_child;+    index++;+    for (size_t i = index; i < 256; ++i) {+        if (node->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+            indexed_child.index = i;+            indexed_child.child = node->children[node->keys[i]];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static inline art_indexed_child_t art_node48_prev_child(+    const art_node48_t *node, int index) {+    if (index > 256) {+        index = 256;+    }+    index--;+    art_indexed_child_t indexed_child;+    for (int i = index; i >= 0; --i) {+        if (node->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+            indexed_child.index = i;+            indexed_child.child = node->children[node->keys[i]];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static inline art_indexed_child_t art_node48_child_at(const art_node48_t *node,+                                                      int index) {+    art_indexed_child_t indexed_child;+    if (index < 0 || index >= 256) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[node->keys[index]];+    indexed_child.key_chunk = index;+    return indexed_child;+}++static inline art_indexed_child_t art_node48_lower_bound(+    art_node48_t *node, art_key_chunk_t key_chunk) {+    art_indexed_child_t indexed_child;+    for (size_t i = key_chunk; i < 256; ++i) {+        if (node->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+            indexed_child.index = i;+            indexed_child.child = node->children[node->keys[i]];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static bool art_node48_internal_validate(const art_t *art,+                                         const art_node48_t *node,+                                         art_internal_validate_t validator) {+    if (node->count <= 16) {+        return art_validate_fail(&validator, "Node48 has too few children");+    }+    if (node->count > 48) {+        return art_validate_fail(&validator, "Node48 has too many children");+    }+    uint64_t used_children = 0;+    for (int i = 0; i < 256; ++i) {+        uint8_t child_idx = node->keys[i];+        if (child_idx != CROARING_ART_NODE48_EMPTY_VAL) {+            if (used_children & (UINT64_C(1) << child_idx)) {+                return art_validate_fail(+                    &validator, "Node48 keys point to the same child index");+            }++            art_ref_t child = node->children[child_idx];+            if (child == CROARING_ART_NULL_REF) {+                return art_validate_fail(&validator, "Node48 has a NULL child");+            }+            used_children |= UINT64_C(1) << child_idx;+        }+    }+    uint64_t expected_used_children =+        (node->available_children) ^ CROARING_NODE48_AVAILABLE_CHILDREN_MASK;+    if (used_children != expected_used_children) {+        return art_validate_fail(+            &validator,+            "Node48 available_children does not match actual children");+    }+    while (used_children != 0) {+        uint8_t child_idx = roaring_trailing_zeroes(used_children);+        used_children &= used_children - 1;++        uint64_t other_children = used_children;+        while (other_children != 0) {+            uint8_t other_child_idx = roaring_trailing_zeroes(other_children);+            if (node->children[child_idx] == node->children[other_child_idx]) {+                return art_validate_fail(&validator,+                                         "Node48 has duplicate children");+            }+            other_children &= other_children - 1;+        }+    }++    validator.depth++;+    for (int i = 0; i < 256; ++i) {+        if (node->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+            validator.current_key[validator.depth - 1] = i;+            if (!art_internal_validate_at(art, node->children[node->keys[i]],+                                          validator)) {+                return false;+            }+        }+    }+    return true;+}++static art_node256_t *art_node256_create(art_t *art,+                                         const art_key_chunk_t prefix[],+                                         uint8_t prefix_size) {+    uint64_t index = art_allocate_index(art, CROARING_ART_NODE256_TYPE);+    art_node256_t *node =+        ((art_node256_t *)art->nodes[CROARING_ART_NODE256_TYPE]) + index;+    art_init_inner_node(&node->base, prefix, prefix_size);+    node->count = 0;+    for (size_t i = 0; i < 256; ++i) {+        node->children[i] = CROARING_ART_NULL_REF;+    }+    return node;+}++static inline art_ref_t art_node256_find_child(const art_node256_t *node,+                                               art_key_chunk_t key) {+    return node->children[key];+}++static art_ref_t art_node256_insert(art_t *art, art_node256_t *node,+                                    art_ref_t child, uint8_t key) {+    node->children[key] = child;+    node->count++;+    return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE256_TYPE);+}++static inline art_ref_t art_node256_erase(art_t *art, art_node256_t *node,+                                          uint8_t key_chunk) {+    node->children[key_chunk] = CROARING_ART_NULL_REF;+    node->count--;+    if (node->count > 48) {+        return art_get_ref(art, (art_node_t *)node, CROARING_ART_NODE256_TYPE);+    }++    art_node48_t *new_node =+        art_node48_create(art, node->base.prefix, node->base.prefix_size);+    for (size_t i = 0; i < 256; ++i) {+        if (node->children[i] != CROARING_ART_NULL_REF) {+            art_node48_insert(art, new_node, node->children[i], i);+        }+    }+    art_node_free(art, (art_node_t *)node, CROARING_ART_NODE256_TYPE);+    return art_get_ref(art, (art_node_t *)new_node, CROARING_ART_NODE48_TYPE);+}++static inline void art_node256_replace(art_node256_t *node,+                                       art_key_chunk_t key_chunk,+                                       art_ref_t new_child) {+    node->children[key_chunk] = new_child;+}++static inline art_indexed_child_t art_node256_next_child(+    const art_node256_t *node, int index) {+    art_indexed_child_t indexed_child;+    index++;+    for (size_t i = index; i < 256; ++i) {+        if (node->children[i] != CROARING_ART_NULL_REF) {+            indexed_child.index = i;+            indexed_child.child = node->children[i];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static inline art_indexed_child_t art_node256_prev_child(+    const art_node256_t *node, int index) {+    if (index > 256) {+        index = 256;+    }+    index--;+    art_indexed_child_t indexed_child;+    for (int i = index; i >= 0; --i) {+        if (node->children[i] != CROARING_ART_NULL_REF) {+            indexed_child.index = i;+            indexed_child.child = node->children[i];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static inline art_indexed_child_t art_node256_child_at(+    const art_node256_t *node, int index) {+    art_indexed_child_t indexed_child;+    if (index < 0 || index >= 256) {+        indexed_child.child = CROARING_ART_NULL_REF;+        return indexed_child;+    }+    indexed_child.index = index;+    indexed_child.child = node->children[index];+    indexed_child.key_chunk = index;+    return indexed_child;+}++static inline art_indexed_child_t art_node256_lower_bound(+    art_node256_t *node, art_key_chunk_t key_chunk) {+    art_indexed_child_t indexed_child;+    for (size_t i = key_chunk; i < 256; ++i) {+        if (node->children[i] != CROARING_ART_NULL_REF) {+            indexed_child.index = i;+            indexed_child.child = node->children[i];+            indexed_child.key_chunk = i;+            return indexed_child;+        }+    }+    indexed_child.child = CROARING_ART_NULL_REF;+    return indexed_child;+}++static bool art_node256_internal_validate(const art_t *art,+                                          const art_node256_t *node,+                                          art_internal_validate_t validator) {+    if (node->count <= 48) {+        return art_validate_fail(&validator, "Node256 has too few children");+    }+    if (node->count > 256) {+        return art_validate_fail(&validator, "Node256 has too many children");+    }+    validator.depth++;+    int actual_count = 0;+    for (int i = 0; i < 256; ++i) {+        if (node->children[i] != CROARING_ART_NULL_REF) {+            actual_count++;++            for (int j = i + 1; j < 256; ++j) {+                if (node->children[i] == node->children[j]) {+                    return art_validate_fail(&validator,+                                             "Node256 has duplicate children");+                }+            }++            validator.current_key[validator.depth - 1] = i;+            if (!art_internal_validate_at(art, node->children[i], validator)) {+                return false;+            }+        }+    }+    if (actual_count != node->count) {+        return art_validate_fail(+            &validator, "Node256 count does not match actual children");+    }+    return true;+}++// Finds the child with the given key chunk in the inner node, returns NULL if+// no such child is found.+static art_ref_t art_find_child(const art_inner_node_t *node,+                                art_typecode_t typecode,+                                art_key_chunk_t key_chunk) {+    switch (typecode) {+        case CROARING_ART_NODE4_TYPE:+            return art_node4_find_child((art_node4_t *)node, key_chunk);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_find_child((art_node16_t *)node, key_chunk);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_find_child((art_node48_t *)node, key_chunk);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_find_child((art_node256_t *)node, key_chunk);+        default:+            assert(false);+            return CROARING_ART_NULL_REF;+    }+}++// Replaces the child with the given key chunk in the inner node.+static void art_replace(art_inner_node_t *node, art_typecode_t typecode,+                        art_key_chunk_t key_chunk, art_ref_t new_child) {+    switch (typecode) {+        case CROARING_ART_NODE4_TYPE:+            art_node4_replace((art_node4_t *)node, key_chunk, new_child);+            break;+        case CROARING_ART_NODE16_TYPE:+            art_node16_replace((art_node16_t *)node, key_chunk, new_child);+            break;+        case CROARING_ART_NODE48_TYPE:+            art_node48_replace((art_node48_t *)node, key_chunk, new_child);+            break;+        case CROARING_ART_NODE256_TYPE:+            art_node256_replace((art_node256_t *)node, key_chunk, new_child);+            break;+        default:+            assert(false);+    }+}++// Erases the child with the given key chunk from the inner node, returns the+// updated node (the same as the initial node if it was not shrunk).+static art_ref_t art_node_erase(art_t *art, art_inner_node_t *node,+                                art_typecode_t typecode,+                                art_key_chunk_t key_chunk) {+    switch (typecode) {+        case CROARING_ART_NODE4_TYPE:+            return art_node4_erase(art, (art_node4_t *)node, key_chunk);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_erase(art, (art_node16_t *)node, key_chunk);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_erase(art, (art_node48_t *)node, key_chunk);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_erase(art, (art_node256_t *)node, key_chunk);+        default:+            assert(false);+            return CROARING_ART_NULL_REF;+    }+}++// Inserts the leaf with the given key chunk in the inner node, returns a+// pointer to the (possibly expanded) node.+static art_ref_t art_node_insert_leaf(art_t *art, art_inner_node_t *node,+                                      art_typecode_t typecode,+                                      art_key_chunk_t key_chunk,+                                      art_ref_t leaf) {+    switch (typecode) {+        case CROARING_ART_NODE4_TYPE:+            return art_node4_insert(art, (art_node4_t *)node, leaf, key_chunk);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_insert(art, (art_node16_t *)node, leaf,+                                     key_chunk);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_insert(art, (art_node48_t *)node, leaf,+                                     key_chunk);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_insert(art, (art_node256_t *)node, leaf,+                                      key_chunk);+        default:+            assert(false);+            return CROARING_ART_NULL_REF;+    }+}++static uint64_t art_node_get_next_free(const art_t *art, art_ref_t ref) {+    art_node_t *node = art_deref(art, ref);+    art_typecode_t typecode = art_ref_typecode(ref);+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return ((art_leaf_t *)node)->next_free;+        case CROARING_ART_NODE4_TYPE:+            return ((art_node4_t *)node)->next_free;+        case CROARING_ART_NODE16_TYPE:+            return ((art_node16_t *)node)->next_free;+        case CROARING_ART_NODE48_TYPE:+            return ((art_node48_t *)node)->next_free;+        case CROARING_ART_NODE256_TYPE:+            return ((art_node256_t *)node)->next_free;+        default:+            assert(false);+            return 0;+    }+}++static void art_node_set_next_free(art_node_t *node, art_typecode_t typecode,+                                   uint64_t next_free) {+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            ((art_leaf_t *)node)->next_free = next_free;+            break;+        case CROARING_ART_NODE4_TYPE:+            ((art_node4_t *)node)->next_free = next_free;+            break;+        case CROARING_ART_NODE16_TYPE:+            ((art_node16_t *)node)->next_free = next_free;+            break;+        case CROARING_ART_NODE48_TYPE:+            ((art_node48_t *)node)->next_free = next_free;+            break;+        case CROARING_ART_NODE256_TYPE:+            ((art_node256_t *)node)->next_free = next_free;+            break;+        default:+            assert(false);+    }+}++// Marks the node as unoccopied and frees its index.+static void art_node_free(art_t *art, art_node_t *node,+                          art_typecode_t typecode) {+    uint64_t index = art_get_index(art, node, typecode);+    uint64_t next_free = art->first_free[typecode];+    art_node_set_next_free(node, typecode, next_free);+    art->first_free[typecode] = index;+}++// Returns the next child in key order, or NULL if called on a leaf.+// Provided index may be in the range [-1, 255].+static art_indexed_child_t art_node_next_child(const art_node_t *node,+                                               art_typecode_t typecode,+                                               int index) {+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return (art_indexed_child_t){+                .child = CROARING_ART_NULL_REF,+                .index = 0,+                .key_chunk = 0,+            };+        case CROARING_ART_NODE4_TYPE:+            return art_node4_next_child((art_node4_t *)node, index);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_next_child((art_node16_t *)node, index);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_next_child((art_node48_t *)node, index);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_next_child((art_node256_t *)node, index);+        default:+            assert(false);+            return (art_indexed_child_t){0, 0, 0};+    }+}++// Returns the previous child in key order, or NULL if called on a leaf.+// Provided index may be in the range [0, 256].+static art_indexed_child_t art_node_prev_child(const art_node_t *node,+                                               art_typecode_t typecode,+                                               int index) {+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return (art_indexed_child_t){+                .child = CROARING_ART_NULL_REF,+                .index = 0,+                .key_chunk = 0,+            };+        case CROARING_ART_NODE4_TYPE:+            return art_node4_prev_child((art_node4_t *)node, index);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_prev_child((art_node16_t *)node, index);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_prev_child((art_node48_t *)node, index);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_prev_child((art_node256_t *)node, index);+        default:+            assert(false);+            return (art_indexed_child_t){0, 0, 0};+    }+}++// Returns the child found at the provided index, or NULL if called on a+// leaf. Provided index is only valid if returned by+// art_node_(next|prev)_child.+static art_indexed_child_t art_node_child_at(const art_node_t *node,+                                             art_typecode_t typecode,+                                             int index) {+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return (art_indexed_child_t){+                .child = CROARING_ART_NULL_REF,+                .index = 0,+                .key_chunk = 0,+            };+        case CROARING_ART_NODE4_TYPE:+            return art_node4_child_at((art_node4_t *)node, index);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_child_at((art_node16_t *)node, index);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_child_at((art_node48_t *)node, index);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_child_at((art_node256_t *)node, index);+        default:+            assert(false);+            return (art_indexed_child_t){0, 0, 0};+    }+}++// Returns the child with the smallest key equal to or greater than the+// given key chunk, NULL if called on a leaf or no such child was found.+static art_indexed_child_t art_node_lower_bound(const art_node_t *node,+                                                art_typecode_t typecode,+                                                art_key_chunk_t key_chunk) {+    switch (typecode) {+        case CROARING_ART_LEAF_TYPE:+            return (art_indexed_child_t){+                .child = CROARING_ART_NULL_REF,+                .index = 0,+                .key_chunk = 0,+            };+        case CROARING_ART_NODE4_TYPE:+            return art_node4_lower_bound((art_node4_t *)node, key_chunk);+        case CROARING_ART_NODE16_TYPE:+            return art_node16_lower_bound((art_node16_t *)node, key_chunk);+        case CROARING_ART_NODE48_TYPE:+            return art_node48_lower_bound((art_node48_t *)node, key_chunk);+        case CROARING_ART_NODE256_TYPE:+            return art_node256_lower_bound((art_node256_t *)node, key_chunk);+        default:+            assert(false);+            return (art_indexed_child_t){0, 0, 0};+    }+}++// ====================== End of node-specific functions ======================++// Compares the given ranges of two keys, returns their relative order:+// * Key range 1 <  key range 2: a negative value+// * Key range 1 == key range 2: 0+// * Key range 1 >  key range 2: a positive value+static inline int art_compare_prefix(const art_key_chunk_t key1[],+                                     uint8_t key1_from,+                                     const art_key_chunk_t key2[],+                                     uint8_t key2_from, uint8_t length) {+    return memcmp(key1 + key1_from, key2 + key2_from, length);+}++// Compares two keys in full, see art_compare_prefix.+int art_compare_keys(const art_key_chunk_t key1[],+                     const art_key_chunk_t key2[]) {+    return art_compare_prefix(key1, 0, key2, 0, ART_KEY_BYTES);+}++// Returns the length of the common prefix between two key ranges.+static uint8_t art_common_prefix(const art_key_chunk_t key1[],+                                 uint8_t key1_from, uint8_t key1_to,+                                 const art_key_chunk_t key2[],+                                 uint8_t key2_from, uint8_t key2_to) {+    uint8_t min_len = key1_to - key1_from;+    uint8_t key2_len = key2_to - key2_from;+    if (key2_len < min_len) {+        min_len = key2_len;+    }+    uint8_t offset = 0;+    for (; offset < min_len; ++offset) {+        if (key1[key1_from + offset] != key2[key2_from + offset]) {+            return offset;+        }+    }+    return offset;+}++/**+ * Grows the array of nodes of the given typecode to `new_capacity` elements,+ * linking the new elements at the end of the free list. Does nothing if+ * `new_capacity` is not larger than the current capacity. Invalidates pointers+ * into the array obtained by `art_deref`.+ */+static void art_grow_to(art_t *art, art_typecode_t typecode,+                        uint64_t new_capacity) {+    uint64_t capacity = art->capacities[typecode];+    if (new_capacity <= capacity) {+        return;+    }+    art->capacities[typecode] = new_capacity;+    art->nodes[typecode] = roaring_realloc(+        art->nodes[typecode], new_capacity * ART_NODE_SIZES[typecode]);+    uint64_t increase = new_capacity - capacity;+    memset(art_get_node(art, capacity, typecode), 0,+           increase * ART_NODE_SIZES[typecode]);+    for (uint64_t i = capacity; i < new_capacity; ++i) {+        art_node_set_next_free(art_get_node(art, i, typecode), typecode, i + 1);+    }+}++/**+ * Extends the array of nodes of the given typecode. Invalidates pointers into+ * the array obtained by `art_deref`.+ */+static void art_extend(art_t *art, art_typecode_t typecode) {+    uint64_t size = art->first_free[typecode];+    uint64_t capacity = art->capacities[typecode];+    if (size < capacity) {+        return;+    }+    uint64_t new_capacity;+    if (capacity == 0) {+        new_capacity = 2;+    } else if (capacity < 1024) {+        new_capacity = 2 * capacity;+    } else {+        new_capacity = 5 * capacity / 4;+    }+    art_grow_to(art, typecode, new_capacity);+}++/**+ * Returns the next free index for the given typecode, may be equal to the+ * capacity of the array.+ */+static uint64_t art_next_free(const art_t *art, art_typecode_t typecode) {+    uint64_t index = art->first_free[typecode];+    return art_node_get_next_free(art, art_to_ref(index, typecode));+}++/**+ * Marks an index for the given typecode as used, expanding the relevant node+ * array if necessary.+ */+static uint64_t art_allocate_index(art_t *art, art_typecode_t typecode) {+    uint64_t first_free = art->first_free[typecode];+    if (first_free == art->capacities[typecode]) {+        art_extend(art, typecode);+        art->first_free[typecode]++;+        return first_free;+    }+    art->first_free[typecode] = art_next_free(art, typecode);+    return first_free;+}++// Returns a pointer to the rootmost node where the value was inserted, may+// not be equal to `node`.+static art_ref_t art_insert_at(art_t *art, art_ref_t ref,+                               const art_key_chunk_t key[], uint8_t depth,+                               art_ref_t new_leaf) {+    if (art_is_leaf(ref)) {+        art_leaf_t *leaf = (art_leaf_t *)art_deref(art, ref);+        uint8_t common_prefix = art_common_prefix(+            leaf->key, depth, ART_KEY_BYTES, key, depth, ART_KEY_BYTES);++        // Previously this was a leaf, create an inner node instead and add+        // both the existing and new leaf to it.+        art_node_t *new_node =+            (art_node_t *)art_node4_create(art, key + depth, common_prefix);++        art_ref_t new_ref = art_node_insert_leaf(+            art, (art_inner_node_t *)new_node, CROARING_ART_NODE4_TYPE,+            leaf->key[depth + common_prefix], ref);+        new_ref = art_node_insert_leaf(art, (art_inner_node_t *)new_node,+                                       CROARING_ART_NODE4_TYPE,+                                       key[depth + common_prefix], new_leaf);++        // The new inner node is now the rootmost node.+        return new_ref;+    }+    art_inner_node_t *inner_node = (art_inner_node_t *)art_deref(art, ref);+    // Not a leaf: inner node+    uint8_t common_prefix =+        art_common_prefix(inner_node->prefix, 0, inner_node->prefix_size, key,+                          depth, ART_KEY_BYTES);+    if (common_prefix != inner_node->prefix_size) {+        // Partial prefix match. Create a new internal node to hold the common+        // prefix.+        // We create a copy of the node's prefix as the creation of a new+        // node may invalidate the prefix pointer.+        art_key_chunk_t *prefix_copy = (art_key_chunk_t *)roaring_malloc(+            common_prefix * sizeof(art_key_chunk_t));+        memcpy(prefix_copy, inner_node->prefix,+               common_prefix * sizeof(art_key_chunk_t));+        art_node4_t *node4 = art_node4_create(art, prefix_copy, common_prefix);+        roaring_free(prefix_copy);++        // Deref as a new node was created.+        inner_node = (art_inner_node_t *)art_deref(art, ref);++        // Make the existing internal node a child of the new internal node.+        art_node4_insert(art, node4, ref, inner_node->prefix[common_prefix]);++        // Deref again as a new node was created.+        inner_node = (art_inner_node_t *)art_deref(art, ref);++        // Correct the prefix of the moved internal node, trimming off the+        // chunk inserted into the new internal node.+        inner_node->prefix_size = inner_node->prefix_size - common_prefix - 1;+        if (inner_node->prefix_size > 0) {+            // Move the remaining prefix to the correct position.+            memmove(inner_node->prefix, inner_node->prefix + common_prefix + 1,+                    inner_node->prefix_size);+        }++        // Insert the value in the new internal node.+        return art_node_insert_leaf(art, (art_inner_node_t *)node4,+                                    CROARING_ART_NODE4_TYPE,+                                    key[common_prefix + depth], new_leaf);+    }+    // Prefix matches entirely or node has no prefix. Look for an existing+    // child.+    art_key_chunk_t key_chunk = key[depth + common_prefix];+    art_ref_t child =+        art_find_child(inner_node, art_ref_typecode(ref), key_chunk);+    if (child != CROARING_ART_NULL_REF) {+        art_ref_t new_child =+            art_insert_at(art, child, key, depth + common_prefix + 1, new_leaf);+        if (new_child != child) {+            // Deref again as a new node may have been created.+            inner_node = (art_inner_node_t *)art_deref(art, ref);+            // Node type changed.+            art_replace(inner_node, art_ref_typecode(ref), key_chunk,+                        new_child);+        }+        return ref;+    }+    return art_node_insert_leaf(art, inner_node, art_ref_typecode(ref),+                                key_chunk, new_leaf);+}++// Erase helper struct.+typedef struct art_erase_result_s {+    // The rootmost node where the value was erased, may not be equal to+    // the original node. If no value was removed, this is+    // CROARING_ART_NULL_REF.+    art_ref_t rootmost_node;++    // True if a value was erased.+    bool erased;++    // Value removed, if any.+    art_val_t value_erased;+} art_erase_result_t;++// Searches for the given key starting at `node`, erases it if found.+static art_erase_result_t art_erase_at(art_t *art, art_ref_t ref,+                                       const art_key_chunk_t *key,+                                       uint8_t depth) {+    art_erase_result_t result;+    result.rootmost_node = CROARING_ART_NULL_REF;+    result.erased = false;++    if (art_is_leaf(ref)) {+        art_leaf_t *leaf = (art_leaf_t *)art_deref(art, ref);+        uint8_t common_prefix = art_common_prefix(leaf->key, 0, ART_KEY_BYTES,+                                                  key, 0, ART_KEY_BYTES);+        if (common_prefix != ART_KEY_BYTES) {+            // Leaf key mismatch.+            return result;+        }+        result.erased = true;+        result.value_erased = leaf->val;+        art_node_free(art, (art_node_t *)leaf, CROARING_ART_LEAF_TYPE);+        return result;+    }+    art_inner_node_t *inner_node = (art_inner_node_t *)art_deref(art, ref);+    uint8_t common_prefix =+        art_common_prefix(inner_node->prefix, 0, inner_node->prefix_size, key,+                          depth, ART_KEY_BYTES);+    if (common_prefix != inner_node->prefix_size) {+        // Prefix mismatch.+        return result;+    }+    art_key_chunk_t key_chunk = key[depth + common_prefix];+    art_ref_t child =+        art_find_child(inner_node, art_ref_typecode(ref), key_chunk);+    if (child == CROARING_ART_NULL_REF) {+        // No child with key chunk.+        return result;+    }+    // Try to erase the key further down. Skip the key chunk associated with+    // the child in the node.+    art_erase_result_t child_result =+        art_erase_at(art, child, key, depth + common_prefix + 1);+    if (!child_result.erased) {+        return result;+    }+    result.erased = true;+    result.value_erased = child_result.value_erased;+    result.rootmost_node = ref;++    // Deref again as nodes may have changed location.+    inner_node = (art_inner_node_t *)art_deref(art, ref);+    if (child_result.rootmost_node == CROARING_ART_NULL_REF) {+        // Child node was fully erased, erase it from this node's children.+        result.rootmost_node =+            art_node_erase(art, inner_node, art_ref_typecode(ref), key_chunk);+    } else if (child_result.rootmost_node != child) {+        // Child node was not fully erased, update the pointer to it in this+        // node.+        art_replace(inner_node, art_ref_typecode(ref), key_chunk,+                    child_result.rootmost_node);+    }+    return result;+}++// Searches for the given key starting at `node`, returns NULL if the key+// was not found.+static art_val_t *art_find_at(const art_t *art, art_ref_t ref,+                              const art_key_chunk_t *key, uint8_t depth) {+#if !CROARING_IS_BIG_ENDIAN+    // A 6-byte memcpy into a word stalls on store forwarding. The+    // prefix-size check below keeps the shift and the child-byte read+    // inside the key.+    uint32_t key_lo;+    uint16_t key_hi;+    memcpy(&key_lo, key, sizeof(key_lo));+    memcpy(&key_hi, key + sizeof(key_lo), sizeof(key_hi));+    const uint64_t key_word = key_lo | ((uint64_t)key_hi << 32);+#endif+    while (!art_is_leaf(ref)) {+        const art_inner_node_t *inner_node =+            (const art_inner_node_t *)art_deref(art, ref);+        uint8_t prefix_size = inner_node->prefix_size;+        if (prefix_size >= ART_KEY_BYTES - depth) {+            return NULL;+        }+#if !CROARING_IS_BIG_ENDIAN+        // In bounds: inner nodes are larger than 8 bytes.+        uint64_t header;+        memcpy(&header, inner_node, sizeof(header));+        uint64_t mask = (UINT64_C(1) << (8 * prefix_size)) - 1;+        if (((header >> 8) ^ (key_word >> (8 * depth))) & mask) {+            return NULL;+        }+#else+        for (uint8_t i = 0; i < prefix_size; ++i) {+            if (inner_node->prefix[i] != key[depth + i]) {+                return NULL;+            }+        }+#endif+        art_key_chunk_t key_chunk = key[depth + prefix_size];+        art_ref_t child;+        switch (art_ref_typecode(ref)) {+            case CROARING_ART_NODE4_TYPE:+                child = art_node4_find_child((const art_node4_t *)inner_node,+                                             key_chunk);+                break;+            case CROARING_ART_NODE16_TYPE:+                child = art_node16_find_child((const art_node16_t *)inner_node,+                                              key_chunk);+                break;+            case CROARING_ART_NODE48_TYPE:+                child = art_node48_find_child((const art_node48_t *)inner_node,+                                              key_chunk);+                break;+            case CROARING_ART_NODE256_TYPE:+                child = art_node256_find_child(+                    (const art_node256_t *)inner_node, key_chunk);+                break;+            default:+                assert(false);+                return NULL;+        }+        if (child == CROARING_ART_NULL_REF) {+            return NULL;+        }+        ref = child;+        // Include both the prefix and the child key chunk in the depth.+        depth += prefix_size + 1;+    }+    art_leaf_t *leaf = (art_leaf_t *)art_deref(art, ref);+    if (depth >= ART_KEY_BYTES || memcmp(leaf->key, key, ART_KEY_BYTES) == 0) {+        return &leaf->val;+    }+    return NULL;+}++static void art_node_print_type(art_ref_t ref) {+    switch (art_ref_typecode(ref)) {+        case CROARING_ART_LEAF_TYPE:+            printf("Leaf");+            return;+        case CROARING_ART_NODE4_TYPE:+            printf("Node4");+            return;+        case CROARING_ART_NODE16_TYPE:+            printf("Node16");+            return;+        case CROARING_ART_NODE48_TYPE:+            printf("Node48");+            return;+        case CROARING_ART_NODE256_TYPE:+            printf("Node256");+            return;+        default:+            assert(false);+            return;+    }+}++static void art_node_printf(const art_t *art, art_ref_t ref, uint8_t depth) {+    if (art_is_leaf(ref)) {+        printf("{ type: Leaf, key: ");+        art_leaf_t *leaf = (art_leaf_t *)art_deref(art, ref);+        for (size_t i = 0; i < ART_KEY_BYTES; ++i) {+            printf("%02x", leaf->key[i]);+        }+        printf(" }\n");+        return;+    }+    printf("{\n");+    depth++;++    printf("%*s", depth, "");+    printf("type: ");+    art_node_print_type(ref);+    printf("\n");++    art_inner_node_t *inner_node = (art_inner_node_t *)art_deref(art, ref);+    printf("%*s", depth, "");+    printf("prefix_size: %d\n", inner_node->prefix_size);++    printf("%*s", depth, "");+    printf("prefix: ");+    for (uint8_t i = 0; i < inner_node->prefix_size; ++i) {+        printf("%02x", inner_node->prefix[i]);+    }+    printf("\n");++    switch (art_ref_typecode(ref)) {+        case CROARING_ART_NODE4_TYPE: {+            art_node4_t *node4 = (art_node4_t *)inner_node;+            for (uint8_t i = 0; i < node4->count; ++i) {+                printf("%*s", depth, "");+                printf("key: %02x ", node4->keys[i]);+                art_node_printf(art, node4->children[i], depth);+            }+        } break;+        case CROARING_ART_NODE16_TYPE: {+            art_node16_t *node16 = (art_node16_t *)inner_node;+            for (uint8_t i = 0; i < node16->count; ++i) {+                printf("%*s", depth, "");+                printf("key: %02x ", node16->keys[i]);+                art_node_printf(art, node16->children[i], depth);+            }+        } break;+        case CROARING_ART_NODE48_TYPE: {+            art_node48_t *node48 = (art_node48_t *)inner_node;+            for (uint16_t i = 0; i < 256; ++i) {+                if (node48->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+                    printf("%*s", depth, "");+                    printf("key: %02x ", i);+                    printf("child: %02x ", node48->keys[i]);+                    art_node_printf(art, node48->children[node48->keys[i]],+                                    depth);+                }+            }+        } break;+        case CROARING_ART_NODE256_TYPE: {+            art_node256_t *node256 = (art_node256_t *)inner_node;+            for (uint16_t i = 0; i < 256; ++i) {+                if (node256->children[i] != CROARING_ART_NULL_REF) {+                    printf("%*s", depth, "");+                    printf("key: %02x ", i);+                    art_node_printf(art, node256->children[i], depth);+                }+            }+        } break;+        default:+            assert(false);+            break;+    }+    depth--;+    printf("%*s", depth, "");+    printf("}\n");+}++/**+ * Shrinks all node arrays to `first_free`. Assumes all indices after+ * `first_free` are unused.+ */+static size_t art_shrink_node_arrays(art_t *art) {+    size_t freed = 0;+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        uint64_t new_capacity = art->first_free[t];+        uint64_t capacity = art->capacities[t];+        if (new_capacity < capacity) {+            bool shrunk;+            art->nodes[t] = croaring_realloc_shrink(+                art->nodes[t], new_capacity * ART_NODE_SIZES[t], &shrunk);+            if (shrunk) {+                freed += (capacity - new_capacity) * ART_NODE_SIZES[t];+            }+            // Even when the array kept its size: the slots past first_free+            // are not a free list.+            art->capacities[t] = new_capacity;+        }+    }+    return freed;+}++// Marks the free slot of a shrunken node array.+#define CROARING_ART_SHRINK_FREE UINT64_MAX++/**+ * Returns `ref` with its index replaced by the index it gets after+ * compaction: `maps[t][i]` is the new index of the node of type `t` at index+ * `i`.+ */+static inline art_ref_t art_shrink_remap(uint64_t *const *maps, art_ref_t ref) {+    art_typecode_t typecode = art_ref_typecode(ref);+    return art_to_ref(maps[typecode][art_ref_index(ref)], typecode);+}++/**+ * Rewrites the child references of an inner node with `maps`.+ */+static void art_shrink_remap_children(art_node_t *node, art_typecode_t typecode,+                                      uint64_t *const *maps) {+    switch (typecode) {+        case CROARING_ART_NODE4_TYPE: {+            art_node4_t *node4 = (art_node4_t *)node;+            for (uint8_t i = 0; i < node4->count; ++i) {+                node4->children[i] = art_shrink_remap(maps, node4->children[i]);+            }+        } break;+        case CROARING_ART_NODE16_TYPE: {+            art_node16_t *node16 = (art_node16_t *)node;+            for (uint8_t i = 0; i < node16->count; ++i) {+                node16->children[i] =+                    art_shrink_remap(maps, node16->children[i]);+            }+        } break;+        case CROARING_ART_NODE48_TYPE: {+            art_node48_t *node48 = (art_node48_t *)node;+            for (int i = 0; i < 256; ++i) {+                if (node48->keys[i] != CROARING_ART_NODE48_EMPTY_VAL) {+                    uint8_t idx = node48->keys[i];+                    node48->children[idx] =+                        art_shrink_remap(maps, node48->children[idx]);+                }+            }+        } break;+        case CROARING_ART_NODE256_TYPE: {+            art_node256_t *node256 = (art_node256_t *)node;+            for (int i = 0; i < 256; ++i) {+                if (node256->children[i] != CROARING_ART_NULL_REF) {+                    node256->children[i] =+                        art_shrink_remap(maps, node256->children[i]);+                }+            }+        } break;+        default:+            assert(false);+            break;+    }+}++void art_init_cleared(art_t *art) {+    art->root = CROARING_ART_NULL_REF;+    memset(art->first_free, 0, sizeof(art->first_free));+    memset(art->capacities, 0, sizeof(art->capacities));+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        art->nodes[t] = NULL;+    }+}++size_t art_shrink_to_fit(art_t *art) {+    if (art_is_shrunken(art)) {+        return 0;+    }+    // Compacts each node array in time linear in its capacity: every live node+    // moves to its rank among the live nodes of its type, which keeps their+    // relative order, and every reference is rewritten to match.+    uint64_t *maps[CROARING_ART_MAX_TYPE + 1] = {NULL};+    uint64_t live[CROARING_ART_MAX_TYPE + 1] = {0};+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        uint64_t capacity = art->capacities[t];+        if (capacity == 0) {+            continue;+        }+        maps[t] = (uint64_t *)roaring_calloc(capacity, sizeof(uint64_t));+        if (maps[t] == NULL) {+            for (art_typecode_t u = CROARING_ART_MIN_TYPE; u < t; ++u) {+                roaring_free(maps[u]);+            }+            return 0;+        }+        for (uint64_t i = art->first_free[t]; i < capacity;+             i = art_node_get_next_free(art, art_to_ref(i, t))) {+            maps[t][i] = CROARING_ART_SHRINK_FREE;+        }+        uint64_t count = 0;+        for (uint64_t i = 0; i < capacity; ++i) {+            if (maps[t][i] != CROARING_ART_SHRINK_FREE) {+                maps[t][i] = count++;+            }+        }+        live[t] = count;+    }+    // Rewrite the references while the nodes are still in place.+    for (art_typecode_t t = CROARING_ART_NODE4_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        for (uint64_t i = 0; i < art->capacities[t]; ++i) {+            if (maps[t][i] != CROARING_ART_SHRINK_FREE) {+                art_shrink_remap_children(art_get_node(art, i, t), t,+                                          (uint64_t *const *)maps);+            }+        }+    }+    if (art->root != CROARING_ART_NULL_REF) {+        art->root = art_shrink_remap((uint64_t *const *)maps, art->root);+    }+    // Move the nodes. Going in ascending order, a node only moves to a slot+    // that is free or already moved out of.+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        for (uint64_t i = 0; i < art->capacities[t]; ++i) {+            uint64_t to = maps[t][i];+            if (to != CROARING_ART_SHRINK_FREE && to != i) {+                memcpy(art_get_node(art, to, t), art_get_node(art, i, t),+                       ART_NODE_SIZES[t]);+            }+        }+        art->first_free[t] = live[t];+        roaring_free(maps[t]);+    }+    return art_shrink_node_arrays(art);+}++#undef CROARING_ART_SHRINK_FREE++bool art_is_shrunken(const art_t *art) {+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        if (art->first_free[t] != art->capacities[t]) {+            return false;+        }+    }+    return true;+}++art_val_t *art_insert(art_t *art, const art_key_chunk_t *key, art_val_t val) {+    art_ref_t leaf = art_leaf_create(art, key, val);+    if (art->root == CROARING_ART_NULL_REF) {+        art->root = leaf;+        return &((art_leaf_t *)art_deref(art, leaf))->val;+    }+    art->root = art_insert_at(art, art->root, key, 0, leaf);+    return &((art_leaf_t *)art_deref(art, leaf))->val;+}++bool art_erase(art_t *art, const art_key_chunk_t *key, art_val_t *erased_val) {+    art_val_t erased_val_local;+    if (erased_val == NULL) {+        erased_val = &erased_val_local;+    }+    if (art->root == CROARING_ART_NULL_REF) {+        return false;+    }+    art_erase_result_t result = art_erase_at(art, art->root, key, 0);+    if (!result.erased) {+        return false;+    }+    art->root = result.rootmost_node;+    *erased_val = result.value_erased;+    return true;+}++art_val_t *art_find(const art_t *art, const art_key_chunk_t *key) {+    if (art->root == CROARING_ART_NULL_REF) {+        return NULL;+    }+    return art_find_at(art, art->root, key, 0);+}++bool art_is_empty(const art_t *art) {+    return art->root == CROARING_ART_NULL_REF;+}++void art_free(art_t *art) {+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        roaring_free(art->nodes[t]);+    }+}++void art_reserve(art_t *art, uint64_t num_leaves) {+    art_grow_to(art, CROARING_ART_LEAF_TYPE, num_leaves);+}++uint64_t art_num_leaves(const art_t *art) {+    uint64_t num_free = 0;+    for (uint64_t i = art->first_free[CROARING_ART_LEAF_TYPE];+         i < art->capacities[CROARING_ART_LEAF_TYPE];+         i = art_node_get_next_free(art,+                                    art_to_ref(i, CROARING_ART_LEAF_TYPE))) {+        num_free++;+    }+    return art->capacities[CROARING_ART_LEAF_TYPE] - num_free;+}++void art_printf(const art_t *art) {+    if (art->root == CROARING_ART_NULL_REF) {+        return;+    }+    art_node_printf(art, art->root, 0);+}++// Returns a reference to the current node that the iterator is positioned+// at.+static inline art_ref_t art_iterator_ref(art_iterator_t *iterator) {+    return iterator->frames[iterator->frame].ref;+}++// Returns the current node that the iterator is positioned at.+static inline art_node_t *art_iterator_node(art_iterator_t *iterator) {+    return art_deref(iterator->art, art_iterator_ref(iterator));+}++// Sets the iterator key and value to the leaf's key and value. Always+// returns true for convenience.+static inline bool art_iterator_valid_loc(art_iterator_t *iterator,+                                          art_ref_t leaf_ref) {+    iterator->frames[iterator->frame].ref = leaf_ref;+    iterator->frames[iterator->frame].index_in_node = 0;+    art_leaf_t *leaf = (art_leaf_t *)art_deref(iterator->art, leaf_ref);+    memcpy(iterator->key, leaf->key, ART_KEY_BYTES);+    iterator->value = &leaf->val;+    return true;+}++// Invalidates the iterator key and value. Always returns false for+// convenience.+static inline bool art_iterator_invalid_loc(art_iterator_t *iterator) {+    memset(iterator->key, 0, ART_KEY_BYTES);+    iterator->value = NULL;+    return false;+}++// Moves the iterator one level down in the tree, given a node at the+// current level and the index of the child that we're going down to.+//+// Note: does not set the index at the new level.+static void art_iterator_down(art_iterator_t *iterator, art_ref_t ref,+                              uint8_t index_in_node) {+    iterator->frames[iterator->frame].ref = ref;+    iterator->frames[iterator->frame].index_in_node = index_in_node;+    iterator->frame++;+    art_inner_node_t *node = (art_inner_node_t *)art_deref(iterator->art, ref);+    art_indexed_child_t indexed_child = art_node_child_at(+        (art_node_t *)node, art_ref_typecode(ref), index_in_node);+    assert(indexed_child.child != CROARING_ART_NULL_REF);+    iterator->frames[iterator->frame].ref = indexed_child.child;+    iterator->depth += node->prefix_size + 1;+}++// Moves the iterator to the next/previous child of the current node.+// Returns the child moved to, or NULL if there is no neighboring child.+static art_ref_t art_iterator_neighbor_child(art_iterator_t *iterator,+                                             bool forward) {+    art_iterator_frame_t frame = iterator->frames[iterator->frame];+    art_node_t *node = art_deref(iterator->art, frame.ref);+    art_indexed_child_t indexed_child;+    if (forward) {+        indexed_child = art_node_next_child(node, art_ref_typecode(frame.ref),+                                            frame.index_in_node);+    } else {+        indexed_child = art_node_prev_child(node, art_ref_typecode(frame.ref),+                                            frame.index_in_node);+    }+    if (indexed_child.child != CROARING_ART_NULL_REF) {+        art_iterator_down(iterator, frame.ref, indexed_child.index);+    }+    return indexed_child.child;+}++// Moves the iterator one level up in the tree, returns false if not+// possible.+static bool art_iterator_up(art_iterator_t *iterator) {+    if (iterator->frame == 0) {+        return false;+    }+    iterator->frame--;+    // We went up, so we are at an inner node.+    iterator->depth -=+        ((art_inner_node_t *)art_iterator_node(iterator))->prefix_size + 1;+    return true;+}++// Moves the iterator one level, followed by a move to the next / previous+// leaf. Sets the status of the iterator.+static bool art_iterator_up_and_move(art_iterator_t *iterator, bool forward) {+    if (!art_iterator_up(iterator)) {+        // We're at the root.+        return art_iterator_invalid_loc(iterator);+    }+    return art_iterator_move(iterator, forward);+}++// Initializes the iterator at the first / last leaf of the given node.+// Returns true for convenience.+static bool art_node_init_iterator(art_ref_t ref, art_iterator_t *iterator,+                                   bool first) {+    while (!art_is_leaf(ref)) {+        art_node_t *node = art_deref(iterator->art, ref);+        art_indexed_child_t indexed_child;+        if (first) {+            indexed_child =+                art_node_next_child(node, art_ref_typecode(ref), -1);+        } else {+            indexed_child =+                art_node_prev_child(node, art_ref_typecode(ref), 256);+        }+        art_iterator_down(iterator, ref, indexed_child.index);+        ref = indexed_child.child;+    }+    // We're at a leaf.+    iterator->frames[iterator->frame].ref = ref;+    iterator->frames[iterator->frame].index_in_node = 0;  // Should not matter.+    return art_iterator_valid_loc(iterator, ref);+}++bool art_iterator_move(art_iterator_t *iterator, bool forward) {+    if (art_is_leaf(art_iterator_ref(iterator))) {+        bool went_up = art_iterator_up(iterator);+        if (!went_up) {+            // This leaf is the root, we're done.+            return art_iterator_invalid_loc(iterator);+        }+    }+    // Advance within inner node.+    art_ref_t neighbor_child = art_iterator_neighbor_child(iterator, forward);+    if (neighbor_child != CROARING_ART_NULL_REF) {+        // There is another child at this level, go down to the first or+        // last leaf.+        return art_node_init_iterator(neighbor_child, iterator, forward);+    }+    // No more children at this level, go up.+    return art_iterator_up_and_move(iterator, forward);+}++// Assumes the iterator is positioned at a node with an equal prefix path up+// to the depth of the iterator.+static bool art_node_iterator_lower_bound(art_ref_t ref,+                                          art_iterator_t *iterator,+                                          const art_key_chunk_t key[]) {+    while (!art_is_leaf(ref)) {+        art_inner_node_t *inner_node =+            (art_inner_node_t *)art_deref(iterator->art, ref);+        int prefix_comparison =+            art_compare_prefix(inner_node->prefix, 0, key, iterator->depth,+                               inner_node->prefix_size);+        if (prefix_comparison < 0) {+            // Prefix so far has been equal, but we've found a smaller key.+            // Since we take the lower bound within each node, we can return+            // the next leaf.+            return art_iterator_up_and_move(iterator, true);+        } else if (prefix_comparison > 0) {+            // No key equal to the key we're looking for, return the first+            // leaf.+            return art_node_init_iterator(ref, iterator, true);+        }+        // Prefix is equal, move to lower bound child.+        art_key_chunk_t key_chunk =+            key[iterator->depth + inner_node->prefix_size];+        art_indexed_child_t indexed_child = art_node_lower_bound(+            (art_node_t *)inner_node, art_ref_typecode(ref), key_chunk);+        if (indexed_child.child == CROARING_ART_NULL_REF) {+            // Only smaller keys among children.+            return art_iterator_up_and_move(iterator, true);+        }+        if (indexed_child.key_chunk > key_chunk) {+            // Only larger children, return the first larger child.+            art_iterator_down(iterator, ref, indexed_child.index);+            return art_node_init_iterator(indexed_child.child, iterator, true);+        }+        // We found a child with an equal prefix.+        art_iterator_down(iterator, ref, indexed_child.index);+        ref = indexed_child.child;+    }+    art_leaf_t *leaf = (art_leaf_t *)art_deref(iterator->art, ref);+    if (art_compare_keys(leaf->key, key) >= 0) {+        // Leaf has an equal or larger key.+        return art_iterator_valid_loc(iterator, ref);+    }+    // Leaf has an equal prefix, but the full key is smaller. Move to the+    // next leaf.+    return art_iterator_up_and_move(iterator, true);+}++art_iterator_t art_init_iterator(art_t *art, bool first) {+    art_iterator_t iterator = CROARING_ZERO_INITIALIZER;+    iterator.art = art;+    if (art->root == CROARING_ART_NULL_REF) {+        return iterator;+    }+    art_node_init_iterator(art->root, &iterator, first);+    return iterator;+}++bool art_iterator_next(art_iterator_t *iterator) {+    return art_iterator_move(iterator, true);+}++bool art_iterator_prev(art_iterator_t *iterator) {+    return art_iterator_move(iterator, false);+}++bool art_iterator_lower_bound(art_iterator_t *iterator,+                              const art_key_chunk_t *key) {+    if (iterator->value == NULL) {+        // We're beyond the end / start of the ART so the iterator does not+        // have a valid key. Start from the root.+        iterator->frame = 0;+        iterator->depth = 0;+        art_ref_t root = art_iterator_ref(iterator);+        if (root == CROARING_ART_NULL_REF) {+            return false;+        }+        return art_node_iterator_lower_bound(root, iterator, key);+    }+    int compare_result =+        art_compare_prefix(iterator->key, 0, key, 0, ART_KEY_BYTES);+    // Move up until we have an equal prefix, after which we can do a normal+    // lower bound search.+    while (compare_result != 0) {+        if (!art_iterator_up(iterator)) {+            if (compare_result < 0) {+                // Only smaller keys found.+                return art_iterator_invalid_loc(iterator);+            } else {+                return art_node_init_iterator(art_iterator_ref(iterator),+                                              iterator, true);+            }+        }+        // Since we're only moving up, we can keep comparing against the+        // iterator key.+        art_inner_node_t *inner_node =+            (art_inner_node_t *)art_iterator_node(iterator);+        compare_result =+            art_compare_prefix(iterator->key, 0, key, 0,+                               iterator->depth + inner_node->prefix_size);+    }+    if (compare_result > 0) {+        return art_node_init_iterator(art_iterator_ref(iterator), iterator,+                                      true);+    }+    return art_node_iterator_lower_bound(art_iterator_ref(iterator), iterator,+                                         key);+}++art_iterator_t art_lower_bound(art_t *art, const art_key_chunk_t *key) {+    art_iterator_t iterator = CROARING_ZERO_INITIALIZER;+    iterator.art = art;+    if (art->root != CROARING_ART_NULL_REF) {+        art_node_iterator_lower_bound(art->root, &iterator, key);+    }+    return iterator;+}++art_iterator_t art_upper_bound(art_t *art, const art_key_chunk_t *key) {+    art_iterator_t iterator = CROARING_ZERO_INITIALIZER;+    iterator.art = art;+    if (art->root != CROARING_ART_NULL_REF) {+        if (art_node_iterator_lower_bound(art->root, &iterator, key) &&+            art_compare_keys(iterator.key, key) == 0) {+            art_iterator_next(&iterator);+        }+    }+    return iterator;+}++void art_iterator_insert(art_iterator_t *iterator, const art_key_chunk_t *key,+                         art_val_t val) {+    // TODO: This can likely be faster.+    art_insert(iterator->art, key, val);+    assert(iterator->art->root != CROARING_ART_NULL_REF);+    iterator->frame = 0;+    iterator->depth = 0;+    art_node_iterator_lower_bound(iterator->art->root, iterator, key);+}++bool art_iterator_erase(art_iterator_t *iterator, art_val_t *erased_val) {+    art_val_t erased_val_local;+    if (erased_val == NULL) {+        erased_val = &erased_val_local;+    }+    if (iterator->value == NULL) {+        return false;+    }+    art_key_chunk_t initial_key[ART_KEY_BYTES];+    memcpy(initial_key, iterator->key, ART_KEY_BYTES);++    *erased_val = *iterator->value;+    // Erase the leaf.+    art_node_free(iterator->art, art_iterator_node(iterator),+                  art_ref_typecode(art_iterator_ref(iterator)));+    bool went_up = art_iterator_up(iterator);+    if (!went_up) {+        // We're erasing the root.+        iterator->art->root = CROARING_ART_NULL_REF;+        art_iterator_invalid_loc(iterator);+        return true;+    }++    // Erase the leaf in its parent.+    art_ref_t parent_ref = art_iterator_ref(iterator);+    art_inner_node_t *parent_node =+        (art_inner_node_t *)art_iterator_node(iterator);+    art_key_chunk_t key_chunk_in_parent =+        iterator->key[iterator->depth + parent_node->prefix_size];+    art_ref_t new_parent_ref =+        art_node_erase(iterator->art, parent_node, art_ref_typecode(parent_ref),+                       key_chunk_in_parent);++    if (new_parent_ref != parent_ref) {+        // Replace the pointer to the inner node we erased from in its+        // parent (it may be a leaf now).+        iterator->frames[iterator->frame].ref = new_parent_ref;+        went_up = art_iterator_up(iterator);+        if (went_up) {+            art_ref_t grandparent_ref = art_iterator_ref(iterator);+            art_inner_node_t *grandparent_node =+                (art_inner_node_t *)art_iterator_node(iterator);+            art_key_chunk_t key_chunk_in_grandparent =+                iterator->key[iterator->depth + grandparent_node->prefix_size];+            art_replace(grandparent_node, art_ref_typecode(grandparent_ref),+                        key_chunk_in_grandparent, new_parent_ref);+        } else {+            // We were already at the rootmost node.+            iterator->art->root = new_parent_ref;+        }+    }++    iterator->frame = 0;+    iterator->depth = 0;+    // Do a lower bound search for the initial key, which will find the+    // first greater key if it exists. This can likely be mildly faster if+    // we instead start from the current position.+    art_node_iterator_lower_bound(iterator->art->root, iterator, initial_key);+    return true;+}++static bool art_internal_validate_at(const art_t *art, art_ref_t ref,+                                     art_internal_validate_t validator) {+    if (ref == CROARING_ART_NULL_REF) {+        return art_validate_fail(&validator, "node is null");+    }+    if (art_is_leaf(ref)) {+        art_leaf_t *leaf = (art_leaf_t *)art_deref(art, ref);+        if (art_compare_prefix(leaf->key, 0, validator.current_key, 0,+                               validator.depth) != 0) {+            return art_validate_fail(&validator,+                                     "leaf key does not match its "+                                     "position's prefix in the tree");+        }+        if (validator.validate_cb != NULL &&+            !validator.validate_cb(leaf->val, validator.reason,+                                   validator.context)) {+            if (*validator.reason == NULL) {+                *validator.reason = "leaf validation failed";+            }+            return false;+        }+    } else {+        art_inner_node_t *inner_node = (art_inner_node_t *)art_deref(art, ref);++        if (validator.depth + inner_node->prefix_size + 1 > ART_KEY_BYTES) {+            return art_validate_fail(&validator,+                                     "node has too much prefix at given depth");+        }+        memcpy(validator.current_key + validator.depth, inner_node->prefix,+               inner_node->prefix_size);+        validator.depth += inner_node->prefix_size;++        switch (art_ref_typecode(ref)) {+            case CROARING_ART_NODE4_TYPE:+                if (!art_node4_internal_validate(art, (art_node4_t *)inner_node,+                                                 validator)) {+                    return false;+                }+                break;+            case CROARING_ART_NODE16_TYPE:+                if (!art_node16_internal_validate(+                        art, (art_node16_t *)inner_node, validator)) {+                    return false;+                }+                break;+            case CROARING_ART_NODE48_TYPE:+                if (!art_node48_internal_validate(+                        art, (art_node48_t *)inner_node, validator)) {+                    return false;+                }+                break;+            case CROARING_ART_NODE256_TYPE:+                if (!art_node256_internal_validate(+                        art, (art_node256_t *)inner_node, validator)) {+                    return false;+                }+                break;+            default:+                return art_validate_fail(&validator, "invalid node type");+        }+    }+    return true;+}++bool art_internal_validate(const art_t *art, const char **reason,+                           art_validate_cb_t validate_cb, void *context) {+    const char *reason_local;+    if (reason == NULL) {+        // Always allow assigning through *reason+        reason = &reason_local;+    }+    *reason = NULL;+    if (art->root == CROARING_ART_NULL_REF) {+        return true;+    }+    art_internal_validate_t validator = {+        .reason = reason,+        .validate_cb = validate_cb,+        .context = context,+        .depth = 0,+        .current_key = CROARING_ZERO_INITIALIZER,+    };+    for (art_typecode_t type = CROARING_ART_LEAF_TYPE;+         type <= CROARING_ART_NODE256_TYPE; ++type) {+        uint64_t capacity = art->capacities[type];+        for (uint64_t i = 0; i < capacity; ++i) {+            uint64_t first_free = art->first_free[type];+            if (first_free > capacity) {+                return art_validate_fail(&validator, "first_free > capacity");+            }+        }+    }+    return art_internal_validate_at(art, art->root, validator);+}++CROARING_STATIC_ASSERT(alignof(art_leaf_t) == alignof(art_node4_t),+                       "Serialization assumes node type alignment is equal");+CROARING_STATIC_ASSERT(alignof(art_leaf_t) == alignof(art_node16_t),+                       "Serialization assumes node type alignment is equal");+CROARING_STATIC_ASSERT(alignof(art_leaf_t) == alignof(art_node48_t),+                       "Serialization assumes node type alignment is equal");+CROARING_STATIC_ASSERT(alignof(art_leaf_t) == alignof(art_node256_t),+                       "Serialization assumes node type alignment is equal");++size_t art_size_in_bytes(const art_t *art) {+    if (!art_is_shrunken(art)) {+        return 0;+    }+    // Root.+    size_t size = sizeof(art->root);+    // Node counts.+    size += sizeof(art->capacities);+    // Alignment for leaves. The rest of the nodes are aligned the same way.+    size +=+        ((size + alignof(art_leaf_t) - 1) & ~(alignof(art_leaf_t) - 1)) - size;+    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        size += art->capacities[t] * ART_NODE_SIZES[t];+    }+    return size;+}++size_t art_serialize(const art_t *art, char *buf) {+    if (buf == NULL) {+        return 0;+    }+    if (!art_is_shrunken(art)) {+        return 0;+    }+    const char *initial_buf = buf;++    // Root.+    memcpy(buf, &art->root, sizeof(art->root));+    buf += sizeof(art->root);++    // Node counts.+    memcpy(buf, art->capacities, sizeof(art->capacities));+    buf += sizeof(art->capacities);++    // Alignment for leaves. The rest of the nodes are aligned the same way.+    size_t align_bytes =+        CROARING_ART_ALIGN_SIZE_RELATIVE(buf, initial_buf, alignof(art_leaf_t));+    memset(buf, 0, align_bytes);+    buf += align_bytes;++    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        if (art->capacities[t] > 0) {+            size_t size = art->capacities[t] * ART_NODE_SIZES[t];+            memcpy(buf, art->nodes[t], size);+            buf += size;+        }+    }++    return buf - initial_buf;+}++size_t art_frozen_view(const char *buf, size_t maxbytes, art_t *art) {+    if (buf == NULL || art == NULL) {+        return 0;+    }+    const char *initial_buf = buf;+    art_init_cleared(art);++    if (maxbytes < sizeof(art->root)) {+        return 0;+    }+    memcpy(&art->root, buf, sizeof(art->root));+    buf += sizeof(art->root);+    maxbytes -= sizeof(art->root);++    if (maxbytes < sizeof(art->capacities)) {+        return 0;+    }+    CROARING_STATIC_ASSERT(sizeof(art->first_free) == sizeof(art->capacities),+                           "first_free is read from capacities");+    memcpy(art->first_free, buf, sizeof(art->capacities));+    memcpy(art->capacities, buf, sizeof(art->capacities));+    buf += sizeof(art->capacities);+    maxbytes -= sizeof(art->capacities);++    // Alignment for leaves. The rest of the nodes are aligned the same way.+    const char *before_align = buf;+    buf = CROARING_ART_ALIGN_BUF(buf, alignof(art_leaf_t));+    if (maxbytes < (size_t)(buf - before_align)) {+        return 0;+    }+    maxbytes -= buf - before_align;++    for (art_typecode_t t = CROARING_ART_MIN_TYPE; t <= CROARING_ART_MAX_TYPE;+         ++t) {+        if (art->capacities[t] > 0) {+            size_t size = art->capacities[t] * ART_NODE_SIZES[t];+            if (maxbytes < size) {+                return 0;+            }+            art->nodes[t] = (char *)buf;+            buf += size;+            maxbytes -= size;+        }+    }+    return buf - initial_buf;+}++#ifdef __cplusplus+}  // extern "C"+}  // namespace roaring+}  // namespace internal+#endif
+ cbits/CRoaring/src/bitset.c view
@@ -0,0 +1,501 @@+#include <limits.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/bitset/bitset.h>+#include <roaring/memory.h>+#include <roaring/portability.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++extern inline void bitset_print(const bitset_t *b);+extern inline bool bitset_for_each(const bitset_t *b, bitset_iterator iterator,+                                   void *ptr);+extern inline size_t bitset_next_set_bits(const bitset_t *bitset,+                                          size_t *buffer, size_t capacity,+                                          size_t *startfrom);+extern inline void bitset_set_to_value(bitset_t *bitset, size_t i, bool flag);+extern inline bool bitset_next_set_bit(const bitset_t *bitset, size_t *i);+extern inline void bitset_set(bitset_t *bitset, size_t i);+extern inline bool bitset_get(const bitset_t *bitset, size_t i);+extern inline size_t bitset_size_in_words(const bitset_t *bitset);+extern inline size_t bitset_size_in_bits(const bitset_t *bitset);+extern inline size_t bitset_size_in_bytes(const bitset_t *bitset);++/* Create a new bitset. Return NULL in case of failure. */+bitset_t *bitset_create(void) {+    bitset_t *bitset = NULL;+    /* Allocate the bitset itself. */+    if ((bitset = (bitset_t *)roaring_malloc(sizeof(bitset_t))) == NULL) {+        return NULL;+    }+    bitset->array = NULL;+    bitset->arraysize = 0;+    bitset->capacity = 0;+    return bitset;+}++/* Create a new bitset able to contain size bits. Return NULL in case of+ * failure. */+bitset_t *bitset_create_with_capacity(size_t size) {+    bitset_t *bitset = NULL;+    /* Allocate the bitset itself. */+    if ((bitset = (bitset_t *)roaring_malloc(sizeof(bitset_t))) == NULL) {+        return NULL;+    }+    bitset->arraysize =+        (size + sizeof(uint64_t) * 8 - 1) / (sizeof(uint64_t) * 8);+    bitset->capacity = bitset->arraysize;+    if ((bitset->array = (uint64_t *)roaring_calloc(+             bitset->arraysize, sizeof(uint64_t))) == NULL) {+        roaring_free(bitset);+        return NULL;+    }+    return bitset;+}++/* Create a copy */+bitset_t *bitset_copy(const bitset_t *bitset) {+    bitset_t *copy = NULL;+    /* Allocate the bitset itself. */+    if ((copy = (bitset_t *)roaring_malloc(sizeof(bitset_t))) == NULL) {+        return NULL;+    }+    memcpy(copy, bitset, sizeof(bitset_t));+    copy->capacity = copy->arraysize;+    if ((copy->array = (uint64_t *)roaring_malloc(sizeof(uint64_t) *+                                                  bitset->arraysize)) == NULL) {+        roaring_free(copy);+        return NULL;+    }+    memcpy(copy->array, bitset->array, sizeof(uint64_t) * bitset->arraysize);+    return copy;+}++void bitset_clear(bitset_t *bitset) {+    memset(bitset->array, 0, sizeof(uint64_t) * bitset->arraysize);+}++void bitset_fill(bitset_t *bitset) {+    memset(bitset->array, 0xff, sizeof(uint64_t) * bitset->arraysize);+}++void bitset_shift_left(bitset_t *bitset, size_t s) {+    size_t extra_words = s / 64;+    int inword_shift = s % 64;+    size_t as = bitset->arraysize;+    if (as == 0) {+        return;  // no bits to shift+    }+    if (inword_shift == 0) {+        if (!bitset_resize(bitset, as + extra_words, false)) {+            return;+        }+        // could be done with a memmove+        for (size_t i = as + extra_words; i > extra_words; i--) {+            bitset->array[i - 1] = bitset->array[i - 1 - extra_words];+        }+    } else {+        if (!bitset_resize(bitset, as + extra_words + 1, true)) {+            return;+        }+        bitset->array[as + extra_words] =+            bitset->array[as - 1] >> (64 - inword_shift);+        for (size_t i = as + extra_words; i >= extra_words + 2; i--) {+            bitset->array[i - 1] =+                (bitset->array[i - 1 - extra_words] << inword_shift) |+                (bitset->array[i - 2 - extra_words] >> (64 - inword_shift));+        }+        bitset->array[extra_words] = bitset->array[0] << inword_shift;+    }+    for (size_t i = 0; i < extra_words; i++) {+        bitset->array[i] = 0;+    }+}++void bitset_shift_right(bitset_t *bitset, size_t s) {+    size_t extra_words = s / 64;+    int inword_shift = s % 64;+    size_t as = bitset->arraysize;+    if (extra_words >= as) {+        // every bit is shifted out+        bitset_resize(bitset, 0, false);+        return;+    }+    if (inword_shift == 0) {+        // could be done with a memmove+        for (size_t i = 0; i < as - extra_words; i++) {+            bitset->array[i] = bitset->array[i + extra_words];+        }+        bitset_resize(bitset, as - extra_words, false);++    } else {+        for (size_t i = 0; i + extra_words + 1 < as; i++) {+            bitset->array[i] =+                (bitset->array[i + extra_words] >> inword_shift) |+                (bitset->array[i + extra_words + 1] << (64 - inword_shift));+        }+        bitset->array[as - extra_words - 1] =+            (bitset->array[as - 1] >> inword_shift);+        bitset_resize(bitset, as - extra_words, false);+    }+}++/* Free memory. */+void bitset_free(bitset_t *bitset) {+    if (bitset == NULL) {+        return;+    }+    roaring_free(bitset->array);+    roaring_free(bitset);+}++/* Resize the bitset so that it can support newarraysize * 64 bits. Return true+ * in case of success, false for failure. */+bool bitset_resize(bitset_t *bitset, size_t newarraysize, bool padwithzeroes) {+    if (newarraysize > SIZE_MAX / 64) {+        return false;+    }+    size_t smallest =+        newarraysize < bitset->arraysize ? newarraysize : bitset->arraysize;+    if (bitset->capacity < newarraysize) {+        uint64_t *newarray;+        size_t newcapacity = bitset->capacity;+        if (newcapacity == 0) {+            newcapacity = 1;+        }+        while (newcapacity < newarraysize) {+            newcapacity *= 2;+        }+        if ((newarray = (uint64_t *)roaring_realloc(+                 bitset->array, sizeof(uint64_t) * newcapacity)) == NULL) {+            return false;+        }+        bitset->capacity = newcapacity;+        bitset->array = newarray;+    }+    if (padwithzeroes && (newarraysize > smallest))+        memset(bitset->array + smallest, 0,+               sizeof(uint64_t) * (newarraysize - smallest));+    bitset->arraysize = newarraysize;+    return true;  // success!+}++size_t bitset_count(const bitset_t *bitset) {+    size_t card = 0;+    size_t k = 0;+    for (; k + 7 < bitset->arraysize; k += 8) {+        card += roaring_hamming(bitset->array[k]);+        card += roaring_hamming(bitset->array[k + 1]);+        card += roaring_hamming(bitset->array[k + 2]);+        card += roaring_hamming(bitset->array[k + 3]);+        card += roaring_hamming(bitset->array[k + 4]);+        card += roaring_hamming(bitset->array[k + 5]);+        card += roaring_hamming(bitset->array[k + 6]);+        card += roaring_hamming(bitset->array[k + 7]);+    }+    for (; k + 3 < bitset->arraysize; k += 4) {+        card += roaring_hamming(bitset->array[k]);+        card += roaring_hamming(bitset->array[k + 1]);+        card += roaring_hamming(bitset->array[k + 2]);+        card += roaring_hamming(bitset->array[k + 3]);+    }+    for (; k < bitset->arraysize; k++) {+        card += roaring_hamming(bitset->array[k]);+    }+    return card;+}++bool bitset_inplace_union(bitset_t *CROARING_CBITSET_RESTRICT b1,+                          const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    for (size_t k = 0; k < minlength; ++k) {+        b1->array[k] |= b2->array[k];+    }+    if (b2->arraysize > b1->arraysize) {+        size_t oldsize = b1->arraysize;+        if (!bitset_resize(b1, b2->arraysize, false)) return false;+        memcpy(b1->array + oldsize, b2->array + oldsize,+               (b2->arraysize - oldsize) * sizeof(uint64_t));+    }+    return true;+}++bool bitset_empty(const bitset_t *bitset) {+    for (size_t k = 0; k < bitset->arraysize; k++) {+        if (bitset->array[k] != 0) {+            return false;+        }+    }+    return true;+}++size_t bitset_minimum(const bitset_t *bitset) {+    for (size_t k = 0; k < bitset->arraysize; k++) {+        uint64_t w = bitset->array[k];+        if (w != 0) {+            return roaring_trailing_zeroes(w) + k * 64;+        }+    }+    return SIZE_MAX;+}++bool bitset_grow(bitset_t *bitset, size_t newarraysize) {+    if (newarraysize < bitset->arraysize) {+        return false;+    }+    if (newarraysize > SIZE_MAX / 64) {+        return false;+    }+    if (bitset->capacity < newarraysize) {+        uint64_t *newarray;+        size_t newcapacity = (UINT64_C(0xFFFFFFFFFFFFFFFF) >>+                              roaring_leading_zeroes(newarraysize)) ++                             1;+        while (newcapacity < newarraysize) {+            newcapacity *= 2;+        }+        if ((newarray = (uint64_t *)roaring_realloc(+                 bitset->array, sizeof(uint64_t) * newcapacity)) == NULL) {+            return false;+        }+        bitset->capacity = newcapacity;+        bitset->array = newarray;+    }+    memset(bitset->array + bitset->arraysize, 0,+           sizeof(uint64_t) * (newarraysize - bitset->arraysize));+    bitset->arraysize = newarraysize;+    return true;  // success!+}++size_t bitset_maximum(const bitset_t *bitset) {+    for (size_t k = bitset->arraysize; k > 0; k--) {+        uint64_t w = bitset->array[k - 1];+        if (w != 0) {+            return 63 - roaring_leading_zeroes(w) + (k - 1) * 64;+        }+    }+    return 0;+}++/* Returns true if bitsets share no common elements, false otherwise.+ *+ * Performs early-out if common element found. */+bool bitsets_disjoint(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                      const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;++    for (size_t k = 0; k < minlength; k++) {+        if ((b1->array[k] & b2->array[k]) != 0) return false;+    }+    return true;+}++/* Returns true if bitsets contain at least 1 common element, false if they are+ * disjoint.+ *+ * Performs early-out if common element found. */+bool bitsets_intersect(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                       const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;++    for (size_t k = 0; k < minlength; k++) {+        if ((b1->array[k] & b2->array[k]) != 0) return true;+    }+    return false;+}++/* Returns true if b has any bits set in or after b->array[starting_loc]. */+static bool any_bits_set(const bitset_t *b, size_t starting_loc) {+    if (starting_loc >= b->arraysize) {+        return false;+    }+    for (size_t k = starting_loc; k < b->arraysize; k++) {+        if (b->array[k] != 0) return true;+    }+    return false;+}++/* Returns true if b1 has all of b2's bits set.+ *+ * Performs early out if a bit is found in b2 that is not found in b1. */+bool bitset_contains_all(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                         const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t min_size = b1->arraysize;+    if (b1->arraysize > b2->arraysize) {+        min_size = b2->arraysize;+    }+    for (size_t k = 0; k < min_size; k++) {+        if ((b1->array[k] & b2->array[k]) != b2->array[k]) {+            return false;+        }+    }+    if (b2->arraysize > b1->arraysize) {+        /* Need to check if b2 has any bits set beyond b1's array */+        return !any_bits_set(b2, b1->arraysize);+    }+    return true;+}++size_t bitset_union_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                          const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t answer = 0;+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    for (; k + 3 < minlength; k += 4) {+        answer += roaring_hamming(b1->array[k] | b2->array[k]);+        answer += roaring_hamming(b1->array[k + 1] | b2->array[k + 1]);+        answer += roaring_hamming(b1->array[k + 2] | b2->array[k + 2]);+        answer += roaring_hamming(b1->array[k + 3] | b2->array[k + 3]);+    }+    for (; k < minlength; ++k) {+        answer += roaring_hamming(b1->array[k] | b2->array[k]);+    }+    if (b2->arraysize > b1->arraysize) {+        // k is equal to b1->arraysize+        for (; k + 3 < b2->arraysize; k += 4) {+            answer += roaring_hamming(b2->array[k]);+            answer += roaring_hamming(b2->array[k + 1]);+            answer += roaring_hamming(b2->array[k + 2]);+            answer += roaring_hamming(b2->array[k + 3]);+        }+        for (; k < b2->arraysize; ++k) {+            answer += roaring_hamming(b2->array[k]);+        }+    } else {+        // k is equal to b2->arraysize+        for (; k + 3 < b1->arraysize; k += 4) {+            answer += roaring_hamming(b1->array[k]);+            answer += roaring_hamming(b1->array[k + 1]);+            answer += roaring_hamming(b1->array[k + 2]);+            answer += roaring_hamming(b1->array[k + 3]);+        }+        for (; k < b1->arraysize; ++k) {+            answer += roaring_hamming(b1->array[k]);+        }+    }+    return answer;+}++void bitset_inplace_intersection(bitset_t *CROARING_CBITSET_RESTRICT b1,+                                 const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    for (; k < minlength; ++k) {+        b1->array[k] &= b2->array[k];+    }+    for (; k < b1->arraysize; ++k) {+        b1->array[k] = 0;  // memset could, maybe, be a tiny bit faster+    }+}++size_t bitset_intersection_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                                 const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t answer = 0;+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    for (size_t k = 0; k < minlength; ++k) {+        answer += roaring_hamming(b1->array[k] & b2->array[k]);+    }+    return answer;+}++void bitset_inplace_difference(bitset_t *CROARING_CBITSET_RESTRICT b1,+                               const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    for (; k < minlength; ++k) {+        b1->array[k] &= ~(b2->array[k]);+    }+}++size_t bitset_difference_count(const bitset_t *CROARING_CBITSET_RESTRICT b1,+                               const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    size_t answer = 0;+    for (; k < minlength; ++k) {+        answer += roaring_hamming(b1->array[k] & ~(b2->array[k]));+    }+    for (; k < b1->arraysize; ++k) {+        answer += roaring_hamming(b1->array[k]);+    }+    return answer;+}++bool bitset_inplace_symmetric_difference(+    bitset_t *CROARING_CBITSET_RESTRICT b1,+    const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    for (; k < minlength; ++k) {+        b1->array[k] ^= b2->array[k];+    }+    if (b2->arraysize > b1->arraysize) {+        size_t oldsize = b1->arraysize;+        if (!bitset_resize(b1, b2->arraysize, false)) return false;+        memcpy(b1->array + oldsize, b2->array + oldsize,+               (b2->arraysize - oldsize) * sizeof(uint64_t));+    }+    return true;+}++size_t bitset_symmetric_difference_count(+    const bitset_t *CROARING_CBITSET_RESTRICT b1,+    const bitset_t *CROARING_CBITSET_RESTRICT b2) {+    size_t minlength =+        b1->arraysize < b2->arraysize ? b1->arraysize : b2->arraysize;+    size_t k = 0;+    size_t answer = 0;+    for (; k < minlength; ++k) {+        answer += roaring_hamming(b1->array[k] ^ b2->array[k]);+    }+    if (b2->arraysize > b1->arraysize) {+        for (; k < b2->arraysize; ++k) {+            answer += roaring_hamming(b2->array[k]);+        }+    } else {+        for (; k < b1->arraysize; ++k) {+            answer += roaring_hamming(b1->array[k]);+        }+    }+    return answer;+}++bool bitset_trim(bitset_t *bitset) {+    size_t newsize = bitset->arraysize;+    while (newsize > 0) {+        if (bitset->array[newsize - 1] == 0)+            newsize -= 1;+        else+            break;+    }+    if (bitset->capacity == newsize) return true;  // nothing to do+    bool shrunk;+    bitset->array = (uint64_t *)croaring_realloc_shrink(+        bitset->array, sizeof(uint64_t) * newsize, &shrunk);+    if (!shrunk) return false;+    bitset->capacity = newsize;+    bitset->arraysize = newsize;+    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/bitset_util.c view
@@ -0,0 +1,1366 @@+#include <assert.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/bitset_util.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+using namespace ::roaring::internal;+extern "C" {+namespace roaring {+namespace api {+#endif++#if CROARING_IS_X64+static uint8_t lengthTable[256] = {+    0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, 1, 2, 2, 3, 2, 3, 3, 4,+    2, 3, 3, 4, 3, 4, 4, 5, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,+    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 1, 2, 2, 3, 2, 3, 3, 4,+    2, 3, 3, 4, 3, 4, 4, 5, 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,+    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6,+    4, 5, 5, 6, 5, 6, 6, 7, 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,+    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 2, 3, 3, 4, 3, 4, 4, 5,+    3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,+    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, 3, 4, 4, 5, 4, 5, 5, 6,+    4, 5, 5, 6, 5, 6, 6, 7, 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,+    4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8};+#endif++#if CROARING_IS_X64+ALIGNED(32)+static uint32_t vecDecodeTable[256][8] = {+    {0, 0, 0, 0, 0, 0, 0, 0}, /* 0x00 (00000000) */+    {1, 0, 0, 0, 0, 0, 0, 0}, /* 0x01 (00000001) */+    {2, 0, 0, 0, 0, 0, 0, 0}, /* 0x02 (00000010) */+    {1, 2, 0, 0, 0, 0, 0, 0}, /* 0x03 (00000011) */+    {3, 0, 0, 0, 0, 0, 0, 0}, /* 0x04 (00000100) */+    {1, 3, 0, 0, 0, 0, 0, 0}, /* 0x05 (00000101) */+    {2, 3, 0, 0, 0, 0, 0, 0}, /* 0x06 (00000110) */+    {1, 2, 3, 0, 0, 0, 0, 0}, /* 0x07 (00000111) */+    {4, 0, 0, 0, 0, 0, 0, 0}, /* 0x08 (00001000) */+    {1, 4, 0, 0, 0, 0, 0, 0}, /* 0x09 (00001001) */+    {2, 4, 0, 0, 0, 0, 0, 0}, /* 0x0A (00001010) */+    {1, 2, 4, 0, 0, 0, 0, 0}, /* 0x0B (00001011) */+    {3, 4, 0, 0, 0, 0, 0, 0}, /* 0x0C (00001100) */+    {1, 3, 4, 0, 0, 0, 0, 0}, /* 0x0D (00001101) */+    {2, 3, 4, 0, 0, 0, 0, 0}, /* 0x0E (00001110) */+    {1, 2, 3, 4, 0, 0, 0, 0}, /* 0x0F (00001111) */+    {5, 0, 0, 0, 0, 0, 0, 0}, /* 0x10 (00010000) */+    {1, 5, 0, 0, 0, 0, 0, 0}, /* 0x11 (00010001) */+    {2, 5, 0, 0, 0, 0, 0, 0}, /* 0x12 (00010010) */+    {1, 2, 5, 0, 0, 0, 0, 0}, /* 0x13 (00010011) */+    {3, 5, 0, 0, 0, 0, 0, 0}, /* 0x14 (00010100) */+    {1, 3, 5, 0, 0, 0, 0, 0}, /* 0x15 (00010101) */+    {2, 3, 5, 0, 0, 0, 0, 0}, /* 0x16 (00010110) */+    {1, 2, 3, 5, 0, 0, 0, 0}, /* 0x17 (00010111) */+    {4, 5, 0, 0, 0, 0, 0, 0}, /* 0x18 (00011000) */+    {1, 4, 5, 0, 0, 0, 0, 0}, /* 0x19 (00011001) */+    {2, 4, 5, 0, 0, 0, 0, 0}, /* 0x1A (00011010) */+    {1, 2, 4, 5, 0, 0, 0, 0}, /* 0x1B (00011011) */+    {3, 4, 5, 0, 0, 0, 0, 0}, /* 0x1C (00011100) */+    {1, 3, 4, 5, 0, 0, 0, 0}, /* 0x1D (00011101) */+    {2, 3, 4, 5, 0, 0, 0, 0}, /* 0x1E (00011110) */+    {1, 2, 3, 4, 5, 0, 0, 0}, /* 0x1F (00011111) */+    {6, 0, 0, 0, 0, 0, 0, 0}, /* 0x20 (00100000) */+    {1, 6, 0, 0, 0, 0, 0, 0}, /* 0x21 (00100001) */+    {2, 6, 0, 0, 0, 0, 0, 0}, /* 0x22 (00100010) */+    {1, 2, 6, 0, 0, 0, 0, 0}, /* 0x23 (00100011) */+    {3, 6, 0, 0, 0, 0, 0, 0}, /* 0x24 (00100100) */+    {1, 3, 6, 0, 0, 0, 0, 0}, /* 0x25 (00100101) */+    {2, 3, 6, 0, 0, 0, 0, 0}, /* 0x26 (00100110) */+    {1, 2, 3, 6, 0, 0, 0, 0}, /* 0x27 (00100111) */+    {4, 6, 0, 0, 0, 0, 0, 0}, /* 0x28 (00101000) */+    {1, 4, 6, 0, 0, 0, 0, 0}, /* 0x29 (00101001) */+    {2, 4, 6, 0, 0, 0, 0, 0}, /* 0x2A (00101010) */+    {1, 2, 4, 6, 0, 0, 0, 0}, /* 0x2B (00101011) */+    {3, 4, 6, 0, 0, 0, 0, 0}, /* 0x2C (00101100) */+    {1, 3, 4, 6, 0, 0, 0, 0}, /* 0x2D (00101101) */+    {2, 3, 4, 6, 0, 0, 0, 0}, /* 0x2E (00101110) */+    {1, 2, 3, 4, 6, 0, 0, 0}, /* 0x2F (00101111) */+    {5, 6, 0, 0, 0, 0, 0, 0}, /* 0x30 (00110000) */+    {1, 5, 6, 0, 0, 0, 0, 0}, /* 0x31 (00110001) */+    {2, 5, 6, 0, 0, 0, 0, 0}, /* 0x32 (00110010) */+    {1, 2, 5, 6, 0, 0, 0, 0}, /* 0x33 (00110011) */+    {3, 5, 6, 0, 0, 0, 0, 0}, /* 0x34 (00110100) */+    {1, 3, 5, 6, 0, 0, 0, 0}, /* 0x35 (00110101) */+    {2, 3, 5, 6, 0, 0, 0, 0}, /* 0x36 (00110110) */+    {1, 2, 3, 5, 6, 0, 0, 0}, /* 0x37 (00110111) */+    {4, 5, 6, 0, 0, 0, 0, 0}, /* 0x38 (00111000) */+    {1, 4, 5, 6, 0, 0, 0, 0}, /* 0x39 (00111001) */+    {2, 4, 5, 6, 0, 0, 0, 0}, /* 0x3A (00111010) */+    {1, 2, 4, 5, 6, 0, 0, 0}, /* 0x3B (00111011) */+    {3, 4, 5, 6, 0, 0, 0, 0}, /* 0x3C (00111100) */+    {1, 3, 4, 5, 6, 0, 0, 0}, /* 0x3D (00111101) */+    {2, 3, 4, 5, 6, 0, 0, 0}, /* 0x3E (00111110) */+    {1, 2, 3, 4, 5, 6, 0, 0}, /* 0x3F (00111111) */+    {7, 0, 0, 0, 0, 0, 0, 0}, /* 0x40 (01000000) */+    {1, 7, 0, 0, 0, 0, 0, 0}, /* 0x41 (01000001) */+    {2, 7, 0, 0, 0, 0, 0, 0}, /* 0x42 (01000010) */+    {1, 2, 7, 0, 0, 0, 0, 0}, /* 0x43 (01000011) */+    {3, 7, 0, 0, 0, 0, 0, 0}, /* 0x44 (01000100) */+    {1, 3, 7, 0, 0, 0, 0, 0}, /* 0x45 (01000101) */+    {2, 3, 7, 0, 0, 0, 0, 0}, /* 0x46 (01000110) */+    {1, 2, 3, 7, 0, 0, 0, 0}, /* 0x47 (01000111) */+    {4, 7, 0, 0, 0, 0, 0, 0}, /* 0x48 (01001000) */+    {1, 4, 7, 0, 0, 0, 0, 0}, /* 0x49 (01001001) */+    {2, 4, 7, 0, 0, 0, 0, 0}, /* 0x4A (01001010) */+    {1, 2, 4, 7, 0, 0, 0, 0}, /* 0x4B (01001011) */+    {3, 4, 7, 0, 0, 0, 0, 0}, /* 0x4C (01001100) */+    {1, 3, 4, 7, 0, 0, 0, 0}, /* 0x4D (01001101) */+    {2, 3, 4, 7, 0, 0, 0, 0}, /* 0x4E (01001110) */+    {1, 2, 3, 4, 7, 0, 0, 0}, /* 0x4F (01001111) */+    {5, 7, 0, 0, 0, 0, 0, 0}, /* 0x50 (01010000) */+    {1, 5, 7, 0, 0, 0, 0, 0}, /* 0x51 (01010001) */+    {2, 5, 7, 0, 0, 0, 0, 0}, /* 0x52 (01010010) */+    {1, 2, 5, 7, 0, 0, 0, 0}, /* 0x53 (01010011) */+    {3, 5, 7, 0, 0, 0, 0, 0}, /* 0x54 (01010100) */+    {1, 3, 5, 7, 0, 0, 0, 0}, /* 0x55 (01010101) */+    {2, 3, 5, 7, 0, 0, 0, 0}, /* 0x56 (01010110) */+    {1, 2, 3, 5, 7, 0, 0, 0}, /* 0x57 (01010111) */+    {4, 5, 7, 0, 0, 0, 0, 0}, /* 0x58 (01011000) */+    {1, 4, 5, 7, 0, 0, 0, 0}, /* 0x59 (01011001) */+    {2, 4, 5, 7, 0, 0, 0, 0}, /* 0x5A (01011010) */+    {1, 2, 4, 5, 7, 0, 0, 0}, /* 0x5B (01011011) */+    {3, 4, 5, 7, 0, 0, 0, 0}, /* 0x5C (01011100) */+    {1, 3, 4, 5, 7, 0, 0, 0}, /* 0x5D (01011101) */+    {2, 3, 4, 5, 7, 0, 0, 0}, /* 0x5E (01011110) */+    {1, 2, 3, 4, 5, 7, 0, 0}, /* 0x5F (01011111) */+    {6, 7, 0, 0, 0, 0, 0, 0}, /* 0x60 (01100000) */+    {1, 6, 7, 0, 0, 0, 0, 0}, /* 0x61 (01100001) */+    {2, 6, 7, 0, 0, 0, 0, 0}, /* 0x62 (01100010) */+    {1, 2, 6, 7, 0, 0, 0, 0}, /* 0x63 (01100011) */+    {3, 6, 7, 0, 0, 0, 0, 0}, /* 0x64 (01100100) */+    {1, 3, 6, 7, 0, 0, 0, 0}, /* 0x65 (01100101) */+    {2, 3, 6, 7, 0, 0, 0, 0}, /* 0x66 (01100110) */+    {1, 2, 3, 6, 7, 0, 0, 0}, /* 0x67 (01100111) */+    {4, 6, 7, 0, 0, 0, 0, 0}, /* 0x68 (01101000) */+    {1, 4, 6, 7, 0, 0, 0, 0}, /* 0x69 (01101001) */+    {2, 4, 6, 7, 0, 0, 0, 0}, /* 0x6A (01101010) */+    {1, 2, 4, 6, 7, 0, 0, 0}, /* 0x6B (01101011) */+    {3, 4, 6, 7, 0, 0, 0, 0}, /* 0x6C (01101100) */+    {1, 3, 4, 6, 7, 0, 0, 0}, /* 0x6D (01101101) */+    {2, 3, 4, 6, 7, 0, 0, 0}, /* 0x6E (01101110) */+    {1, 2, 3, 4, 6, 7, 0, 0}, /* 0x6F (01101111) */+    {5, 6, 7, 0, 0, 0, 0, 0}, /* 0x70 (01110000) */+    {1, 5, 6, 7, 0, 0, 0, 0}, /* 0x71 (01110001) */+    {2, 5, 6, 7, 0, 0, 0, 0}, /* 0x72 (01110010) */+    {1, 2, 5, 6, 7, 0, 0, 0}, /* 0x73 (01110011) */+    {3, 5, 6, 7, 0, 0, 0, 0}, /* 0x74 (01110100) */+    {1, 3, 5, 6, 7, 0, 0, 0}, /* 0x75 (01110101) */+    {2, 3, 5, 6, 7, 0, 0, 0}, /* 0x76 (01110110) */+    {1, 2, 3, 5, 6, 7, 0, 0}, /* 0x77 (01110111) */+    {4, 5, 6, 7, 0, 0, 0, 0}, /* 0x78 (01111000) */+    {1, 4, 5, 6, 7, 0, 0, 0}, /* 0x79 (01111001) */+    {2, 4, 5, 6, 7, 0, 0, 0}, /* 0x7A (01111010) */+    {1, 2, 4, 5, 6, 7, 0, 0}, /* 0x7B (01111011) */+    {3, 4, 5, 6, 7, 0, 0, 0}, /* 0x7C (01111100) */+    {1, 3, 4, 5, 6, 7, 0, 0}, /* 0x7D (01111101) */+    {2, 3, 4, 5, 6, 7, 0, 0}, /* 0x7E (01111110) */+    {1, 2, 3, 4, 5, 6, 7, 0}, /* 0x7F (01111111) */+    {8, 0, 0, 0, 0, 0, 0, 0}, /* 0x80 (10000000) */+    {1, 8, 0, 0, 0, 0, 0, 0}, /* 0x81 (10000001) */+    {2, 8, 0, 0, 0, 0, 0, 0}, /* 0x82 (10000010) */+    {1, 2, 8, 0, 0, 0, 0, 0}, /* 0x83 (10000011) */+    {3, 8, 0, 0, 0, 0, 0, 0}, /* 0x84 (10000100) */+    {1, 3, 8, 0, 0, 0, 0, 0}, /* 0x85 (10000101) */+    {2, 3, 8, 0, 0, 0, 0, 0}, /* 0x86 (10000110) */+    {1, 2, 3, 8, 0, 0, 0, 0}, /* 0x87 (10000111) */+    {4, 8, 0, 0, 0, 0, 0, 0}, /* 0x88 (10001000) */+    {1, 4, 8, 0, 0, 0, 0, 0}, /* 0x89 (10001001) */+    {2, 4, 8, 0, 0, 0, 0, 0}, /* 0x8A (10001010) */+    {1, 2, 4, 8, 0, 0, 0, 0}, /* 0x8B (10001011) */+    {3, 4, 8, 0, 0, 0, 0, 0}, /* 0x8C (10001100) */+    {1, 3, 4, 8, 0, 0, 0, 0}, /* 0x8D (10001101) */+    {2, 3, 4, 8, 0, 0, 0, 0}, /* 0x8E (10001110) */+    {1, 2, 3, 4, 8, 0, 0, 0}, /* 0x8F (10001111) */+    {5, 8, 0, 0, 0, 0, 0, 0}, /* 0x90 (10010000) */+    {1, 5, 8, 0, 0, 0, 0, 0}, /* 0x91 (10010001) */+    {2, 5, 8, 0, 0, 0, 0, 0}, /* 0x92 (10010010) */+    {1, 2, 5, 8, 0, 0, 0, 0}, /* 0x93 (10010011) */+    {3, 5, 8, 0, 0, 0, 0, 0}, /* 0x94 (10010100) */+    {1, 3, 5, 8, 0, 0, 0, 0}, /* 0x95 (10010101) */+    {2, 3, 5, 8, 0, 0, 0, 0}, /* 0x96 (10010110) */+    {1, 2, 3, 5, 8, 0, 0, 0}, /* 0x97 (10010111) */+    {4, 5, 8, 0, 0, 0, 0, 0}, /* 0x98 (10011000) */+    {1, 4, 5, 8, 0, 0, 0, 0}, /* 0x99 (10011001) */+    {2, 4, 5, 8, 0, 0, 0, 0}, /* 0x9A (10011010) */+    {1, 2, 4, 5, 8, 0, 0, 0}, /* 0x9B (10011011) */+    {3, 4, 5, 8, 0, 0, 0, 0}, /* 0x9C (10011100) */+    {1, 3, 4, 5, 8, 0, 0, 0}, /* 0x9D (10011101) */+    {2, 3, 4, 5, 8, 0, 0, 0}, /* 0x9E (10011110) */+    {1, 2, 3, 4, 5, 8, 0, 0}, /* 0x9F (10011111) */+    {6, 8, 0, 0, 0, 0, 0, 0}, /* 0xA0 (10100000) */+    {1, 6, 8, 0, 0, 0, 0, 0}, /* 0xA1 (10100001) */+    {2, 6, 8, 0, 0, 0, 0, 0}, /* 0xA2 (10100010) */+    {1, 2, 6, 8, 0, 0, 0, 0}, /* 0xA3 (10100011) */+    {3, 6, 8, 0, 0, 0, 0, 0}, /* 0xA4 (10100100) */+    {1, 3, 6, 8, 0, 0, 0, 0}, /* 0xA5 (10100101) */+    {2, 3, 6, 8, 0, 0, 0, 0}, /* 0xA6 (10100110) */+    {1, 2, 3, 6, 8, 0, 0, 0}, /* 0xA7 (10100111) */+    {4, 6, 8, 0, 0, 0, 0, 0}, /* 0xA8 (10101000) */+    {1, 4, 6, 8, 0, 0, 0, 0}, /* 0xA9 (10101001) */+    {2, 4, 6, 8, 0, 0, 0, 0}, /* 0xAA (10101010) */+    {1, 2, 4, 6, 8, 0, 0, 0}, /* 0xAB (10101011) */+    {3, 4, 6, 8, 0, 0, 0, 0}, /* 0xAC (10101100) */+    {1, 3, 4, 6, 8, 0, 0, 0}, /* 0xAD (10101101) */+    {2, 3, 4, 6, 8, 0, 0, 0}, /* 0xAE (10101110) */+    {1, 2, 3, 4, 6, 8, 0, 0}, /* 0xAF (10101111) */+    {5, 6, 8, 0, 0, 0, 0, 0}, /* 0xB0 (10110000) */+    {1, 5, 6, 8, 0, 0, 0, 0}, /* 0xB1 (10110001) */+    {2, 5, 6, 8, 0, 0, 0, 0}, /* 0xB2 (10110010) */+    {1, 2, 5, 6, 8, 0, 0, 0}, /* 0xB3 (10110011) */+    {3, 5, 6, 8, 0, 0, 0, 0}, /* 0xB4 (10110100) */+    {1, 3, 5, 6, 8, 0, 0, 0}, /* 0xB5 (10110101) */+    {2, 3, 5, 6, 8, 0, 0, 0}, /* 0xB6 (10110110) */+    {1, 2, 3, 5, 6, 8, 0, 0}, /* 0xB7 (10110111) */+    {4, 5, 6, 8, 0, 0, 0, 0}, /* 0xB8 (10111000) */+    {1, 4, 5, 6, 8, 0, 0, 0}, /* 0xB9 (10111001) */+    {2, 4, 5, 6, 8, 0, 0, 0}, /* 0xBA (10111010) */+    {1, 2, 4, 5, 6, 8, 0, 0}, /* 0xBB (10111011) */+    {3, 4, 5, 6, 8, 0, 0, 0}, /* 0xBC (10111100) */+    {1, 3, 4, 5, 6, 8, 0, 0}, /* 0xBD (10111101) */+    {2, 3, 4, 5, 6, 8, 0, 0}, /* 0xBE (10111110) */+    {1, 2, 3, 4, 5, 6, 8, 0}, /* 0xBF (10111111) */+    {7, 8, 0, 0, 0, 0, 0, 0}, /* 0xC0 (11000000) */+    {1, 7, 8, 0, 0, 0, 0, 0}, /* 0xC1 (11000001) */+    {2, 7, 8, 0, 0, 0, 0, 0}, /* 0xC2 (11000010) */+    {1, 2, 7, 8, 0, 0, 0, 0}, /* 0xC3 (11000011) */+    {3, 7, 8, 0, 0, 0, 0, 0}, /* 0xC4 (11000100) */+    {1, 3, 7, 8, 0, 0, 0, 0}, /* 0xC5 (11000101) */+    {2, 3, 7, 8, 0, 0, 0, 0}, /* 0xC6 (11000110) */+    {1, 2, 3, 7, 8, 0, 0, 0}, /* 0xC7 (11000111) */+    {4, 7, 8, 0, 0, 0, 0, 0}, /* 0xC8 (11001000) */+    {1, 4, 7, 8, 0, 0, 0, 0}, /* 0xC9 (11001001) */+    {2, 4, 7, 8, 0, 0, 0, 0}, /* 0xCA (11001010) */+    {1, 2, 4, 7, 8, 0, 0, 0}, /* 0xCB (11001011) */+    {3, 4, 7, 8, 0, 0, 0, 0}, /* 0xCC (11001100) */+    {1, 3, 4, 7, 8, 0, 0, 0}, /* 0xCD (11001101) */+    {2, 3, 4, 7, 8, 0, 0, 0}, /* 0xCE (11001110) */+    {1, 2, 3, 4, 7, 8, 0, 0}, /* 0xCF (11001111) */+    {5, 7, 8, 0, 0, 0, 0, 0}, /* 0xD0 (11010000) */+    {1, 5, 7, 8, 0, 0, 0, 0}, /* 0xD1 (11010001) */+    {2, 5, 7, 8, 0, 0, 0, 0}, /* 0xD2 (11010010) */+    {1, 2, 5, 7, 8, 0, 0, 0}, /* 0xD3 (11010011) */+    {3, 5, 7, 8, 0, 0, 0, 0}, /* 0xD4 (11010100) */+    {1, 3, 5, 7, 8, 0, 0, 0}, /* 0xD5 (11010101) */+    {2, 3, 5, 7, 8, 0, 0, 0}, /* 0xD6 (11010110) */+    {1, 2, 3, 5, 7, 8, 0, 0}, /* 0xD7 (11010111) */+    {4, 5, 7, 8, 0, 0, 0, 0}, /* 0xD8 (11011000) */+    {1, 4, 5, 7, 8, 0, 0, 0}, /* 0xD9 (11011001) */+    {2, 4, 5, 7, 8, 0, 0, 0}, /* 0xDA (11011010) */+    {1, 2, 4, 5, 7, 8, 0, 0}, /* 0xDB (11011011) */+    {3, 4, 5, 7, 8, 0, 0, 0}, /* 0xDC (11011100) */+    {1, 3, 4, 5, 7, 8, 0, 0}, /* 0xDD (11011101) */+    {2, 3, 4, 5, 7, 8, 0, 0}, /* 0xDE (11011110) */+    {1, 2, 3, 4, 5, 7, 8, 0}, /* 0xDF (11011111) */+    {6, 7, 8, 0, 0, 0, 0, 0}, /* 0xE0 (11100000) */+    {1, 6, 7, 8, 0, 0, 0, 0}, /* 0xE1 (11100001) */+    {2, 6, 7, 8, 0, 0, 0, 0}, /* 0xE2 (11100010) */+    {1, 2, 6, 7, 8, 0, 0, 0}, /* 0xE3 (11100011) */+    {3, 6, 7, 8, 0, 0, 0, 0}, /* 0xE4 (11100100) */+    {1, 3, 6, 7, 8, 0, 0, 0}, /* 0xE5 (11100101) */+    {2, 3, 6, 7, 8, 0, 0, 0}, /* 0xE6 (11100110) */+    {1, 2, 3, 6, 7, 8, 0, 0}, /* 0xE7 (11100111) */+    {4, 6, 7, 8, 0, 0, 0, 0}, /* 0xE8 (11101000) */+    {1, 4, 6, 7, 8, 0, 0, 0}, /* 0xE9 (11101001) */+    {2, 4, 6, 7, 8, 0, 0, 0}, /* 0xEA (11101010) */+    {1, 2, 4, 6, 7, 8, 0, 0}, /* 0xEB (11101011) */+    {3, 4, 6, 7, 8, 0, 0, 0}, /* 0xEC (11101100) */+    {1, 3, 4, 6, 7, 8, 0, 0}, /* 0xED (11101101) */+    {2, 3, 4, 6, 7, 8, 0, 0}, /* 0xEE (11101110) */+    {1, 2, 3, 4, 6, 7, 8, 0}, /* 0xEF (11101111) */+    {5, 6, 7, 8, 0, 0, 0, 0}, /* 0xF0 (11110000) */+    {1, 5, 6, 7, 8, 0, 0, 0}, /* 0xF1 (11110001) */+    {2, 5, 6, 7, 8, 0, 0, 0}, /* 0xF2 (11110010) */+    {1, 2, 5, 6, 7, 8, 0, 0}, /* 0xF3 (11110011) */+    {3, 5, 6, 7, 8, 0, 0, 0}, /* 0xF4 (11110100) */+    {1, 3, 5, 6, 7, 8, 0, 0}, /* 0xF5 (11110101) */+    {2, 3, 5, 6, 7, 8, 0, 0}, /* 0xF6 (11110110) */+    {1, 2, 3, 5, 6, 7, 8, 0}, /* 0xF7 (11110111) */+    {4, 5, 6, 7, 8, 0, 0, 0}, /* 0xF8 (11111000) */+    {1, 4, 5, 6, 7, 8, 0, 0}, /* 0xF9 (11111001) */+    {2, 4, 5, 6, 7, 8, 0, 0}, /* 0xFA (11111010) */+    {1, 2, 4, 5, 6, 7, 8, 0}, /* 0xFB (11111011) */+    {3, 4, 5, 6, 7, 8, 0, 0}, /* 0xFC (11111100) */+    {1, 3, 4, 5, 6, 7, 8, 0}, /* 0xFD (11111101) */+    {2, 3, 4, 5, 6, 7, 8, 0}, /* 0xFE (11111110) */+    {1, 2, 3, 4, 5, 6, 7, 8}  /* 0xFF (11111111) */+};++#endif  // #if CROARING_IS_X64++#if CROARING_IS_X64+// same as vecDecodeTable but in 16 bits+ALIGNED(32)+static uint16_t vecDecodeTable_uint16[256][8] = {+    {0, 0, 0, 0, 0, 0, 0, 0}, /* 0x00 (00000000) */+    {1, 0, 0, 0, 0, 0, 0, 0}, /* 0x01 (00000001) */+    {2, 0, 0, 0, 0, 0, 0, 0}, /* 0x02 (00000010) */+    {1, 2, 0, 0, 0, 0, 0, 0}, /* 0x03 (00000011) */+    {3, 0, 0, 0, 0, 0, 0, 0}, /* 0x04 (00000100) */+    {1, 3, 0, 0, 0, 0, 0, 0}, /* 0x05 (00000101) */+    {2, 3, 0, 0, 0, 0, 0, 0}, /* 0x06 (00000110) */+    {1, 2, 3, 0, 0, 0, 0, 0}, /* 0x07 (00000111) */+    {4, 0, 0, 0, 0, 0, 0, 0}, /* 0x08 (00001000) */+    {1, 4, 0, 0, 0, 0, 0, 0}, /* 0x09 (00001001) */+    {2, 4, 0, 0, 0, 0, 0, 0}, /* 0x0A (00001010) */+    {1, 2, 4, 0, 0, 0, 0, 0}, /* 0x0B (00001011) */+    {3, 4, 0, 0, 0, 0, 0, 0}, /* 0x0C (00001100) */+    {1, 3, 4, 0, 0, 0, 0, 0}, /* 0x0D (00001101) */+    {2, 3, 4, 0, 0, 0, 0, 0}, /* 0x0E (00001110) */+    {1, 2, 3, 4, 0, 0, 0, 0}, /* 0x0F (00001111) */+    {5, 0, 0, 0, 0, 0, 0, 0}, /* 0x10 (00010000) */+    {1, 5, 0, 0, 0, 0, 0, 0}, /* 0x11 (00010001) */+    {2, 5, 0, 0, 0, 0, 0, 0}, /* 0x12 (00010010) */+    {1, 2, 5, 0, 0, 0, 0, 0}, /* 0x13 (00010011) */+    {3, 5, 0, 0, 0, 0, 0, 0}, /* 0x14 (00010100) */+    {1, 3, 5, 0, 0, 0, 0, 0}, /* 0x15 (00010101) */+    {2, 3, 5, 0, 0, 0, 0, 0}, /* 0x16 (00010110) */+    {1, 2, 3, 5, 0, 0, 0, 0}, /* 0x17 (00010111) */+    {4, 5, 0, 0, 0, 0, 0, 0}, /* 0x18 (00011000) */+    {1, 4, 5, 0, 0, 0, 0, 0}, /* 0x19 (00011001) */+    {2, 4, 5, 0, 0, 0, 0, 0}, /* 0x1A (00011010) */+    {1, 2, 4, 5, 0, 0, 0, 0}, /* 0x1B (00011011) */+    {3, 4, 5, 0, 0, 0, 0, 0}, /* 0x1C (00011100) */+    {1, 3, 4, 5, 0, 0, 0, 0}, /* 0x1D (00011101) */+    {2, 3, 4, 5, 0, 0, 0, 0}, /* 0x1E (00011110) */+    {1, 2, 3, 4, 5, 0, 0, 0}, /* 0x1F (00011111) */+    {6, 0, 0, 0, 0, 0, 0, 0}, /* 0x20 (00100000) */+    {1, 6, 0, 0, 0, 0, 0, 0}, /* 0x21 (00100001) */+    {2, 6, 0, 0, 0, 0, 0, 0}, /* 0x22 (00100010) */+    {1, 2, 6, 0, 0, 0, 0, 0}, /* 0x23 (00100011) */+    {3, 6, 0, 0, 0, 0, 0, 0}, /* 0x24 (00100100) */+    {1, 3, 6, 0, 0, 0, 0, 0}, /* 0x25 (00100101) */+    {2, 3, 6, 0, 0, 0, 0, 0}, /* 0x26 (00100110) */+    {1, 2, 3, 6, 0, 0, 0, 0}, /* 0x27 (00100111) */+    {4, 6, 0, 0, 0, 0, 0, 0}, /* 0x28 (00101000) */+    {1, 4, 6, 0, 0, 0, 0, 0}, /* 0x29 (00101001) */+    {2, 4, 6, 0, 0, 0, 0, 0}, /* 0x2A (00101010) */+    {1, 2, 4, 6, 0, 0, 0, 0}, /* 0x2B (00101011) */+    {3, 4, 6, 0, 0, 0, 0, 0}, /* 0x2C (00101100) */+    {1, 3, 4, 6, 0, 0, 0, 0}, /* 0x2D (00101101) */+    {2, 3, 4, 6, 0, 0, 0, 0}, /* 0x2E (00101110) */+    {1, 2, 3, 4, 6, 0, 0, 0}, /* 0x2F (00101111) */+    {5, 6, 0, 0, 0, 0, 0, 0}, /* 0x30 (00110000) */+    {1, 5, 6, 0, 0, 0, 0, 0}, /* 0x31 (00110001) */+    {2, 5, 6, 0, 0, 0, 0, 0}, /* 0x32 (00110010) */+    {1, 2, 5, 6, 0, 0, 0, 0}, /* 0x33 (00110011) */+    {3, 5, 6, 0, 0, 0, 0, 0}, /* 0x34 (00110100) */+    {1, 3, 5, 6, 0, 0, 0, 0}, /* 0x35 (00110101) */+    {2, 3, 5, 6, 0, 0, 0, 0}, /* 0x36 (00110110) */+    {1, 2, 3, 5, 6, 0, 0, 0}, /* 0x37 (00110111) */+    {4, 5, 6, 0, 0, 0, 0, 0}, /* 0x38 (00111000) */+    {1, 4, 5, 6, 0, 0, 0, 0}, /* 0x39 (00111001) */+    {2, 4, 5, 6, 0, 0, 0, 0}, /* 0x3A (00111010) */+    {1, 2, 4, 5, 6, 0, 0, 0}, /* 0x3B (00111011) */+    {3, 4, 5, 6, 0, 0, 0, 0}, /* 0x3C (00111100) */+    {1, 3, 4, 5, 6, 0, 0, 0}, /* 0x3D (00111101) */+    {2, 3, 4, 5, 6, 0, 0, 0}, /* 0x3E (00111110) */+    {1, 2, 3, 4, 5, 6, 0, 0}, /* 0x3F (00111111) */+    {7, 0, 0, 0, 0, 0, 0, 0}, /* 0x40 (01000000) */+    {1, 7, 0, 0, 0, 0, 0, 0}, /* 0x41 (01000001) */+    {2, 7, 0, 0, 0, 0, 0, 0}, /* 0x42 (01000010) */+    {1, 2, 7, 0, 0, 0, 0, 0}, /* 0x43 (01000011) */+    {3, 7, 0, 0, 0, 0, 0, 0}, /* 0x44 (01000100) */+    {1, 3, 7, 0, 0, 0, 0, 0}, /* 0x45 (01000101) */+    {2, 3, 7, 0, 0, 0, 0, 0}, /* 0x46 (01000110) */+    {1, 2, 3, 7, 0, 0, 0, 0}, /* 0x47 (01000111) */+    {4, 7, 0, 0, 0, 0, 0, 0}, /* 0x48 (01001000) */+    {1, 4, 7, 0, 0, 0, 0, 0}, /* 0x49 (01001001) */+    {2, 4, 7, 0, 0, 0, 0, 0}, /* 0x4A (01001010) */+    {1, 2, 4, 7, 0, 0, 0, 0}, /* 0x4B (01001011) */+    {3, 4, 7, 0, 0, 0, 0, 0}, /* 0x4C (01001100) */+    {1, 3, 4, 7, 0, 0, 0, 0}, /* 0x4D (01001101) */+    {2, 3, 4, 7, 0, 0, 0, 0}, /* 0x4E (01001110) */+    {1, 2, 3, 4, 7, 0, 0, 0}, /* 0x4F (01001111) */+    {5, 7, 0, 0, 0, 0, 0, 0}, /* 0x50 (01010000) */+    {1, 5, 7, 0, 0, 0, 0, 0}, /* 0x51 (01010001) */+    {2, 5, 7, 0, 0, 0, 0, 0}, /* 0x52 (01010010) */+    {1, 2, 5, 7, 0, 0, 0, 0}, /* 0x53 (01010011) */+    {3, 5, 7, 0, 0, 0, 0, 0}, /* 0x54 (01010100) */+    {1, 3, 5, 7, 0, 0, 0, 0}, /* 0x55 (01010101) */+    {2, 3, 5, 7, 0, 0, 0, 0}, /* 0x56 (01010110) */+    {1, 2, 3, 5, 7, 0, 0, 0}, /* 0x57 (01010111) */+    {4, 5, 7, 0, 0, 0, 0, 0}, /* 0x58 (01011000) */+    {1, 4, 5, 7, 0, 0, 0, 0}, /* 0x59 (01011001) */+    {2, 4, 5, 7, 0, 0, 0, 0}, /* 0x5A (01011010) */+    {1, 2, 4, 5, 7, 0, 0, 0}, /* 0x5B (01011011) */+    {3, 4, 5, 7, 0, 0, 0, 0}, /* 0x5C (01011100) */+    {1, 3, 4, 5, 7, 0, 0, 0}, /* 0x5D (01011101) */+    {2, 3, 4, 5, 7, 0, 0, 0}, /* 0x5E (01011110) */+    {1, 2, 3, 4, 5, 7, 0, 0}, /* 0x5F (01011111) */+    {6, 7, 0, 0, 0, 0, 0, 0}, /* 0x60 (01100000) */+    {1, 6, 7, 0, 0, 0, 0, 0}, /* 0x61 (01100001) */+    {2, 6, 7, 0, 0, 0, 0, 0}, /* 0x62 (01100010) */+    {1, 2, 6, 7, 0, 0, 0, 0}, /* 0x63 (01100011) */+    {3, 6, 7, 0, 0, 0, 0, 0}, /* 0x64 (01100100) */+    {1, 3, 6, 7, 0, 0, 0, 0}, /* 0x65 (01100101) */+    {2, 3, 6, 7, 0, 0, 0, 0}, /* 0x66 (01100110) */+    {1, 2, 3, 6, 7, 0, 0, 0}, /* 0x67 (01100111) */+    {4, 6, 7, 0, 0, 0, 0, 0}, /* 0x68 (01101000) */+    {1, 4, 6, 7, 0, 0, 0, 0}, /* 0x69 (01101001) */+    {2, 4, 6, 7, 0, 0, 0, 0}, /* 0x6A (01101010) */+    {1, 2, 4, 6, 7, 0, 0, 0}, /* 0x6B (01101011) */+    {3, 4, 6, 7, 0, 0, 0, 0}, /* 0x6C (01101100) */+    {1, 3, 4, 6, 7, 0, 0, 0}, /* 0x6D (01101101) */+    {2, 3, 4, 6, 7, 0, 0, 0}, /* 0x6E (01101110) */+    {1, 2, 3, 4, 6, 7, 0, 0}, /* 0x6F (01101111) */+    {5, 6, 7, 0, 0, 0, 0, 0}, /* 0x70 (01110000) */+    {1, 5, 6, 7, 0, 0, 0, 0}, /* 0x71 (01110001) */+    {2, 5, 6, 7, 0, 0, 0, 0}, /* 0x72 (01110010) */+    {1, 2, 5, 6, 7, 0, 0, 0}, /* 0x73 (01110011) */+    {3, 5, 6, 7, 0, 0, 0, 0}, /* 0x74 (01110100) */+    {1, 3, 5, 6, 7, 0, 0, 0}, /* 0x75 (01110101) */+    {2, 3, 5, 6, 7, 0, 0, 0}, /* 0x76 (01110110) */+    {1, 2, 3, 5, 6, 7, 0, 0}, /* 0x77 (01110111) */+    {4, 5, 6, 7, 0, 0, 0, 0}, /* 0x78 (01111000) */+    {1, 4, 5, 6, 7, 0, 0, 0}, /* 0x79 (01111001) */+    {2, 4, 5, 6, 7, 0, 0, 0}, /* 0x7A (01111010) */+    {1, 2, 4, 5, 6, 7, 0, 0}, /* 0x7B (01111011) */+    {3, 4, 5, 6, 7, 0, 0, 0}, /* 0x7C (01111100) */+    {1, 3, 4, 5, 6, 7, 0, 0}, /* 0x7D (01111101) */+    {2, 3, 4, 5, 6, 7, 0, 0}, /* 0x7E (01111110) */+    {1, 2, 3, 4, 5, 6, 7, 0}, /* 0x7F (01111111) */+    {8, 0, 0, 0, 0, 0, 0, 0}, /* 0x80 (10000000) */+    {1, 8, 0, 0, 0, 0, 0, 0}, /* 0x81 (10000001) */+    {2, 8, 0, 0, 0, 0, 0, 0}, /* 0x82 (10000010) */+    {1, 2, 8, 0, 0, 0, 0, 0}, /* 0x83 (10000011) */+    {3, 8, 0, 0, 0, 0, 0, 0}, /* 0x84 (10000100) */+    {1, 3, 8, 0, 0, 0, 0, 0}, /* 0x85 (10000101) */+    {2, 3, 8, 0, 0, 0, 0, 0}, /* 0x86 (10000110) */+    {1, 2, 3, 8, 0, 0, 0, 0}, /* 0x87 (10000111) */+    {4, 8, 0, 0, 0, 0, 0, 0}, /* 0x88 (10001000) */+    {1, 4, 8, 0, 0, 0, 0, 0}, /* 0x89 (10001001) */+    {2, 4, 8, 0, 0, 0, 0, 0}, /* 0x8A (10001010) */+    {1, 2, 4, 8, 0, 0, 0, 0}, /* 0x8B (10001011) */+    {3, 4, 8, 0, 0, 0, 0, 0}, /* 0x8C (10001100) */+    {1, 3, 4, 8, 0, 0, 0, 0}, /* 0x8D (10001101) */+    {2, 3, 4, 8, 0, 0, 0, 0}, /* 0x8E (10001110) */+    {1, 2, 3, 4, 8, 0, 0, 0}, /* 0x8F (10001111) */+    {5, 8, 0, 0, 0, 0, 0, 0}, /* 0x90 (10010000) */+    {1, 5, 8, 0, 0, 0, 0, 0}, /* 0x91 (10010001) */+    {2, 5, 8, 0, 0, 0, 0, 0}, /* 0x92 (10010010) */+    {1, 2, 5, 8, 0, 0, 0, 0}, /* 0x93 (10010011) */+    {3, 5, 8, 0, 0, 0, 0, 0}, /* 0x94 (10010100) */+    {1, 3, 5, 8, 0, 0, 0, 0}, /* 0x95 (10010101) */+    {2, 3, 5, 8, 0, 0, 0, 0}, /* 0x96 (10010110) */+    {1, 2, 3, 5, 8, 0, 0, 0}, /* 0x97 (10010111) */+    {4, 5, 8, 0, 0, 0, 0, 0}, /* 0x98 (10011000) */+    {1, 4, 5, 8, 0, 0, 0, 0}, /* 0x99 (10011001) */+    {2, 4, 5, 8, 0, 0, 0, 0}, /* 0x9A (10011010) */+    {1, 2, 4, 5, 8, 0, 0, 0}, /* 0x9B (10011011) */+    {3, 4, 5, 8, 0, 0, 0, 0}, /* 0x9C (10011100) */+    {1, 3, 4, 5, 8, 0, 0, 0}, /* 0x9D (10011101) */+    {2, 3, 4, 5, 8, 0, 0, 0}, /* 0x9E (10011110) */+    {1, 2, 3, 4, 5, 8, 0, 0}, /* 0x9F (10011111) */+    {6, 8, 0, 0, 0, 0, 0, 0}, /* 0xA0 (10100000) */+    {1, 6, 8, 0, 0, 0, 0, 0}, /* 0xA1 (10100001) */+    {2, 6, 8, 0, 0, 0, 0, 0}, /* 0xA2 (10100010) */+    {1, 2, 6, 8, 0, 0, 0, 0}, /* 0xA3 (10100011) */+    {3, 6, 8, 0, 0, 0, 0, 0}, /* 0xA4 (10100100) */+    {1, 3, 6, 8, 0, 0, 0, 0}, /* 0xA5 (10100101) */+    {2, 3, 6, 8, 0, 0, 0, 0}, /* 0xA6 (10100110) */+    {1, 2, 3, 6, 8, 0, 0, 0}, /* 0xA7 (10100111) */+    {4, 6, 8, 0, 0, 0, 0, 0}, /* 0xA8 (10101000) */+    {1, 4, 6, 8, 0, 0, 0, 0}, /* 0xA9 (10101001) */+    {2, 4, 6, 8, 0, 0, 0, 0}, /* 0xAA (10101010) */+    {1, 2, 4, 6, 8, 0, 0, 0}, /* 0xAB (10101011) */+    {3, 4, 6, 8, 0, 0, 0, 0}, /* 0xAC (10101100) */+    {1, 3, 4, 6, 8, 0, 0, 0}, /* 0xAD (10101101) */+    {2, 3, 4, 6, 8, 0, 0, 0}, /* 0xAE (10101110) */+    {1, 2, 3, 4, 6, 8, 0, 0}, /* 0xAF (10101111) */+    {5, 6, 8, 0, 0, 0, 0, 0}, /* 0xB0 (10110000) */+    {1, 5, 6, 8, 0, 0, 0, 0}, /* 0xB1 (10110001) */+    {2, 5, 6, 8, 0, 0, 0, 0}, /* 0xB2 (10110010) */+    {1, 2, 5, 6, 8, 0, 0, 0}, /* 0xB3 (10110011) */+    {3, 5, 6, 8, 0, 0, 0, 0}, /* 0xB4 (10110100) */+    {1, 3, 5, 6, 8, 0, 0, 0}, /* 0xB5 (10110101) */+    {2, 3, 5, 6, 8, 0, 0, 0}, /* 0xB6 (10110110) */+    {1, 2, 3, 5, 6, 8, 0, 0}, /* 0xB7 (10110111) */+    {4, 5, 6, 8, 0, 0, 0, 0}, /* 0xB8 (10111000) */+    {1, 4, 5, 6, 8, 0, 0, 0}, /* 0xB9 (10111001) */+    {2, 4, 5, 6, 8, 0, 0, 0}, /* 0xBA (10111010) */+    {1, 2, 4, 5, 6, 8, 0, 0}, /* 0xBB (10111011) */+    {3, 4, 5, 6, 8, 0, 0, 0}, /* 0xBC (10111100) */+    {1, 3, 4, 5, 6, 8, 0, 0}, /* 0xBD (10111101) */+    {2, 3, 4, 5, 6, 8, 0, 0}, /* 0xBE (10111110) */+    {1, 2, 3, 4, 5, 6, 8, 0}, /* 0xBF (10111111) */+    {7, 8, 0, 0, 0, 0, 0, 0}, /* 0xC0 (11000000) */+    {1, 7, 8, 0, 0, 0, 0, 0}, /* 0xC1 (11000001) */+    {2, 7, 8, 0, 0, 0, 0, 0}, /* 0xC2 (11000010) */+    {1, 2, 7, 8, 0, 0, 0, 0}, /* 0xC3 (11000011) */+    {3, 7, 8, 0, 0, 0, 0, 0}, /* 0xC4 (11000100) */+    {1, 3, 7, 8, 0, 0, 0, 0}, /* 0xC5 (11000101) */+    {2, 3, 7, 8, 0, 0, 0, 0}, /* 0xC6 (11000110) */+    {1, 2, 3, 7, 8, 0, 0, 0}, /* 0xC7 (11000111) */+    {4, 7, 8, 0, 0, 0, 0, 0}, /* 0xC8 (11001000) */+    {1, 4, 7, 8, 0, 0, 0, 0}, /* 0xC9 (11001001) */+    {2, 4, 7, 8, 0, 0, 0, 0}, /* 0xCA (11001010) */+    {1, 2, 4, 7, 8, 0, 0, 0}, /* 0xCB (11001011) */+    {3, 4, 7, 8, 0, 0, 0, 0}, /* 0xCC (11001100) */+    {1, 3, 4, 7, 8, 0, 0, 0}, /* 0xCD (11001101) */+    {2, 3, 4, 7, 8, 0, 0, 0}, /* 0xCE (11001110) */+    {1, 2, 3, 4, 7, 8, 0, 0}, /* 0xCF (11001111) */+    {5, 7, 8, 0, 0, 0, 0, 0}, /* 0xD0 (11010000) */+    {1, 5, 7, 8, 0, 0, 0, 0}, /* 0xD1 (11010001) */+    {2, 5, 7, 8, 0, 0, 0, 0}, /* 0xD2 (11010010) */+    {1, 2, 5, 7, 8, 0, 0, 0}, /* 0xD3 (11010011) */+    {3, 5, 7, 8, 0, 0, 0, 0}, /* 0xD4 (11010100) */+    {1, 3, 5, 7, 8, 0, 0, 0}, /* 0xD5 (11010101) */+    {2, 3, 5, 7, 8, 0, 0, 0}, /* 0xD6 (11010110) */+    {1, 2, 3, 5, 7, 8, 0, 0}, /* 0xD7 (11010111) */+    {4, 5, 7, 8, 0, 0, 0, 0}, /* 0xD8 (11011000) */+    {1, 4, 5, 7, 8, 0, 0, 0}, /* 0xD9 (11011001) */+    {2, 4, 5, 7, 8, 0, 0, 0}, /* 0xDA (11011010) */+    {1, 2, 4, 5, 7, 8, 0, 0}, /* 0xDB (11011011) */+    {3, 4, 5, 7, 8, 0, 0, 0}, /* 0xDC (11011100) */+    {1, 3, 4, 5, 7, 8, 0, 0}, /* 0xDD (11011101) */+    {2, 3, 4, 5, 7, 8, 0, 0}, /* 0xDE (11011110) */+    {1, 2, 3, 4, 5, 7, 8, 0}, /* 0xDF (11011111) */+    {6, 7, 8, 0, 0, 0, 0, 0}, /* 0xE0 (11100000) */+    {1, 6, 7, 8, 0, 0, 0, 0}, /* 0xE1 (11100001) */+    {2, 6, 7, 8, 0, 0, 0, 0}, /* 0xE2 (11100010) */+    {1, 2, 6, 7, 8, 0, 0, 0}, /* 0xE3 (11100011) */+    {3, 6, 7, 8, 0, 0, 0, 0}, /* 0xE4 (11100100) */+    {1, 3, 6, 7, 8, 0, 0, 0}, /* 0xE5 (11100101) */+    {2, 3, 6, 7, 8, 0, 0, 0}, /* 0xE6 (11100110) */+    {1, 2, 3, 6, 7, 8, 0, 0}, /* 0xE7 (11100111) */+    {4, 6, 7, 8, 0, 0, 0, 0}, /* 0xE8 (11101000) */+    {1, 4, 6, 7, 8, 0, 0, 0}, /* 0xE9 (11101001) */+    {2, 4, 6, 7, 8, 0, 0, 0}, /* 0xEA (11101010) */+    {1, 2, 4, 6, 7, 8, 0, 0}, /* 0xEB (11101011) */+    {3, 4, 6, 7, 8, 0, 0, 0}, /* 0xEC (11101100) */+    {1, 3, 4, 6, 7, 8, 0, 0}, /* 0xED (11101101) */+    {2, 3, 4, 6, 7, 8, 0, 0}, /* 0xEE (11101110) */+    {1, 2, 3, 4, 6, 7, 8, 0}, /* 0xEF (11101111) */+    {5, 6, 7, 8, 0, 0, 0, 0}, /* 0xF0 (11110000) */+    {1, 5, 6, 7, 8, 0, 0, 0}, /* 0xF1 (11110001) */+    {2, 5, 6, 7, 8, 0, 0, 0}, /* 0xF2 (11110010) */+    {1, 2, 5, 6, 7, 8, 0, 0}, /* 0xF3 (11110011) */+    {3, 5, 6, 7, 8, 0, 0, 0}, /* 0xF4 (11110100) */+    {1, 3, 5, 6, 7, 8, 0, 0}, /* 0xF5 (11110101) */+    {2, 3, 5, 6, 7, 8, 0, 0}, /* 0xF6 (11110110) */+    {1, 2, 3, 5, 6, 7, 8, 0}, /* 0xF7 (11110111) */+    {4, 5, 6, 7, 8, 0, 0, 0}, /* 0xF8 (11111000) */+    {1, 4, 5, 6, 7, 8, 0, 0}, /* 0xF9 (11111001) */+    {2, 4, 5, 6, 7, 8, 0, 0}, /* 0xFA (11111010) */+    {1, 2, 4, 5, 6, 7, 8, 0}, /* 0xFB (11111011) */+    {3, 4, 5, 6, 7, 8, 0, 0}, /* 0xFC (11111100) */+    {1, 3, 4, 5, 6, 7, 8, 0}, /* 0xFD (11111101) */+    {2, 3, 4, 5, 6, 7, 8, 0}, /* 0xFE (11111110) */+    {1, 2, 3, 4, 5, 6, 7, 8}  /* 0xFF (11111111) */+};++#endif++#if CROARING_IS_X64+#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+const uint8_t vbmi2_table[64] = {+    0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14, 15,+    16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,+    32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,+    48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63};+// Reference:+// https://lemire.me/blog/2022/05/10/faster-bitset-decoding-using-intel-avx-512/+//+// One VPCOMPRESSB (AVX-512 VBMI2) turns a whole 64-bit word into the (up to+// 64) byte-sized positions of its set bits in a single shot; the positions are+// then widened 16 at a time with VPMOVZXBD and added to a running base vector.+//+// Two details matter for speed. Each block is stored under a mask derived from+// BZHI(-1, popcount), so a store writes exactly the values it produced: the+// caller needs no padding past the values it asked for, and no store bandwidth+// goes to values nobody wants. And only the ceil(popcount/16) blocks that+// carry a value are computed at all -- bitset containers are usually well+// under half full, so the later blocks are almost always skipped.+CROARING_ALLOW_UNALIGNED+size_t bitset_extract_setbits_avx512(const uint64_t *words, size_t length,+                                     uint32_t *vout, size_t outcapacity,+                                     uint32_t base) {+    if (outcapacity == 0) return 0;+    uint32_t *out = vout;+    uint32_t *initout = out;+    uint32_t *safeout = out + outcapacity;+    // base_v holds the value of bit 0 of the word being decoded.+    __m512i base_v = _mm512_set1_epi32((int)base);+    const __m512i inc_v = _mm512_set1_epi32(64);+    const __m512i index_table = _mm512_loadu_si512(vbmi2_table);+    size_t i = 0;++    for (; i < length; i++) {+        uint64_t v = words[i];+        if (v != 0) {+            size_t advance = (size_t)roaring_hamming(v);+            if (out + advance > safeout) break;+            __m512i vec = _mm512_maskz_compress_epi8((__mmask64)v, index_table);+            // BZHI(-1, advance): advance is in [1, 64] here, so the shift is+            // well defined.+            uint64_t bits = ~UINT64_C(0) >> (64 - advance);+            _mm512_mask_storeu_epi32(+                out, (__mmask16)bits,+                _mm512_add_epi32(+                    base_v, _mm512_cvtepu8_epi32(_mm512_castsi512_si128(vec))));+            if (advance > 16) {+                _mm512_mask_storeu_epi32(+                    out + 16, (__mmask16)(bits >> 16),+                    _mm512_add_epi32(base_v,+                                     _mm512_cvtepu8_epi32(+                                         _mm512_extracti32x4_epi32(vec, 1))));+                if (advance > 32) {+                    _mm512_mask_storeu_epi32(+                        out + 32, (__mmask16)(bits >> 32),+                        _mm512_add_epi32(+                            base_v, _mm512_cvtepu8_epi32(+                                        _mm512_extracti32x4_epi32(vec, 2))));+                    if (advance > 48) {+                        _mm512_mask_storeu_epi32(+                            out + 48, (__mmask16)(bits >> 48),+                            _mm512_add_epi32(+                                base_v,+                                _mm512_cvtepu8_epi32(+                                    _mm512_extracti32x4_epi32(vec, 3))));+                    }+                }+            }+            out += advance;+        }+        base_v = _mm512_add_epi32(base_v, inc_v);+    }++    base += (uint32_t)(i * 64);++    for (; (i < length) && (out < safeout); ++i) {+        uint64_t w = words[i];+        while ((w != 0) && (out < safeout)) {+            int r =+                roaring_trailing_zeroes(w);  // on x64, should compile to TZCNT+            uint32_t val = r + base;+            memcpy(out, &val,+                   sizeof(uint32_t));  // should be compiled as a MOV on x64+            out++;+            w &= (w - 1);+        }+        base += 64;+    }++    return out - initout;+}++// Writes the "advance" values of the word "v" to "out", adding them to the+// 16-bit lanes of "base_v".+static inline void avx512_decode_word_uint16(uint64_t v, unsigned advance,+                                             __m512i base_v,+                                             __m512i index_table,+                                             uint16_t *out) {+    __m512i vec = _mm512_maskz_compress_epi8((__mmask64)v, index_table);+    _mm512_mask_storeu_epi16(+        out, (__mmask32)_bzhi_u64(~UINT64_C(0), advance),+        _mm512_add_epi16(base_v,+                         _mm512_cvtepu8_epi16(_mm512_castsi512_si256(vec))));+    if (advance > 32) {+        _mm512_mask_storeu_epi16(+            out + 32, (__mmask32)_bzhi_u64(~UINT64_C(0), advance - 32),+            _mm512_add_epi16(base_v, _mm512_cvtepu8_epi16(+                                         _mm512_extracti64x4_epi64(vec, 1))));+    }+}++// Same kernel as bitset_extract_setbits_avx512, writing 16-bit values, so the+// blocks hold 32 values instead of 16. It is called by+// array_container_from_bitset, with "capacity" set to the cardinality, on+// sparse input (at most 4096 values), where many words can be zero. How best+// to skip them depends on how many there are:+//+// - Testing each word ("if (v != 0)") is mispredicted about once per value+//   when zero and nonzero words alternate, as when about 1000 values are+//   spread out: that was up to 2.7x slower than the loops below.+// - From 2048 values (two per word on average), we decode every word without+//   testing it: with v == 0, the masked store writes nothing and we advance by+//   zero.+// - Below 2048 values, decoding every word wastes time when most are zero, so+//   we test eight words at a time and visit only the nonzero ones: a run of+//   zero words costs little, and a group of eight words costs about one+//   misprediction, when the inner loop exits.+//+// See microbenchmarks/bitset_to_array_bench.cpp. Either way, we stop once+// "capacity" values are written, so that we do not scan the words past the+// last value.+CROARING_ALLOW_UNALIGNED+size_t bitset_extract_setbits_avx512_uint16(const uint64_t *array,+                                            size_t length, uint16_t *vout,+                                            size_t capacity, uint16_t base) {+    if (capacity == 0) return 0;+    uint16_t *out = vout;+    uint16_t *initout = out;+    uint16_t *safeout = vout + capacity;++    const __m512i index_table = _mm512_loadu_si512(vbmi2_table);+    size_t i = 0;++    // When "capacity" truncates the output, the loops below stop at the word+    // that does not fit, and the scalar loop at the end writes the values that+    // do.+    if (capacity >= 2048) {+        __m512i base_v = _mm512_set1_epi16((short)base);+        const __m512i inc_v = _mm512_set1_epi16(64);+        // Four words hold at most 256 values: no need to check the capacity+        // for each one while there is room for that many.+        for (; (i + 4 <= length) && ((size_t)(safeout - out) >= 4 * 64);+             i += 4) {+            for (size_t j = 0; j < 4; j++) {+                uint64_t v = array[i + j];+                unsigned advance = (unsigned)roaring_hamming(v);+                avx512_decode_word_uint16(v, advance, base_v, index_table, out);+                out += advance;+                base_v = _mm512_add_epi16(base_v, inc_v);+            }+        }+        for (; (i < length) && (out < safeout); i++) {+            uint64_t v = array[i];+            unsigned advance = (unsigned)roaring_hamming(v);+            if ((size_t)(safeout - out) < advance) break;+            avx512_decode_word_uint16(v, advance, base_v, index_table, out);+            out += advance;+            base_v = _mm512_add_epi16(base_v, inc_v);+        }+    } else {+        bool truncated = false;+        while ((i + 8 <= length) && (out < safeout)) {+            __m512i words = _mm512_loadu_si512((const __m512i *)(array + i));+            uint32_t nonzero = (uint32_t)_mm512_test_epi64_mask(words, words);+            while (nonzero != 0) {+                size_t j = (size_t)roaring_trailing_zeroes(nonzero);+                uint64_t v = array[i + j];+                unsigned advance = (unsigned)roaring_hamming(v);+                if ((size_t)(safeout - out) < advance) {+                    i += j;+                    truncated = true;+                    break;+                }+                avx512_decode_word_uint16(+                    v, advance,+                    _mm512_set1_epi16((short)(uint16_t)(base + (i + j) * 64)),+                    index_table, out);+                out += advance;+                nonzero &= nonzero - 1;+            }+            if (truncated) break;+            i += 8;+        }+    }++    base = (uint16_t)(base + i * 64);++    for (; (i < length) && (out < safeout); ++i) {+        uint64_t w = array[i];+        while ((w != 0) && (out < safeout)) {+            int r =+                roaring_trailing_zeroes(w);  // on x64, should compile to TZCNT+            *out = (uint16_t)(r + base);+            out++;+            w &= (w - 1);+        }+        base += 64;+    }++    return out - initout;+}++// Unlike bitset_extract_setbits_avx512_uint16, there is no "if (v != 0)"+// test: on sparse intersections, about half the words are zero and the test+// is mispredicted about once per value. With v == 0, the masked store writes+// nothing and we advance by zero. The stores never write past the last value.+// We stop once all "card" values are written: masked stores with an empty+// mask can still need a slow microcode assist when they touch a page that+// was never written, such as one past the end of "out".+CROARING_ALLOW_UNALIGNED+size_t bitset_extract_intersection_setbits_avx512_uint16(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, size_t card, uint16_t base) {+    uint16_t *initout = out;+    uint16_t *const endout = out + card;+    __m512i base_v = _mm512_set1_epi16((short)base);+    const __m512i inc_v = _mm512_set1_epi16(64);+    const __m512i index_table = _mm512_loadu_si512(vbmi2_table);+    for (size_t i = 0; i < length && out < endout; i++) {+        uint64_t v = words1[i] & words2[i];+        unsigned advance = (unsigned)roaring_hamming(v);+        __m512i vec = _mm512_maskz_compress_epi8((__mmask64)v, index_table);+        _mm512_mask_storeu_epi16(+            out, (__mmask32)_bzhi_u64(~UINT64_C(0), advance),+            _mm512_add_epi16(+                base_v, _mm512_cvtepu8_epi16(_mm512_castsi512_si256(vec))));+        if (advance > 32) {+            _mm512_mask_storeu_epi16(+                out + 32, (__mmask32)_bzhi_u64(~UINT64_C(0), advance - 32),+                _mm512_add_epi16(+                    base_v,+                    _mm512_cvtepu8_epi16(_mm512_extracti64x4_epi64(vec, 1))));+        }+        out += advance;+        base_v = _mm512_add_epi16(base_v, inc_v);+    }+    return out - initout;+}+CROARING_UNTARGET_AVX512+#endif++CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+size_t bitset_extract_setbits_avx2(const uint64_t *words, size_t length,+                                   uint32_t *out, size_t outcapacity,+                                   uint32_t base) {+    if (outcapacity == 0) return 0;+    uint32_t *initout = out;+    __m256i baseVec = _mm256_set1_epi32(base - 1);+    __m256i incVec = _mm256_set1_epi32(64);+    __m256i add8 = _mm256_set1_epi32(8);+    uint32_t *safeout = out + outcapacity;+    size_t i = 0;+    for (; (i < length) && (out + 64 <= safeout); ++i) {+        uint64_t w = words[i];+        if (w == 0) {+            baseVec = _mm256_add_epi32(baseVec, incVec);+        } else {+            for (int k = 0; k < 4; ++k) {+                uint8_t byteA = (uint8_t)w;+                uint8_t byteB = (uint8_t)(w >> 8);+                w >>= 16;+                __m256i vecA =+                    _mm256_loadu_si256((const __m256i *)vecDecodeTable[byteA]);+                __m256i vecB =+                    _mm256_loadu_si256((const __m256i *)vecDecodeTable[byteB]);+                uint8_t advanceA = lengthTable[byteA];+                uint8_t advanceB = lengthTable[byteB];+                vecA = _mm256_add_epi32(baseVec, vecA);+                baseVec = _mm256_add_epi32(baseVec, add8);+                vecB = _mm256_add_epi32(baseVec, vecB);+                baseVec = _mm256_add_epi32(baseVec, add8);+                _mm256_storeu_si256((__m256i *)out, vecA);+                out += advanceA;+                _mm256_storeu_si256((__m256i *)out, vecB);+                out += advanceB;+            }+        }+    }+    base += i * 64;+    for (; (i < length) && (out < safeout); ++i) {+        uint64_t w = words[i];+        while ((w != 0) && (out < safeout)) {+            int r =+                roaring_trailing_zeroes(w);  // on x64, should compile to TZCNT+            uint32_t val = r + base;+            memcpy(out, &val,+                   sizeof(uint32_t));  // should be compiled as a MOV on x64+            out++;+            w &= (w - 1);+        }+        base += 64;+    }+    return out - initout;+}+CROARING_UNTARGET_AVX2+#endif  // CROARING_IS_X64++CROARING_ALLOW_UNALIGNED+size_t bitset_extract_setbits(const uint64_t *words, size_t length,+                              uint32_t *out, uint32_t base) {+    int outpos = 0;+    for (size_t i = 0; i < length; ++i) {+        uint64_t w = words[i];+        while (w != 0) {+            int r =+                roaring_trailing_zeroes(w);  // on x64, should compile to TZCNT+            uint32_t val = r + base;+            memcpy(out + outpos, &val,+                   sizeof(uint32_t));  // should be compiled as a MOV on x64+            outpos++;+            w &= (w - 1);+        }+        base += 64;+    }+    return outpos;+}++size_t bitset_extract_intersection_setbits_uint16(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, uint16_t base) {+    int outpos = 0;+    for (size_t i = 0; i < length; ++i) {+        uint64_t w = words1[i] & words2[i];+        while (w != 0) {+            int r = roaring_trailing_zeroes(w);+            out[outpos++] = (uint16_t)(r + base);+            w &= (w - 1);+        }+        base += 64;+    }+    return outpos;+}++// Decodes the words of (words1 AND words2) from index *i onward, writing K+// values per word unconditionally and advancing only past the real ones, so+// that there is no data-dependent branch unless a word has more than K bits.+// The trailing-zero count of (w | 1 << 63) is that of w when w != 0, and 63+// (a harmless value, later overwritten) when w == 0. The unconditional writes+// can land up to K - 1 values past the last real one, so it stops while 64+// slots of slack remain in "out"; the caller finishes with the plain loop.+#define CROARING_DEFINE_EXTRACT_INTERSECTION_K(K)                             \+    static inline size_t extract_intersection_branchless_##K(                 \+        const uint64_t *__restrict__ words1,                                  \+        const uint64_t *__restrict__ words2, size_t length, uint16_t *out,    \+        size_t card, size_t *i, uint16_t *base) {                             \+        size_t pos = 0;                                                       \+        size_t j = *i;                                                        \+        uint16_t b = *base;                                                   \+        for (; j < length && pos + 64 <= card; ++j) {                         \+            /* Skip groups of four empty words: rare on random data, where */ \+            /* the branch is well predicted, but common on clustered data. */ \+            if (j + 4 <= length &&                                            \+                ((words1[j] & words2[j]) | (words1[j + 1] & words2[j + 1]) |  \+                 (words1[j + 2] & words2[j + 2]) |                            \+                 (words1[j + 3] & words2[j + 3])) == 0) {                     \+                j += 3;                                                       \+                b = (uint16_t)(b + 256);                                      \+                continue;                                                     \+            }                                                                 \+            uint64_t w = words1[j] & words2[j];                               \+            for (int k = 0; k < K; k++) {                                     \+                out[pos] = (uint16_t)(b + roaring_trailing_zeroes(            \+                                              w | (UINT64_C(1) << 63)));      \+                pos += (w != 0);                                              \+                w &= (w - 1);                                                 \+            }                                                                 \+            while (w != 0) {                                                  \+                out[pos++] = (uint16_t)(b + roaring_trailing_zeroes(w));      \+                w &= (w - 1);                                                 \+            }                                                                 \+            b = (uint16_t)(b + 64);                                           \+        }                                                                     \+        *i = j;                                                               \+        *base = b;                                                            \+        return pos;                                                           \+    }+CROARING_DEFINE_EXTRACT_INTERSECTION_K(1)+CROARING_DEFINE_EXTRACT_INTERSECTION_K(2)+CROARING_DEFINE_EXTRACT_INTERSECTION_K(4)+CROARING_DEFINE_EXTRACT_INTERSECTION_K(8)+#undef CROARING_DEFINE_EXTRACT_INTERSECTION_K++size_t bitset_extract_intersection_setbits_uint16_with_card(+    const uint64_t *__restrict__ words1, const uint64_t *__restrict__ words2,+    size_t length, uint16_t *out, size_t card, uint16_t base) {+    size_t i = 0;+    size_t pos;+    // Write about 2.5 times as many values per word as the average word+    // holds, so that words with more bits than that (handled by a branch)+    // are rare. Past 1.6 bits per word, writing 8 values costs more than the+    // occasional branch, so 4 values serve up to 3 bits per word. The+    // thresholds were measured on an Emerald Rapids Xeon.+    if (card * 5 < length * 2) {+        pos = extract_intersection_branchless_1(words1, words2, length, out,+                                                card, &i, &base);+    } else if (card * 5 < length * 4) {+        pos = extract_intersection_branchless_2(words1, words2, length, out,+                                                card, &i, &base);+    } else if (card < length * 3) {+        pos = extract_intersection_branchless_4(words1, words2, length, out,+                                                card, &i, &base);+    } else {+        pos = extract_intersection_branchless_8(words1, words2, length, out,+                                                card, &i, &base);+    }+    return pos + bitset_extract_intersection_setbits_uint16(+                     words1 + i, words2 + i, length - i, out + pos, base);+}++#if CROARING_IS_X64+/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out" as 16-bit integers, values start at "base" (can+ *be set to zero).+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ *set.+ *+ * Returns how many values were actually decoded.+ *+ * This function uses SSE decoding.+ */+CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+size_t bitset_extract_setbits_sse_uint16(const uint64_t *words, size_t length,+                                         uint16_t *out, size_t outcapacity,+                                         uint16_t base) {+    if (outcapacity == 0) return 0;+    uint16_t *initout = out;+    __m128i baseVec = _mm_set1_epi16(base - 1);+    __m128i incVec = _mm_set1_epi16(64);+    __m128i add8 = _mm_set1_epi16(8);+    uint16_t *safeout = out + outcapacity;+    const int numberofbytes = 2;  // process two bytes at a time+    size_t i = 0;+    for (; (i < length) && (out + numberofbytes * 8 <= safeout); ++i) {+        uint64_t w = words[i];+        if (w == 0) {+            baseVec = _mm_add_epi16(baseVec, incVec);+        } else {+            for (int k = 0; k < 4; ++k) {+                uint8_t byteA = (uint8_t)w;+                uint8_t byteB = (uint8_t)(w >> 8);+                w >>= 16;+                __m128i vecA = _mm_loadu_si128(+                    (const __m128i *)vecDecodeTable_uint16[byteA]);+                __m128i vecB = _mm_loadu_si128(+                    (const __m128i *)vecDecodeTable_uint16[byteB]);+                uint8_t advanceA = lengthTable[byteA];+                uint8_t advanceB = lengthTable[byteB];+                vecA = _mm_add_epi16(baseVec, vecA);+                baseVec = _mm_add_epi16(baseVec, add8);+                vecB = _mm_add_epi16(baseVec, vecB);+                baseVec = _mm_add_epi16(baseVec, add8);+                _mm_storeu_si128((__m128i *)out, vecA);+                out += advanceA;+                _mm_storeu_si128((__m128i *)out, vecB);+                out += advanceB;+            }+        }+    }+    base += (uint16_t)(i * 64);+    for (; (i < length) && (out < safeout); ++i) {+        uint64_t w = words[i];+        while ((w != 0) && (out < safeout)) {+            int r = roaring_trailing_zeroes(w);+            *out = (uint16_t)(r + base);+            out++;+            w &= (w - 1);+        }+        base += 64;+    }+    return out - initout;+}+CROARING_UNTARGET_AVX2+#endif++/*+ * Given a bitset containing "length" 64-bit words, write out the position+ * of all the set bits to "out", values start at "base" (can be set to zero).+ *+ * The "out" pointer should be sufficient to store the actual number of bits+ *set.+ *+ * Returns how many values were actually decoded.+ */+size_t bitset_extract_setbits_uint16(const uint64_t *words, size_t length,+                                     uint16_t *out, uint16_t base) {+    int outpos = 0;+    for (size_t i = 0; i < length; ++i) {+        uint64_t w = words[i];+        while (w != 0) {+            int r = roaring_trailing_zeroes(w);+            out[outpos++] = (uint16_t)(r + base);+            w &= (w - 1);+        }+        base += 64;+    }+    return outpos;+}++#if defined(CROARING_ASMBITMANIPOPTIMIZATION) && defined(CROARING_IS_X64)++static inline uint64_t _asm_bitset_set_list_withcard(uint64_t *words,+                                                     uint64_t card,+                                                     const uint16_t *list,+                                                     uint64_t length) {+    uint64_t offset, load, pos;+    uint64_t shift = 6;+    const uint16_t *end = list + length;+    if (!length) return card;+    // TODO: could unroll for performance, see bitset_set_list+    // bts is not available as an intrinsic in GCC+    __asm volatile(+        "1:\n"+        "movzwq (%[list]), %[pos]\n"+        "shrx %[shift], %[pos], %[offset]\n"+        "mov (%[words],%[offset],8), %[load]\n"+        "bts %[pos], %[load]\n"+        "mov %[load], (%[words],%[offset],8)\n"+        "sbb $-1, %[card]\n"+        "add $2, %[list]\n"+        "cmp %[list], %[end]\n"+        "jnz 1b"+        : [card] "+&r"(card), [list] "+&r"(list), [load] "=&r"(load),+          [pos] "=&r"(pos), [offset] "=&r"(offset)+        : [end] "r"(end), [words] "r"(words), [shift] "r"(shift));+    return card;+}++static inline void _asm_bitset_set_list(uint64_t *words, const uint16_t *list,+                                        uint64_t length) {+    uint64_t pos;+    const uint16_t *end = list + length;++    uint64_t shift = 6;+    uint64_t offset;+    uint64_t load;+    for (; list + 3 < end; list += 4) {+        pos = list[0];+        __asm volatile(+            "shrx %[shift], %[pos], %[offset]\n"+            "mov (%[words],%[offset],8), %[load]\n"+            "bts %[pos], %[load]\n"+            "mov %[load], (%[words],%[offset],8)"+            : [load] "=&r"(load), [offset] "=&r"(offset)+            : [words] "r"(words), [shift] "r"(shift), [pos] "r"(pos));+        pos = list[1];+        __asm volatile(+            "shrx %[shift], %[pos], %[offset]\n"+            "mov (%[words],%[offset],8), %[load]\n"+            "bts %[pos], %[load]\n"+            "mov %[load], (%[words],%[offset],8)"+            : [load] "=&r"(load), [offset] "=&r"(offset)+            : [words] "r"(words), [shift] "r"(shift), [pos] "r"(pos));+        pos = list[2];+        __asm volatile(+            "shrx %[shift], %[pos], %[offset]\n"+            "mov (%[words],%[offset],8), %[load]\n"+            "bts %[pos], %[load]\n"+            "mov %[load], (%[words],%[offset],8)"+            : [load] "=&r"(load), [offset] "=&r"(offset)+            : [words] "r"(words), [shift] "r"(shift), [pos] "r"(pos));+        pos = list[3];+        __asm volatile(+            "shrx %[shift], %[pos], %[offset]\n"+            "mov (%[words],%[offset],8), %[load]\n"+            "bts %[pos], %[load]\n"+            "mov %[load], (%[words],%[offset],8)"+            : [load] "=&r"(load), [offset] "=&r"(offset)+            : [words] "r"(words), [shift] "r"(shift), [pos] "r"(pos));+    }++    while (list != end) {+        pos = list[0];+        __asm volatile(+            "shrx %[shift], %[pos], %[offset]\n"+            "mov (%[words],%[offset],8), %[load]\n"+            "bts %[pos], %[load]\n"+            "mov %[load], (%[words],%[offset],8)"+            : [load] "=&r"(load), [offset] "=&r"(offset)+            : [words] "r"(words), [shift] "r"(shift), [pos] "r"(pos));+        list++;+    }+}++static inline uint64_t _asm_bitset_clear_list(uint64_t *words, uint64_t card,+                                              const uint16_t *list,+                                              uint64_t length) {+    uint64_t offset, load, pos;+    uint64_t shift = 6;+    const uint16_t *end = list + length;+    if (!length) return card;+    // btr is not available as an intrinsic in GCC+    __asm volatile(+        "1:\n"+        "movzwq (%[list]), %[pos]\n"+        "shrx %[shift], %[pos], %[offset]\n"+        "mov (%[words],%[offset],8), %[load]\n"+        "btr %[pos], %[load]\n"+        "mov %[load], (%[words],%[offset],8)\n"+        "sbb $0, %[card]\n"+        "add $2, %[list]\n"+        "cmp %[list], %[end]\n"+        "jnz 1b"+        : [card] "+&r"(card), [list] "+&r"(list), [load] "=&r"(load),+          [pos] "=&r"(pos), [offset] "=&r"(offset)+        : [end] "r"(end), [words] "r"(words), [shift] "r"(shift)+        :+        /* clobbers */ "memory");+    return card;+}++static inline uint64_t _scalar_bitset_clear_list(uint64_t *words, uint64_t card,+                                                 const uint16_t *list,+                                                 uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *(const uint16_t *)list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load & ~(UINT64_C(1) << index);+        card -= (load ^ newload) >> index;+        words[offset] = newload;+        list++;+    }+    return card;+}++static inline uint64_t _scalar_bitset_set_list_withcard(uint64_t *words,+                                                        uint64_t card,+                                                        const uint16_t *list,+                                                        uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load | (UINT64_C(1) << index);+        card += (load ^ newload) >> index;+        words[offset] = newload;+        list++;+    }+    return card;+}++static inline void _scalar_bitset_set_list(uint64_t *words,+                                           const uint16_t *list,+                                           uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load | (UINT64_C(1) << index);+        words[offset] = newload;+        list++;+    }+}++uint64_t bitset_clear_list(uint64_t *words, uint64_t card, const uint16_t *list,+                           uint64_t length) {+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+        return _asm_bitset_clear_list(words, card, list, length);+    } else {+        return _scalar_bitset_clear_list(words, card, list, length);+    }+}++uint64_t bitset_set_list_withcard(uint64_t *words, uint64_t card,+                                  const uint16_t *list, uint64_t length) {+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+        return _asm_bitset_set_list_withcard(words, card, list, length);+    } else {+        return _scalar_bitset_set_list_withcard(words, card, list, length);+    }+}++void bitset_set_list(uint64_t *words, const uint16_t *list, uint64_t length) {+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+        _asm_bitset_set_list(words, list, length);+    } else {+        _scalar_bitset_set_list(words, list, length);+    }+}+#else+uint64_t bitset_clear_list(uint64_t *words, uint64_t card, const uint16_t *list,+                           uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *(const uint16_t *)list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load & ~(UINT64_C(1) << index);+        card -= (load ^ newload) >> index;+        words[offset] = newload;+        list++;+    }+    return card;+}++uint64_t bitset_set_list_withcard(uint64_t *words, uint64_t card,+                                  const uint16_t *list, uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load | (UINT64_C(1) << index);+        card += (load ^ newload) >> index;+        words[offset] = newload;+        list++;+    }+    return card;+}++void bitset_set_list(uint64_t *words, const uint16_t *list, uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load | (UINT64_C(1) << index);+        words[offset] = newload;+        list++;+    }+}++#endif++/* flip specified bits */+/* TODO: consider whether worthwhile to make an asm version */++uint64_t bitset_flip_list_withcard(uint64_t *words, uint64_t card,+                                   const uint16_t *list, uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load ^ (UINT64_C(1) << index);+        // todo: is a branch here all that bad?+        card +=+            (1 - 2 * (((UINT64_C(1) << index) & load) >> index));  // +1 or -1+        words[offset] = newload;+        list++;+    }+    return card;+}++void bitset_flip_list(uint64_t *words, const uint16_t *list, uint64_t length) {+    uint64_t offset, load, newload, pos, index;+    const uint16_t *end = list + length;+    while (list != end) {+        pos = *list;+        offset = pos >> 6;+        index = pos % 64;+        load = words[offset];+        newload = load ^ (UINT64_C(1) << index);+        words[offset] = newload;+        list++;+    }+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace api {+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif
+ cbits/CRoaring/src/containers/array.c view
@@ -0,0 +1,635 @@+/*+ * array.c+ *+ */++#include <assert.h>+#include <stdio.h>+#include <stdlib.h>++#include <roaring/containers/array.h>+#include <roaring/memory.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++extern inline uint16_t array_container_minimum(const array_container_t *arr);+extern inline uint16_t array_container_maximum(const array_container_t *arr);+extern inline int array_container_index_equalorlarger(+    const array_container_t *arr, uint16_t x);++extern inline int array_container_rank(const array_container_t *arr,+                                       uint16_t x);+extern inline uint32_t array_container_rank_many(const array_container_t *arr,+                                                 uint64_t start_rank,+                                                 const uint32_t *begin,+                                                 const uint32_t *end,+                                                 uint64_t *ans);+extern inline int array_container_get_index(const array_container_t *arr,+                                            uint16_t x);+extern inline bool array_container_contains(const array_container_t *arr,+                                            uint16_t pos);+extern inline int array_container_cardinality(const array_container_t *array);+extern inline bool array_container_nonzero_cardinality(+    const array_container_t *array);+extern inline int32_t array_container_serialized_size_in_bytes(int32_t card);+extern inline bool array_container_empty(const array_container_t *array);+extern inline bool array_container_full(const array_container_t *array);++/* Create a new array with capacity size. Return NULL in case of failure. */+array_container_t *array_container_create_given_capacity(int32_t size) {+    array_container_t *container;++    if ((container = (array_container_t *)roaring_malloc(+             sizeof(array_container_t))) == NULL) {+        return NULL;+    }++    if (size <= 0) {  // we don't want to rely on malloc(0)+        container->array = NULL;+    } else if ((container->array = (uint16_t *)roaring_malloc(sizeof(uint16_t) *+                                                              size)) == NULL) {+        roaring_free(container);+        return NULL;+    }++    container->capacity = size;+    container->cardinality = 0;++    return container;+}++/* Create a new array. Return NULL in case of failure. */+array_container_t *array_container_create(void) {+    return array_container_create_given_capacity(ARRAY_DEFAULT_INIT_SIZE);+}++/* Create a new array containing all values in [min,max). */+array_container_t *array_container_create_range(uint32_t min, uint32_t max) {+    array_container_t *answer =+        array_container_create_given_capacity(max - min + 1);+    if (answer == NULL) return answer;+    answer->cardinality = 0;+    for (uint32_t k = min; k < max; k++) {+        answer->array[answer->cardinality++] = k;+    }+    return answer;+}++/* Duplicate container */+CROARING_ALLOW_UNALIGNED+array_container_t *array_container_clone(const array_container_t *src) {+    array_container_t *newcontainer =+        array_container_create_given_capacity(src->capacity);+    if (newcontainer == NULL) return NULL;++    newcontainer->cardinality = src->cardinality;++    memcpy(newcontainer->array, src->array,+           src->cardinality * sizeof(uint16_t));++    return newcontainer;+}++void array_container_offset(const array_container_t *c, container_t **loc,+                            container_t **hic, uint16_t offset) {+    array_container_t *lo = NULL, *hi = NULL;+    int top, lo_cap, hi_cap;++    top = (1 << 16) - offset;++    const uint16_t *array = c->array;+    lo_cap = count_less(array, c->cardinality, top);+    if (loc && lo_cap) {+        lo = array_container_create_given_capacity(lo_cap);+        uint16_t *out = lo->array;+        for (int i = 0; i < lo_cap; ++i) {+            out[i] = array[i] + offset;+        }+        lo->cardinality = lo_cap;+        *loc = (container_t *)lo;+    }++    hi_cap = c->cardinality - lo_cap;+    if (hic && hi_cap) {+        hi = array_container_create_given_capacity(hi_cap);+        uint16_t *out = hi->array;+        for (int i = 0; i < hi_cap; ++i) {+            out[i] = array[lo_cap + i] + offset;+        }+        hi->cardinality = hi_cap;+        *hic = (container_t *)hi;+    }+}++int array_container_shrink_to_fit(array_container_t *src) {+    int32_t cardinality = src->cardinality;+    int32_t capacity = src->capacity;+    if (cardinality == capacity) return 0;  // nothing to do+    bool shrunk;+    src->array = (uint16_t *)croaring_realloc_shrink(+        src->array, cardinality * sizeof(uint16_t), &shrunk);+    if (!shrunk) return 0;+    src->capacity = cardinality;+    return (capacity - cardinality) * (int)sizeof(uint16_t);+}++/* Free memory. */+void array_container_free(array_container_t *arr) {+    if (arr == NULL) return;+    roaring_free(arr->array);+    roaring_free(arr);+}++static inline int32_t grow_capacity(int32_t capacity) {+    return (capacity <= 0)   ? ARRAY_DEFAULT_INIT_SIZE+           : capacity < 64   ? capacity * 2+           : capacity < 1024 ? capacity * 3 / 2+                             : capacity * 5 / 4;+}++static inline int32_t clamp(int32_t val, int32_t min, int32_t max) {+    return ((val < min) ? min : (val > max) ? max : val);+}++void array_container_grow(array_container_t *container, int32_t min,+                          bool preserve) {+    int32_t max = (min <= DEFAULT_MAX_SIZE ? DEFAULT_MAX_SIZE : 65536);+    int32_t new_capacity = clamp(grow_capacity(container->capacity), min, max);++    container->capacity = new_capacity;+    uint16_t *array = container->array;++    if (preserve) {+        container->array =+            (uint16_t *)roaring_realloc(array, new_capacity * sizeof(uint16_t));+        if (container->array == NULL) roaring_free(array);+    } else {+        roaring_free(array);+        container->array =+            (uint16_t *)roaring_malloc(new_capacity * sizeof(uint16_t));+    }++    // if realloc fails, we have container->array == NULL.+}++/* Copy one container into another. We assume that they are distinct. */+void array_container_copy(const array_container_t *src,+                          array_container_t *dst) {+    const int32_t cardinality = src->cardinality;+    if (cardinality > dst->capacity) {+        array_container_grow(dst, cardinality, false);+    }++    dst->cardinality = cardinality;+    memcpy(dst->array, src->array, cardinality * sizeof(uint16_t));+}++void array_container_add_from_range(array_container_t *arr, uint32_t min,+                                    uint32_t max, uint16_t step) {+    for (uint32_t value = min; value < max; value += step) {+        array_container_append(arr, value);+    }+}++/* Computes the union of array1 and array2 and write the result to arrayout.+ * It is assumed that arrayout is distinct from both array1 and array2.+ */+void array_container_union(const array_container_t *array_1,+                           const array_container_t *array_2,+                           array_container_t *out) {+    const int32_t card_1 = array_1->cardinality, card_2 = array_2->cardinality;+    const int32_t max_cardinality = card_1 + card_2;++    if (out->capacity < max_cardinality) {+        array_container_grow(out, max_cardinality, false);+    }+    out->cardinality = (int32_t)fast_union_uint16(+        array_1->array, card_1, array_2->array, card_2, out->array);+}++/* Computes the  difference of array1 and array2 and write the result+ * to array out.+ * Array out does not need to be distinct from array_1+ */+void array_container_andnot(const array_container_t *array_1,+                            const array_container_t *array_2,+                            array_container_t *out) {+    if (out->capacity < array_1->cardinality)+        array_container_grow(out, array_1->cardinality, false);+#if CROARING_IS_X64+    // difference_vector16 only vectorizes while 8 values remain on each+    // side, and loses to the galloping scalar code when one side is+    // much shorter.+    const int32_t card_1 = array_1->cardinality, card_2 = array_2->cardinality;+    const int32_t card_min = card_1 < card_2 ? card_1 : card_2;+    const int32_t card_max = card_1 < card_2 ? card_2 : card_1;+    const bool skewed = card_min < 8 || card_min * 128 < card_max;+    if (!skewed && (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) &&+        (out != array_1) && (out != array_2)) {+        out->cardinality = difference_vector16(+            array_1->array, array_1->cardinality, array_2->array,+            array_2->cardinality, out->array);+    } else {+        out->cardinality =+            difference_uint16(array_1->array, array_1->cardinality,+                              array_2->array, array_2->cardinality, out->array);+    }+#else+    out->cardinality =+        difference_uint16(array_1->array, array_1->cardinality, array_2->array,+                          array_2->cardinality, out->array);+#endif+}++/* Computes the symmetric difference of array1 and array2 and write the+ * result+ * to arrayout.+ * It is assumed that arrayout is distinct from both array1 and array2.+ */+void array_container_xor(const array_container_t *array_1,+                         const array_container_t *array_2,+                         array_container_t *out) {+    const int32_t card_1 = array_1->cardinality, card_2 = array_2->cardinality;+    const int32_t max_cardinality = card_1 + card_2;+    if (out->capacity < max_cardinality) {+        array_container_grow(out, max_cardinality, false);+    }++#if CROARING_IS_X64+    if (!(card_1 * CROARING_SKEWED_VECTOR_MERGE_RATIO < card_2 ||+          card_2 * CROARING_SKEWED_VECTOR_MERGE_RATIO < card_1) &&+        (croaring_hardware_support() & ROARING_SUPPORTS_AVX2)) {+        out->cardinality =+            xor_vector16(array_1->array, array_1->cardinality, array_2->array,+                         array_2->cardinality, out->array);+    } else {+        out->cardinality =+            xor_uint16(array_1->array, array_1->cardinality, array_2->array,+                       array_2->cardinality, out->array);+    }+#else+    out->cardinality =+        xor_uint16(array_1->array, array_1->cardinality, array_2->array,+                   array_2->cardinality, out->array);+#endif+}++static inline int32_t minimum_int32(int32_t a, int32_t b) {+    return (a < b) ? a : b;+}++/* computes the intersection of array1 and array2 and write the result to+ * arrayout.+ * It is assumed that arrayout is distinct from both array1 and array2.+ * */+void array_container_intersection(const array_container_t *array1,+                                  const array_container_t *array2,+                                  array_container_t *out) {+    int32_t card_1 = array1->cardinality, card_2 = array2->cardinality,+            min_card = minimum_int32(card_1, card_2);+    const int threshold = 64;  // subject to tuning+#if CROARING_IS_X64+    if (out->capacity < min_card) {+        array_container_grow(out, min_card + sizeof(__m128i) / sizeof(uint16_t),+                             false);+    }+#else+    if (out->capacity < min_card) {+        array_container_grow(out, min_card, false);+    }+#endif++    if (card_1 * threshold < card_2) {+        out->cardinality = intersect_skewed_uint16(+            array1->array, card_1, array2->array, card_2, out->array);+    } else if (card_2 * threshold < card_1) {+        out->cardinality = intersect_skewed_uint16(+            array2->array, card_2, array1->array, card_1, out->array);+    } else {+#if CROARING_IS_X64+        if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+            out->cardinality = intersect_vector16(+                array1->array, card_1, array2->array, card_2, out->array);+        } else {+            out->cardinality = intersect_uint16(+                array1->array, card_1, array2->array, card_2, out->array);+        }+#else+        out->cardinality = intersect_uint16(array1->array, card_1,+                                            array2->array, card_2, out->array);+#endif+    }+}++/* computes the size of the intersection of array1 and array2+ * */+int array_container_intersection_cardinality(const array_container_t *array1,+                                             const array_container_t *array2) {+    int32_t card_1 = array1->cardinality, card_2 = array2->cardinality;+    const int threshold = 64;  // subject to tuning+    if (card_1 * threshold < card_2) {+        return intersect_skewed_uint16_cardinality(array1->array, card_1,+                                                   array2->array, card_2);+    } else if (card_2 * threshold < card_1) {+        return intersect_skewed_uint16_cardinality(array2->array, card_2,+                                                   array1->array, card_1);+    } else {+#if CROARING_IS_X64+        if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+            return intersect_vector16_cardinality(array1->array, card_1,+                                                  array2->array, card_2);+        } else {+            return intersect_uint16_cardinality(array1->array, card_1,+                                                array2->array, card_2);+        }+#else+        return intersect_uint16_cardinality(array1->array, card_1,+                                            array2->array, card_2);+#endif+    }+}++bool array_container_intersect(const array_container_t *array1,+                               const array_container_t *array2) {+    int32_t card_1 = array1->cardinality, card_2 = array2->cardinality;+    const int threshold = 64;  // subject to tuning+    if (card_1 * threshold < card_2) {+        return intersect_skewed_uint16_nonempty(array1->array, card_1,+                                                array2->array, card_2);+    } else if (card_2 * threshold < card_1) {+        return intersect_skewed_uint16_nonempty(array2->array, card_2,+                                                array1->array, card_1);+    } else {+        // we do not bother vectorizing+        return intersect_uint16_nonempty(array1->array, card_1, array2->array,+                                         card_2);+    }+}++/* computes the intersection of array1 and array2 and write the result to+ * array1.+ * */+void array_container_intersection_inplace(array_container_t *src_1,+                                          const array_container_t *src_2) {+    int32_t card_1 = src_1->cardinality, card_2 = src_2->cardinality;+    const int threshold = 64;  // subject to tuning+    if (card_1 * threshold < card_2) {+        src_1->cardinality = intersect_skewed_uint16(+            src_1->array, card_1, src_2->array, card_2, src_1->array);+    } else if (card_2 * threshold < card_1) {+        src_1->cardinality = intersect_skewed_uint16(+            src_2->array, card_2, src_1->array, card_1, src_1->array);+    } else {+#if CROARING_IS_X64+        if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+            src_1->cardinality = intersect_vector16_inplace(+                src_1->array, card_1, src_2->array, card_2);+        } else {+            src_1->cardinality = intersect_uint16(+                src_1->array, card_1, src_2->array, card_2, src_1->array);+        }+#else+        src_1->cardinality = intersect_uint16(+            src_1->array, card_1, src_2->array, card_2, src_1->array);+#endif+    }+}++/* Constant n: one or two vector operations on every compiler. */+CROARING_ALLOW_UNALIGNED+static inline void array_put(uint32_t *out, const uint16_t *array,+                             uint32_t base, size_t n) {+    for (size_t k = 0; k < n; ++k) {+        out[k] = base + array[k];+    }+}++CROARING_ALLOW_UNALIGNED+int array_container_to_uint32_array(void *vout, const array_container_t *cont,+                                    uint32_t base) {+#if CROARING_IS_X64+    int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX512) {+        return avx512_array_container_to_uint32_array(vout, cont->array,+                                                      cont->cardinality, base);+    }+#endif+    if (support & ROARING_SUPPORTS_AVX2) {+        return array_container_to_uint32_array_vector16(+            vout, cont->array, cont->cardinality, base);+    }+#endif  // CROARING_IS_X64+    const size_t cardinality = (size_t)cont->cardinality;+    const uint16_t *array = cont->array;+    // Typed stores vectorize on every compiler; all callers but+    // roaring_bitmap_serialize pass an aligned vout.+    if (cardinality >= 4 && ((uintptr_t)vout & 3) == 0) {+        uint32_t *out = (uint32_t *)vout;+        uint32_t *out_end = out + cardinality;+        const uint16_t *array_end = array + cardinality;+        if (cardinality < 8) {  // overlapping writes end at cardinality+            array_put(out, array, base, 4);+            array_put(out_end - 4, array_end - 4, base, 4);+            return (int)cardinality;+        }+        const size_t full = cardinality & ~(size_t)15;+        for (size_t i = 0; i < full; i += 16) {+            array_put(out + i, array + i, base, 16);+        }+        if (cardinality - full > 8) {+            array_put(out + full, array + full, base, 8);+        }+        if (full != cardinality) {+            array_put(out_end - 8, array_end - 8, base, 8);+        }+        return (int)cardinality;+    }+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;+    size_t i = 0;+    for (; i < (size_t)cont->cardinality; ++i) {+        const uint32_t val = base + cont->array[i];+        memcpy(out + outpos, &val,+               sizeof(uint32_t));  // should be compiled as a MOV on x64+        outpos++;+    }+    return outpos;+}++void array_container_printf(const array_container_t *v) {+    if (v->cardinality == 0) {+        printf("{}");+        return;+    }+    printf("{");+    printf("%d", v->array[0]);+    for (int i = 1; i < v->cardinality; ++i) {+        printf(",%d", v->array[i]);+    }+    printf("}");+}++void array_container_printf_as_uint32_array(const array_container_t *v,+                                            uint32_t base) {+    if (v->cardinality == 0) {+        return;+    }+    printf("%u", v->array[0] + base);+    for (int i = 1; i < v->cardinality; ++i) {+        printf(",%u", v->array[i] + base);+    }+}++/*+ * Validate the container. Returns true if valid.+ */+CROARING_ALLOW_UNALIGNED+bool array_container_validate(const array_container_t *v, const char **reason) {+    if (v->capacity < 0) {+        *reason = "negative capacity";+        return false;+    }+    if (v->cardinality < 0) {+        *reason = "negative cardinality";+        return false;+    }+    if (v->cardinality > v->capacity) {+        *reason = "cardinality exceeds capacity";+        return false;+    }+    if (v->cardinality > DEFAULT_MAX_SIZE) {+        *reason = "cardinality exceeds DEFAULT_MAX_SIZE";+        return false;+    }+    if (v->cardinality == 0) {+        *reason = "zero cardinality";+        return false;+    }++    if (v->array == NULL) {+        *reason = "NULL array pointer";+        return false;+    }+    uint16_t prev = v->array[0];+    for (int i = 1; i < v->cardinality; ++i) {+        if (v->array[i] <= prev) {+            *reason = "array elements not strictly increasing";+            return false;+        }+        prev = v->array[i];+    }++    return true;+}++/* Compute the number of runs */+int32_t array_container_number_of_runs(const array_container_t *ac) {+    const uint16_t *array = ac->array;+    const int32_t card = ac->cardinality;+    if (card == 0) return 0;+    // Every value starts a run unless it follows its predecessor. The values+    // are strictly increasing, so array[i - 1] + 1 cannot wrap around to a+    // match. The loop has no branch on the data and compilers vectorize it.+    int32_t continuing = 0;+    for (int32_t i = 1; i < card; ++i) {+        continuing += (array[i] == (uint16_t)(array[i - 1] + 1));+    }+    return card - continuing;+}++/**+ * Writes the underlying array to buf, outputs how many bytes were written.+ * The number of bytes written should be+ * array_container_size_in_bytes(container).+ *+ */+CROARING_ALLOW_UNALIGNED+int32_t array_container_write(const array_container_t *container, char *buf) {+#if CROARING_IS_BIG_ENDIAN+    for (int32_t i = 0; i < container->cardinality; ++i) {+        uint16_t v_le = croaring_htole16(container->array[i]);+        memcpy(buf + i * sizeof(uint16_t), &v_le, sizeof(uint16_t));+    }+#else+    memcpy(buf, container->array, container->cardinality * sizeof(uint16_t));+#endif+    return array_container_size_in_bytes(container);+}++CROARING_ALLOW_UNALIGNED+bool array_container_is_subset(const array_container_t *container1,+                               const array_container_t *container2) {+    if (container1->cardinality > container2->cardinality) {+        return false;+    }+    int i1 = 0, i2 = 0;+    while (i1 < container1->cardinality && i2 < container2->cardinality) {+        if (container1->array[i1] == container2->array[i2]) {+            i1++;+            i2++;+        } else if (container1->array[i1] > container2->array[i2]) {+            i2++;+        } else {  // container1->array[i1] < container2->array[i2]+            return false;+        }+    }+    if (i1 == container1->cardinality) {+        return true;+    } else {+        return false;+    }+}++int32_t array_container_read(int32_t cardinality, array_container_t *container,+                             const char *buf) {+    if (container->capacity < cardinality) {+        array_container_grow(container, cardinality, false);+    }+    container->cardinality = cardinality;+#if CROARING_IS_BIG_ENDIAN+    for (int32_t i = 0; i < cardinality; ++i) {+        uint16_t v_le;+        memcpy(&v_le, buf + i * sizeof(uint16_t), sizeof(uint16_t));+        container->array[i] = croaring_letoh16(v_le);+    }+#else+    memcpy(container->array, buf, container->cardinality * sizeof(uint16_t));+#endif++    return array_container_size_in_bytes(container);+}++CROARING_ALLOW_UNALIGNED+bool array_container_iterate(const array_container_t *cont, uint32_t base,+                             roaring_iterator iterator, void *ptr) {+    for (int i = 0; i < cont->cardinality; i++)+        if (!iterator(cont->array[i] + base, ptr)) return false;+    return true;+}++CROARING_ALLOW_UNALIGNED+bool array_container_iterate64(const array_container_t *cont, uint32_t base,+                               roaring_iterator64 iterator, uint64_t high_bits,+                               void *ptr) {+    for (int i = 0; i < cont->cardinality; i++)+        if (!iterator(high_bits | (uint64_t)(cont->array[i] + base), ptr))+            return false;+    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/bitset.c view
@@ -0,0 +1,1396 @@+/*+ * bitset.c+ *+ */+#include <assert.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/bitset_util.h>+#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/memory.h>+#include <roaring/portability.h>+#include <roaring/utilasm.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif++#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++extern inline int bitset_container_cardinality(+    const bitset_container_t *bitset);+extern inline void bitset_container_set(bitset_container_t *bitset,+                                        uint16_t pos);+// unused at this time:+// extern inline void bitset_container_unset(bitset_container_t *bitset,+// uint16_t pos);+extern inline bool bitset_container_get(const bitset_container_t *bitset,+                                        uint16_t pos);+extern inline int32_t bitset_container_serialized_size_in_bytes(void);+extern inline bool bitset_container_add(bitset_container_t *bitset,+                                        uint16_t pos);+extern inline bool bitset_container_remove(bitset_container_t *bitset,+                                           uint16_t pos);+extern inline bool bitset_container_contains(const bitset_container_t *bitset,+                                             uint16_t pos);++void bitset_container_clear(bitset_container_t *bitset) {+    memset(bitset->words, 0, sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+    bitset->cardinality = 0;+}++void bitset_container_set_all(bitset_container_t *bitset) {+    memset(bitset->words, INT64_C(-1),+           sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+    bitset->cardinality = (1 << 16);+}++static bitset_container_t *bitset_container_allocate(void) {+    bitset_container_t *bitset =+        (bitset_container_t *)roaring_malloc(sizeof(bitset_container_t));++    if (!bitset) {+        return NULL;+    }++    size_t align_size = 32;+#if CROARING_IS_X64+    int support = croaring_hardware_support();+    if (support & ROARING_SUPPORTS_AVX512) {+        // sizeof(__m512i) == 64+        align_size = 64;+    } else {+        // sizeof(__m256i) == 32+        align_size = 32;+    }+#endif+    bitset->words = (uint64_t *)roaring_aligned_malloc(+        align_size, sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+    if (!bitset->words) {+        roaring_free(bitset);+        return NULL;+    }+    return bitset;+}++/* Create a new bitset. Return NULL in case of failure. */+bitset_container_t *bitset_container_create(void) {+    bitset_container_t *bitset = bitset_container_allocate();+    if (bitset) {+        bitset_container_clear(bitset);+    }+    return bitset;+}++bitset_container_t *bitset_container_create_uninitialized(void) {+    bitset_container_t *bitset = bitset_container_allocate();+    if (bitset) {+        bitset->cardinality = 0;+    }+    return bitset;+}++/* Copy one container into another. We assume that they are distinct. */+void bitset_container_copy(const bitset_container_t *source,+                           bitset_container_t *dest) {+    dest->cardinality = source->cardinality;+    memcpy(dest->words, source->words,+           sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+}++void bitset_container_add_from_range(bitset_container_t *bitset, uint32_t min,+                                     uint32_t max, uint16_t step) {+    if (step == 0) return;   // refuse to crash+    if ((64 % step) == 0) {  // step divides 64+        uint64_t mask = 0;   // construct the repeated mask+        for (uint32_t value = (min % step); value < 64; value += step) {+            mask |= ((uint64_t)1 << value);+        }+        uint32_t firstword = min / 64;+        uint32_t endword = (max - 1) / 64;+        bitset->cardinality = (max - min + step - 1) / step;+        if (firstword == endword) {+            bitset->words[firstword] |=+                mask & (((~UINT64_C(0)) << (min % 64)) &+                        ((~UINT64_C(0)) >> ((~max + 1) % 64)));+            return;+        }+        bitset->words[firstword] = mask & ((~UINT64_C(0)) << (min % 64));+        for (uint32_t i = firstword + 1; i < endword; i++)+            bitset->words[i] = mask;+        bitset->words[endword] = mask & ((~UINT64_C(0)) >> ((~max + 1) % 64));+    } else {+        for (uint32_t value = min; value < max; value += step) {+            bitset_container_add(bitset, value);+        }+    }+}++/* Free memory. */+void bitset_container_free(bitset_container_t *bitset) {+    if (bitset == NULL) return;+    roaring_aligned_free(bitset->words);+    roaring_free(bitset);+}++/* duplicate container. */+CROARING_ALLOW_UNALIGNED+bitset_container_t *bitset_container_clone(const bitset_container_t *src) {+    bitset_container_t *bitset =+        (bitset_container_t *)roaring_malloc(sizeof(bitset_container_t));++    if (!bitset) {+        return NULL;+    }++    size_t align_size = 32;+#if CROARING_IS_X64+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX512) {+        // sizeof(__m512i) == 64+        align_size = 64;+    } else {+        // sizeof(__m256i) == 32+        align_size = 32;+    }+#endif+    bitset->words = (uint64_t *)roaring_aligned_malloc(+        align_size, sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+    if (!bitset->words) {+        roaring_free(bitset);+        return NULL;+    }+    bitset->cardinality = src->cardinality;+    memcpy(bitset->words, src->words,+           sizeof(uint64_t) * BITSET_CONTAINER_SIZE_IN_WORDS);+    return bitset;+}++void bitset_container_offset(const bitset_container_t *c, container_t **loc,+                             container_t **hic, uint16_t offset) {+    bitset_container_t *bc = NULL;+    uint64_t val;+    uint16_t b, i, end;++    b = offset >> 6;+    i = offset % 64;+    end = 1024 - b;++    const uint64_t *words = c->words;+    if (loc != NULL) {+        bc = bitset_container_create();+        uint64_t *out = bc->words + b;+        if (i == 0) {+            memcpy(out, words, 8 * end);+        } else {+            out[0] = words[0] << i;+            for (uint32_t k = 1; k < end; ++k) {+                val = words[k] << i;+                val |= words[k - 1] >> (64 - i);+                out[k] = val;+            }+        }++        bc->cardinality = bitset_container_compute_cardinality(bc);+        if (bc->cardinality != 0) {+            *loc = bc;+        }+        if (bc->cardinality == c->cardinality) {+            return;+        }+    }++    if (hic == NULL) {+        // Both hic and loc can't be NULL, so bc is never NULL here+        if (bc->cardinality == 0) {+            bitset_container_free(bc);+        }+        return;+    }++    if (bc == NULL || bc->cardinality != 0) {+        bc = bitset_container_create();+    }++    uint64_t *out = bc->words;+    if (i == 0) {+        memcpy(out, words + end, 8 * b);+    } else {+        for (uint32_t k = end; k < 1024; ++k) {+            val = words[k] << i;+            val |= words[k - 1] >> (64 - i);+            out[k - end] = val;+        }+        out[b] = words[1023] >> (64 - i);+    }++    bc->cardinality = bitset_container_compute_cardinality(bc);+    if (bc->cardinality == 0) {+        bitset_container_free(bc);+        return;+    }+    *hic = bc;+}++void bitset_container_set_range(bitset_container_t *bitset, uint32_t begin,+                                uint32_t end) {+    bitset_set_range(bitset->words, begin, end);+    bitset->cardinality =+        bitset_container_compute_cardinality(bitset);  // could be smarter+}++CROARING_ALLOW_UNALIGNED+bool bitset_container_intersect(const bitset_container_t *src_1,+                                const bitset_container_t *src_2) {+    // could vectorize, but this is probably already quite fast in practice+    const uint64_t *__restrict__ words_1 = src_1->words;+    const uint64_t *__restrict__ words_2 = src_2->words;+    for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) {+        if ((words_1[i] & words_2[i]) != 0) return true;+    }+    return false;+}++#if CROARING_IS_X64+#ifndef CROARING_WORDS_IN_AVX2_REG+#define CROARING_WORDS_IN_AVX2_REG sizeof(__m256i) / sizeof(uint64_t)+#endif+#ifndef WORDS_IN_AVX512_REG+#define WORDS_IN_AVX512_REG sizeof(__m512i) / sizeof(uint64_t)+#endif+/* Get the number of bits set (force computation) */+CROARING_ALLOW_UNALIGNED+static inline int _scalar_bitset_container_compute_cardinality(+    const bitset_container_t *bitset) {+    const uint64_t *words = bitset->words;+    int32_t sum = 0;+    for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 4) {+        sum += roaring_hamming(words[i]);+        sum += roaring_hamming(words[i + 1]);+        sum += roaring_hamming(words[i + 2]);+        sum += roaring_hamming(words[i + 3]);+    }+    return sum;+}+/* Get the number of bits set (force computation) */+int bitset_container_compute_cardinality(const bitset_container_t *bitset) {+    int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX512) {+        return (int)avx512_vpopcount(+            (const __m512i *)bitset->words,+            BITSET_CONTAINER_SIZE_IN_WORDS / (WORDS_IN_AVX512_REG));+    } else+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+        if (support & ROARING_SUPPORTS_AVX2) {+            return (int)avx2_harley_seal_popcount256(+                (const __m256i *)bitset->words,+                BITSET_CONTAINER_SIZE_IN_WORDS / (CROARING_WORDS_IN_AVX2_REG));+        } else {+            return _scalar_bitset_container_compute_cardinality(bitset);+        }+}++#elif defined(CROARING_USENEON)+int bitset_container_compute_cardinality(const bitset_container_t *bitset) {+    uint16x8_t n0 = vdupq_n_u16(0);+    uint16x8_t n1 = vdupq_n_u16(0);+    uint16x8_t n2 = vdupq_n_u16(0);+    uint16x8_t n3 = vdupq_n_u16(0);+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 8) {+        uint64x2_t c0 = vld1q_u64(&bitset->words[i + 0]);+        n0 = vaddq_u16(n0, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c0))));+        uint64x2_t c1 = vld1q_u64(&bitset->words[i + 2]);+        n1 = vaddq_u16(n1, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c1))));+        uint64x2_t c2 = vld1q_u64(&bitset->words[i + 4]);+        n2 = vaddq_u16(n2, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c2))));+        uint64x2_t c3 = vld1q_u64(&bitset->words[i + 6]);+        n3 = vaddq_u16(n3, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c3))));+    }+    uint64x2_t n = vdupq_n_u64(0);+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n0)));+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n1)));+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n2)));+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n3)));+    return vgetq_lane_u64(n, 0) + vgetq_lane_u64(n, 1);+}++#else  // CROARING_IS_X64++/* Get the number of bits set (force computation) */+int bitset_container_compute_cardinality(const bitset_container_t *bitset) {+    const uint64_t *words = bitset->words;+    int32_t sum = 0;+    for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 4) {+        sum += roaring_hamming(words[i]);+        sum += roaring_hamming(words[i + 1]);+        sum += roaring_hamming(words[i + 2]);+        sum += roaring_hamming(words[i + 3]);+    }+    return sum;+}++#endif  // CROARING_IS_X64++#if CROARING_IS_X64++#define CROARING_BITSET_CONTAINER_FN_REPEAT 8+#ifndef WORDS_IN_AVX512_REG+#define WORDS_IN_AVX512_REG sizeof(__m512i) / sizeof(uint64_t)+#endif  // WORDS_IN_AVX512_REG++/* Computes a binary operation (eg union) on bitset1 and bitset2 and write the+   result to bitsetout */+// clang-format off+#define CROARING_AVX512_BITSET_CONTAINER_FN1(before, opname, opsymbol, avx_intrinsic,   \+                                neon_intrinsic, after)                         \+  static inline int _avx512_bitset_container_##opname##_nocard(                \+      const bitset_container_t *src_1, const bitset_container_t *src_2,        \+      bitset_container_t *dst) {                                               \+    const uint8_t * __restrict__ words_1 = (const uint8_t *)src_1->words;      \+    const uint8_t * __restrict__ words_2 = (const uint8_t *)src_2->words;      \+    /* not using the blocking optimization for some reason*/                   \+    uint8_t *out = (uint8_t*)dst->words;                                       \+    const int innerloop = 8;                                                   \+    for (size_t i = 0;                                                         \+        i < BITSET_CONTAINER_SIZE_IN_WORDS / (WORDS_IN_AVX512_REG);            \+                                                         i+=innerloop) {       \+        __m512i A1, A2, AO;                                                    \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1));                   \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2));                   \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)out, AO);                               \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 64));              \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 64));              \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+64), AO);                          \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 128));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 128));             \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+128), AO);                         \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 192));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 192));             \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+192), AO);                         \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 256));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 256));             \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+256), AO);                         \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 320));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 320));             \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+320), AO);                         \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 384));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 384));             \+        AO = avx_intrinsic(A2, A1);                                            \+        _mm512_storeu_si512((__m512i *)(out+384), AO);                         \+        A1 = _mm512_loadu_si512((const __m512i *)(words_1 + 448));             \+        A2 = _mm512_loadu_si512((const __m512i *)(words_2 + 448));             \+        AO = avx_intrinsic(A2, A1);                                     \+        _mm512_storeu_si512((__m512i *)(out+448), AO);                  \+        out+=512;                                                       \+        words_1 += 512;                                                 \+        words_2 += 512;                                                 \+    }                                                                   \+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;                      \+    return dst->cardinality;                                            \+  }++#define CROARING_AVX512_BITSET_CONTAINER_FN2(before, opname, opsymbol, avx_intrinsic,           \+                                neon_intrinsic, after)                                 \+  /* next, a version that updates cardinality*/                                        \+  static inline int _avx512_bitset_container_##opname(const bitset_container_t *src_1, \+                                      const bitset_container_t *src_2,                 \+                                      bitset_container_t *dst) {                       \+    const __m512i * __restrict__ words_1 = (const __m512i *) src_1->words;             \+    const __m512i * __restrict__ words_2 = (const __m512i *) src_2->words;             \+    __m512i *out = (__m512i *) dst->words;                                             \+    dst->cardinality = (int32_t)avx512_harley_seal_popcount512andstore_##opname(words_2,\+				words_1, out,BITSET_CONTAINER_SIZE_IN_WORDS / (WORDS_IN_AVX512_REG));           \+    return dst->cardinality;                                                            \+  }++#define CROARING_AVX512_BITSET_CONTAINER_FN3(before, opname, opsymbol, avx_intrinsic,            \+                                neon_intrinsic, after)                                  \+  /* next, a version that just computes the cardinality*/                               \+  static inline int _avx512_bitset_container_##opname##_justcard(                       \+      const bitset_container_t *src_1, const bitset_container_t *src_2) {               \+    const __m512i * __restrict__ data1 = (const __m512i *) src_1->words;                \+    const __m512i * __restrict__ data2 = (const __m512i *) src_2->words;                \+    return (int)avx512_harley_seal_popcount512_##opname(data2,                          \+				data1, BITSET_CONTAINER_SIZE_IN_WORDS / (WORDS_IN_AVX512_REG));                 \+  }+++// we duplicate the function because other containers use the "or" term, makes API more consistent+#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, or,    |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, union, |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, and,          &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, intersection, &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, xor,    ^,  _mm512_xor_si512,    veorq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX512, andnot, &~, _mm512_andnot_si512, vbicq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, or,    |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, union, |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, and,          &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, intersection, &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, xor,    ^,  _mm512_xor_si512,    veorq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX512, andnot, &~, _mm512_andnot_si512, vbicq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, or,    |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, union, |, _mm512_or_si512, vorrq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, and,          &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, intersection, &, _mm512_and_si512, vandq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512++CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, xor,    ^,  _mm512_xor_si512,    veorq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+CROARING_TARGET_AVX512+CROARING_AVX512_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX512, andnot, &~, _mm512_andnot_si512, vbicq_u64, CROARING_UNTARGET_AVX512)+CROARING_UNTARGET_AVX512+#endif // CROARING_COMPILER_SUPPORTS_AVX512++#ifndef CROARING_WORDS_IN_AVX2_REG+#define CROARING_WORDS_IN_AVX2_REG sizeof(__m256i) / sizeof(uint64_t)+#endif // CROARING_WORDS_IN_AVX2_REG+#define CROARING_LOOP_SIZE                    \+    BITSET_CONTAINER_SIZE_IN_WORDS / \+        ((CROARING_WORDS_IN_AVX2_REG)*CROARING_BITSET_CONTAINER_FN_REPEAT)++/* Computes a binary operation (eg union) on bitset1 and bitset2 and write the+   result to bitsetout */+// clang-format off+#define CROARING_AVX_BITSET_CONTAINER_FN1(before, opname, opsymbol, avx_intrinsic,               \+                                neon_intrinsic, after)                                \+  static inline int _avx2_bitset_container_##opname##_nocard(                                \+      const bitset_container_t *src_1, const bitset_container_t *src_2,        \+      bitset_container_t *dst) {                                               \+    const uint8_t *__restrict__ words_1 = (const uint8_t *)src_1->words;       \+    const uint8_t *__restrict__ words_2 = (const uint8_t *)src_2->words;       \+    /* not using the blocking optimization for some reason*/                   \+    uint8_t *out = (uint8_t *)dst->words;                                      \+    const int innerloop = 8;                                                   \+    for (size_t i = 0;                                                         \+         i < BITSET_CONTAINER_SIZE_IN_WORDS / (CROARING_WORDS_IN_AVX2_REG);             \+         i += innerloop) {                                                     \+      __m256i A1, A2, AO;                                                      \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1));                     \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2));                     \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)out, AO);                                 \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 32));                \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 32));                \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 32), AO);                          \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 64));                \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 64));                \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 64), AO);                          \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 96));                \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 96));                \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 96), AO);                          \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 128));               \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 128));               \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 128), AO);                         \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 160));               \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 160));               \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 160), AO);                         \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 192));               \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 192));               \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 192), AO);                         \+      A1 = _mm256_loadu_si256((const __m256i *)(words_1 + 224));               \+      A2 = _mm256_loadu_si256((const __m256i *)(words_2 + 224));               \+      AO = avx_intrinsic(A2, A1);                                              \+      _mm256_storeu_si256((__m256i *)(out + 224), AO);                         \+      out += 256;                                                              \+      words_1 += 256;                                                          \+      words_2 += 256;                                                          \+    }                                                                          \+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;                             \+    return dst->cardinality;                                                   \+  }++#define CROARING_AVX_BITSET_CONTAINER_FN2(before, opname, opsymbol, avx_intrinsic,               \+                                neon_intrinsic, after)                                \+  /* next, a version that updates cardinality*/                                \+  static inline int _avx2_bitset_container_##opname(const bitset_container_t *src_1,         \+                                      const bitset_container_t *src_2,         \+                                      bitset_container_t *dst) {               \+    const __m256i *__restrict__ words_1 = (const __m256i *)src_1->words;       \+    const __m256i *__restrict__ words_2 = (const __m256i *)src_2->words;       \+    __m256i *out = (__m256i *)dst->words;                                      \+    dst->cardinality = (int32_t)avx2_harley_seal_popcount256andstore_##opname( \+        words_2, words_1, out,                                                 \+        BITSET_CONTAINER_SIZE_IN_WORDS / (CROARING_WORDS_IN_AVX2_REG));                 \+    return dst->cardinality;                                                   \+  }                                                                            \++#define CROARING_AVX_BITSET_CONTAINER_FN3(before, opname, opsymbol, avx_intrinsic,               \+                                neon_intrinsic, after)                                \+  /* next, a version that just computes the cardinality*/                      \+  static inline int _avx2_bitset_container_##opname##_justcard(                              \+      const bitset_container_t *src_1, const bitset_container_t *src_2) {      \+    const __m256i *__restrict__ data1 = (const __m256i *)src_1->words;         \+    const __m256i *__restrict__ data2 = (const __m256i *)src_2->words;         \+    return (int)avx2_harley_seal_popcount256_##opname(                         \+        data2, data1, BITSET_CONTAINER_SIZE_IN_WORDS / (CROARING_WORDS_IN_AVX2_REG));   \+  }+++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, or,    |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, union, |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, and,          &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, intersection, &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, xor,    ^,  _mm256_xor_si256,    veorq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN1(CROARING_TARGET_AVX2, andnot, &~, _mm256_andnot_si256, vbicq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, or,    |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, union, |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, and,          &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, intersection, &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, xor,    ^,  _mm256_xor_si256,    veorq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN2(CROARING_TARGET_AVX2, andnot, &~, _mm256_andnot_si256, vbicq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, or,    |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, union, |, _mm256_or_si256, vorrq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, and,          &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, intersection, &, _mm256_and_si256, vandq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2++CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, xor,    ^,  _mm256_xor_si256,    veorq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+CROARING_TARGET_AVX2+CROARING_AVX_BITSET_CONTAINER_FN3(CROARING_TARGET_AVX2, andnot, &~, _mm256_andnot_si256, vbicq_u64, CROARING_UNTARGET_AVX2)+CROARING_UNTARGET_AVX2+++#define SCALAR_BITSET_CONTAINER_FN(opname, opsymbol, avx_intrinsic,            \+                                   neon_intrinsic)                             \+  CROARING_ALLOW_UNALIGNED                                                               \+  static inline int _scalar_bitset_container_##opname(const bitset_container_t *src_1,       \+                                        const bitset_container_t *src_2,       \+                                        bitset_container_t *dst) {             \+    const uint64_t *__restrict__ words_1 = src_1->words;                       \+    const uint64_t *__restrict__ words_2 = src_2->words;                       \+    uint64_t *out = dst->words;                                                \+    int32_t sum = 0;                                                           \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 2) {           \+      const uint64_t word_1 = (words_1[i])opsymbol(words_2[i]),                \+                     word_2 = (words_1[i + 1]) opsymbol(words_2[i + 1]);       \+      out[i] = word_1;                                                         \+      out[i + 1] = word_2;                                                     \+      sum += roaring_hamming(word_1);                                                  \+      sum += roaring_hamming(word_2);                                                  \+    }                                                                          \+    dst->cardinality = sum;                                                    \+    return dst->cardinality;                                                   \+  }                                                                            \+  CROARING_ALLOW_UNALIGNED                                                               \+  static inline int _scalar_bitset_container_##opname##_nocard(                              \+      const bitset_container_t *src_1, const bitset_container_t *src_2,        \+      bitset_container_t *dst) {                                               \+    const uint64_t *__restrict__ words_1 = src_1->words;                       \+    const uint64_t *__restrict__ words_2 = src_2->words;                       \+    uint64_t *out = dst->words;                                                \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) {              \+      out[i] = (words_1[i])opsymbol(words_2[i]);                               \+    }                                                                          \+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;                             \+    return dst->cardinality;                                                   \+  }                                                                            \+  CROARING_ALLOW_UNALIGNED                                                               \+  static inline int _scalar_bitset_container_##opname##_justcard(                            \+      const bitset_container_t *src_1, const bitset_container_t *src_2) {      \+    const uint64_t *__restrict__ words_1 = src_1->words;                       \+    const uint64_t *__restrict__ words_2 = src_2->words;                       \+    int32_t sum = 0;                                                           \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 2) {           \+      const uint64_t word_1 = (words_1[i])opsymbol(words_2[i]),                \+                     word_2 = (words_1[i + 1]) opsymbol(words_2[i + 1]);       \+      sum += roaring_hamming(word_1);                                                  \+      sum += roaring_hamming(word_2);                                                  \+    }                                                                          \+    return sum;                                                                \+  }++// we duplicate the function because other containers use the "or" term, makes API more consistent+SCALAR_BITSET_CONTAINER_FN(or,    |, _mm256_or_si256, vorrq_u64)+SCALAR_BITSET_CONTAINER_FN(union, |, _mm256_or_si256, vorrq_u64)++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+SCALAR_BITSET_CONTAINER_FN(and,          &, _mm256_and_si256, vandq_u64)+SCALAR_BITSET_CONTAINER_FN(intersection, &, _mm256_and_si256, vandq_u64)++SCALAR_BITSET_CONTAINER_FN(xor,    ^,  _mm256_xor_si256,    veorq_u64)+SCALAR_BITSET_CONTAINER_FN(andnot, &~, _mm256_andnot_si256, vbicq_u64)++#if CROARING_COMPILER_SUPPORTS_AVX512+#define CROARING_BITSET_CONTAINER_FN(opname, opsymbol, avx_intrinsic, neon_intrinsic)   \+  int bitset_container_##opname(const bitset_container_t *src_1,               \+                                const bitset_container_t *src_2,               \+                                bitset_container_t *dst) {                     \+    int support = croaring_hardware_support();                                 \+    if ( support & ROARING_SUPPORTS_AVX512 ) {                                 \+      return _avx512_bitset_container_##opname(src_1, src_2, dst);             \+    }                                                                          \+    else if ( support & ROARING_SUPPORTS_AVX2 ) {                              \+      return _avx2_bitset_container_##opname(src_1, src_2, dst);               \+    } else {                                                                   \+      return _scalar_bitset_container_##opname(src_1, src_2, dst);             \+    }                                                                          \+  }                                                                            \+  int bitset_container_##opname##_nocard(const bitset_container_t *src_1,      \+                                         const bitset_container_t *src_2,      \+                                         bitset_container_t *dst) {            \+    int support = croaring_hardware_support();                                 \+    if ( support & ROARING_SUPPORTS_AVX512 ) {                                 \+      return _avx512_bitset_container_##opname##_nocard(src_1, src_2, dst);    \+    }                                                                          \+    else if ( support & ROARING_SUPPORTS_AVX2 ) {                              \+      return _avx2_bitset_container_##opname##_nocard(src_1, src_2, dst);      \+    } else {                                                                   \+      return _scalar_bitset_container_##opname##_nocard(src_1, src_2, dst);    \+    }                                                                          \+  }                                                                            \+  int bitset_container_##opname##_justcard(const bitset_container_t *src_1,    \+                                           const bitset_container_t *src_2) {  \+     int support = croaring_hardware_support();                                \+    if ( support & ROARING_SUPPORTS_AVX512 ) {                                 \+      return _avx512_bitset_container_##opname##_justcard(src_1, src_2);       \+    }                                                                          \+    else if ( support & ROARING_SUPPORTS_AVX2 ) {                              \+      return _avx2_bitset_container_##opname##_justcard(src_1, src_2);         \+    } else {                                                                   \+      return _scalar_bitset_container_##opname##_justcard(src_1, src_2);       \+    }                                                                          \+  }++#else // CROARING_COMPILER_SUPPORTS_AVX512+++#define CROARING_BITSET_CONTAINER_FN(opname, opsymbol, avx_intrinsic, neon_intrinsic)   \+  int bitset_container_##opname(const bitset_container_t *src_1,               \+                                const bitset_container_t *src_2,               \+                                bitset_container_t *dst) {                     \+    if ( croaring_hardware_support() & ROARING_SUPPORTS_AVX2 ) {               \+      return _avx2_bitset_container_##opname(src_1, src_2, dst);               \+    } else {                                                                   \+      return _scalar_bitset_container_##opname(src_1, src_2, dst);             \+    }                                                                          \+  }                                                                            \+  int bitset_container_##opname##_nocard(const bitset_container_t *src_1,      \+                                         const bitset_container_t *src_2,      \+                                         bitset_container_t *dst) {            \+    if ( croaring_hardware_support() & ROARING_SUPPORTS_AVX2 ) {               \+      return _avx2_bitset_container_##opname##_nocard(src_1, src_2, dst);      \+    } else {                                                                   \+      return _scalar_bitset_container_##opname##_nocard(src_1, src_2, dst);    \+    }                                                                          \+  }                                                                            \+  int bitset_container_##opname##_justcard(const bitset_container_t *src_1,    \+                                           const bitset_container_t *src_2) {  \+    if ( croaring_hardware_support() & ROARING_SUPPORTS_AVX2 ) {               \+      return _avx2_bitset_container_##opname##_justcard(src_1, src_2);         \+    } else {                                                                   \+      return _scalar_bitset_container_##opname##_justcard(src_1, src_2);       \+    }                                                                          \+  }++#endif //  CROARING_COMPILER_SUPPORTS_AVX512++#elif defined(CROARING_USENEON)++#define CROARING_BITSET_CONTAINER_FN(opname, opsymbol, avx_intrinsic, neon_intrinsic)  \+int bitset_container_##opname(const bitset_container_t *src_1,                \+                              const bitset_container_t *src_2,                \+                              bitset_container_t *dst) {                      \+    const uint64_t * __restrict__ words_1 = src_1->words;                     \+    const uint64_t * __restrict__ words_2 = src_2->words;                     \+    uint64_t *out = dst->words;                                               \+    uint16x8_t n0 = vdupq_n_u16(0);                                           \+    uint16x8_t n1 = vdupq_n_u16(0);                                           \+    uint16x8_t n2 = vdupq_n_u16(0);                                           \+    uint16x8_t n3 = vdupq_n_u16(0);                                           \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 8) {          \+        uint64x2_t c0 = neon_intrinsic(vld1q_u64(&words_1[i + 0]),            \+                                       vld1q_u64(&words_2[i + 0]));           \+        n0 = vaddq_u16(n0, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c0))));   \+        vst1q_u64(&out[i + 0], c0);                                           \+        uint64x2_t c1 = neon_intrinsic(vld1q_u64(&words_1[i + 2]),            \+                                       vld1q_u64(&words_2[i + 2]));           \+        n1 = vaddq_u16(n1, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c1))));   \+        vst1q_u64(&out[i + 2], c1);                                           \+        uint64x2_t c2 = neon_intrinsic(vld1q_u64(&words_1[i + 4]),            \+                                       vld1q_u64(&words_2[i + 4]));           \+        n2 = vaddq_u16(n2, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c2))));   \+        vst1q_u64(&out[i + 4], c2);                                           \+        uint64x2_t c3 = neon_intrinsic(vld1q_u64(&words_1[i + 6]),            \+                                       vld1q_u64(&words_2[i + 6]));           \+        n3 = vaddq_u16(n3, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c3))));   \+        vst1q_u64(&out[i + 6], c3);                                           \+    }                                                                         \+    uint64x2_t n = vdupq_n_u64(0);                                            \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n0)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n1)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n2)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n3)));                           \+    dst->cardinality = vgetq_lane_u64(n, 0) + vgetq_lane_u64(n, 1);           \+    return dst->cardinality;                                                  \+}                                                                             \+int bitset_container_##opname##_nocard(const bitset_container_t *src_1,       \+                                       const bitset_container_t *src_2,       \+                                             bitset_container_t *dst) {       \+    const uint64_t * __restrict__ words_1 = src_1->words;                     \+    const uint64_t * __restrict__ words_2 = src_2->words;                     \+    uint64_t *out = dst->words;                                               \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 8) {          \+        vst1q_u64(&out[i + 0], neon_intrinsic(vld1q_u64(&words_1[i + 0]),     \+                                              vld1q_u64(&words_2[i + 0])));   \+        vst1q_u64(&out[i + 2], neon_intrinsic(vld1q_u64(&words_1[i + 2]),     \+                                              vld1q_u64(&words_2[i + 2])));   \+        vst1q_u64(&out[i + 4], neon_intrinsic(vld1q_u64(&words_1[i + 4]),     \+                                              vld1q_u64(&words_2[i + 4])));   \+        vst1q_u64(&out[i + 6], neon_intrinsic(vld1q_u64(&words_1[i + 6]),     \+                                              vld1q_u64(&words_2[i + 6])));   \+    }                                                                         \+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;                            \+    return dst->cardinality;                                                  \+}                                                                             \+int bitset_container_##opname##_justcard(const bitset_container_t *src_1,     \+                                         const bitset_container_t *src_2) {   \+    const uint64_t * __restrict__ words_1 = src_1->words;                     \+    const uint64_t * __restrict__ words_2 = src_2->words;                     \+    uint16x8_t n0 = vdupq_n_u16(0);                                           \+    uint16x8_t n1 = vdupq_n_u16(0);                                           \+    uint16x8_t n2 = vdupq_n_u16(0);                                           \+    uint16x8_t n3 = vdupq_n_u16(0);                                           \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 8) {          \+        uint64x2_t c0 = neon_intrinsic(vld1q_u64(&words_1[i + 0]),            \+                                       vld1q_u64(&words_2[i + 0]));           \+        n0 = vaddq_u16(n0, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c0))));   \+        uint64x2_t c1 = neon_intrinsic(vld1q_u64(&words_1[i + 2]),            \+                                       vld1q_u64(&words_2[i + 2]));           \+        n1 = vaddq_u16(n1, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c1))));   \+        uint64x2_t c2 = neon_intrinsic(vld1q_u64(&words_1[i + 4]),            \+                                       vld1q_u64(&words_2[i + 4]));           \+        n2 = vaddq_u16(n2, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c2))));   \+        uint64x2_t c3 = neon_intrinsic(vld1q_u64(&words_1[i + 6]),            \+                                       vld1q_u64(&words_2[i + 6]));           \+        n3 = vaddq_u16(n3, vpaddlq_u8(vcntq_u8(vreinterpretq_u8_u64(c3))));   \+    }                                                                         \+    uint64x2_t n = vdupq_n_u64(0);                                            \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n0)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n1)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n2)));                           \+    n = vaddq_u64(n, vpaddlq_u32(vpaddlq_u16(n3)));                           \+    return vgetq_lane_u64(n, 0) + vgetq_lane_u64(n, 1);                       \+}++#else++#define CROARING_BITSET_CONTAINER_FN(opname, opsymbol, avx_intrinsic, neon_intrinsic)  \+int bitset_container_##opname(const bitset_container_t *src_1,            \+                              const bitset_container_t *src_2,            \+                              bitset_container_t *dst) {                  \+    const uint64_t * __restrict__ words_1 = src_1->words;                 \+    const uint64_t * __restrict__ words_2 = src_2->words;                 \+    uint64_t *out = dst->words;                                           \+    int32_t sum = 0;                                                      \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 2) {      \+        const uint64_t word_1 = (words_1[i])opsymbol(words_2[i]),         \+                       word_2 = (words_1[i + 1])opsymbol(words_2[i + 1]); \+        out[i] = word_1;                                                  \+        out[i + 1] = word_2;                                              \+        sum += roaring_hamming(word_1);                                    \+        sum += roaring_hamming(word_2);                                    \+    }                                                                     \+    dst->cardinality = sum;                                               \+    return dst->cardinality;                                              \+}                                                                         \+int bitset_container_##opname##_nocard(const bitset_container_t *src_1,   \+                                       const bitset_container_t *src_2,   \+                                       bitset_container_t *dst) {         \+    const uint64_t * __restrict__ words_1 = src_1->words;                 \+    const uint64_t * __restrict__ words_2 = src_2->words;                 \+    uint64_t *out = dst->words;                                           \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) {         \+        out[i] = (words_1[i])opsymbol(words_2[i]);                        \+    }                                                                     \+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;                        \+    return dst->cardinality;                                              \+}                                                                         \+int bitset_container_##opname##_justcard(const bitset_container_t *src_1, \+                              const bitset_container_t *src_2) {          \+    const uint64_t * __restrict__ words_1 = src_1->words;                 \+    const uint64_t * __restrict__ words_2 = src_2->words;                 \+    int32_t sum = 0;                                                      \+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i += 2) {      \+        const uint64_t word_1 = (words_1[i])opsymbol(words_2[i]),         \+                       word_2 = (words_1[i + 1])opsymbol(words_2[i + 1]); \+        sum += roaring_hamming(word_1);                                    \+        sum += roaring_hamming(word_2);                                    \+    }                                                                     \+    return sum;                                                           \+}++#endif // CROARING_IS_X64++// we duplicate the function because other containers use the "or" term, makes API more consistent+CROARING_BITSET_CONTAINER_FN(or,    |, _mm256_or_si256, vorrq_u64)+CROARING_BITSET_CONTAINER_FN(union, |, _mm256_or_si256, vorrq_u64)++// we duplicate the function because other containers use the "intersection" term, makes API more consistent+CROARING_BITSET_CONTAINER_FN(and,          &, _mm256_and_si256, vandq_u64)+CROARING_BITSET_CONTAINER_FN(intersection, &, _mm256_and_si256, vandq_u64)++CROARING_BITSET_CONTAINER_FN(xor,    ^,  _mm256_xor_si256,    veorq_u64)+CROARING_BITSET_CONTAINER_FN(andnot, &~, _mm256_andnot_si256, vbicq_u64)+// clang-format On+++CROARING_ALLOW_UNALIGNED+int bitset_container_to_uint32_array(+    uint32_t *out,+    const bitset_container_t *bc,+    uint32_t base+){+#if CROARING_IS_X64+   int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+   // Unlike the AVX2 kernel, the AVX-512 one needs no cardinality heuristic:+   // it skips empty words and only stores the blocks that carry a value, so it+   // beat the scalar loop at every cardinality measured from 16 to 65536 (on+   // an Emerald Rapids Xeon, 1.3x at 512 values, 6x at 4096, 7.5x at 8192).+   if( support & ROARING_SUPPORTS_AVX512 )+		return (int) bitset_extract_setbits_avx512(bc->words,+                BITSET_CONTAINER_SIZE_IN_WORDS, out, bc->cardinality, base);+   else+#endif+   if(( support & ROARING_SUPPORTS_AVX2 ) &&  (bc->cardinality >= 8192))  // heuristic+		return (int) bitset_extract_setbits_avx2(bc->words,+                BITSET_CONTAINER_SIZE_IN_WORDS, out, bc->cardinality, base);+	else+		return (int) bitset_extract_setbits(bc->words,+                BITSET_CONTAINER_SIZE_IN_WORDS, out, base);+#else+	return (int) bitset_extract_setbits(bc->words,+                BITSET_CONTAINER_SIZE_IN_WORDS, out, base);+#endif+}++/*+ * Print this container using printf (useful for debugging).+ */+void bitset_container_printf(const bitset_container_t * v) {+	printf("{");+	uint32_t base = 0;+	bool iamfirst = true;// TODO: rework so that this is not necessary yet still readable+	for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+		uint64_t w = v->words[i];+		while (w != 0) {+			uint64_t t = w & (~w + 1);+			int r = roaring_trailing_zeroes(w);+			if(iamfirst) {// predicted to be false+				printf("%u",base + r);+				iamfirst = false;+			} else {+				printf(",%u",base + r);+			}+			w ^= t;+		}+		base += 64;+	}+	printf("}");+}+++/*+ * Print this container using printf as a comma-separated list of 32-bit integers starting at base.+ */+void bitset_container_printf_as_uint32_array(const bitset_container_t * v, uint32_t base) {+	bool iamfirst = true;// TODO: rework so that this is not necessary yet still readable+	for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+		uint64_t w = v->words[i];+		while (w != 0) {+			uint64_t t = w & (~w + 1);+			int r = roaring_trailing_zeroes(w);+			if(iamfirst) {// predicted to be false+				printf("%u", r + base);+				iamfirst = false;+			} else {+				printf(",%u",r + base);+			}+			w ^= t;+		}+		base += 64;+	}+}++/*+ * Validate the container. Returns true if valid.+ */+CROARING_ALLOW_UNALIGNED+bool bitset_container_validate(const bitset_container_t *v, const char **reason) {+    if (v->words == NULL) {+        *reason = "words is NULL";+        return false;+    }+    if (v->cardinality != bitset_container_compute_cardinality(v)) {+        *reason = "cardinality is incorrect";+        return false;+    }+    if (v->cardinality <= DEFAULT_MAX_SIZE) {+        *reason = "cardinality is too small for a bitmap container";+        return false;+    }+    // Attempt to forcibly load the first and last words, hopefully causing+    // a segfault or an address sanitizer error if words is not allocated.+    volatile uint64_t *words = v->words;+    (void) words[0];+    (void) words[BITSET_CONTAINER_SIZE_IN_WORDS - 1];+    return true;+}+++// TODO: use the fast lower bound, also+int bitset_container_number_of_runs(bitset_container_t *bc) {+  int num_runs = 0;+  uint64_t next_word = bc->words[0];++  for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS-1; ++i) {+    uint64_t word = next_word;+    next_word = bc->words[i+1];+    num_runs += roaring_hamming((~word) & (word << 1)) + ( (word >> 63) & ~next_word);+  }++  uint64_t word = next_word;+  num_runs += roaring_hamming((~word) & (word << 1));+  if((word & 0x8000000000000000ULL) != 0)+    num_runs++;+  return num_runs;+}+++CROARING_ALLOW_UNALIGNED+int32_t bitset_container_write(const bitset_container_t *container,+                                  char *buf) {+#if CROARING_IS_BIG_ENDIAN+	for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+		uint64_t w_le = croaring_htole64(container->words[i]);+		memcpy(buf + i * sizeof(uint64_t), &w_le, sizeof(uint64_t));+	}+#else+	memcpy(buf, container->words, BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t));+#endif+	return bitset_container_size_in_bytes(container);+}+++int32_t bitset_container_read(int32_t cardinality, bitset_container_t *container,+		const char *buf)  {+	container->cardinality = cardinality;+#if CROARING_IS_BIG_ENDIAN+	for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+		uint64_t w_le;+		memcpy(&w_le, buf + i * sizeof(uint64_t), sizeof(uint64_t));+		container->words[i] = croaring_letoh64(w_le);+	}+#else+	memcpy(container->words, buf, BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t));+#endif+	return bitset_container_size_in_bytes(container);+}++CROARING_ALLOW_UNALIGNED+bool bitset_container_iterate(const bitset_container_t *cont, uint32_t base, roaring_iterator iterator, void *ptr) {+  for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i ) {+    uint64_t w = cont->words[i];+    while (w != 0) {+      uint64_t t = w & (~w + 1);+      int r = roaring_trailing_zeroes(w);+      if(!iterator(r + base, ptr)) return false;+      w ^= t;+    }+    base += 64;+  }+  return true;+}++CROARING_ALLOW_UNALIGNED+bool bitset_container_iterate64(const bitset_container_t *cont, uint32_t base, roaring_iterator64 iterator, uint64_t high_bits, void *ptr) {+  for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i ) {+    uint64_t w = cont->words[i];+    while (w != 0) {+      uint64_t t = w & (~w + 1);+      int r = roaring_trailing_zeroes(w);+      if(!iterator(high_bits | (uint64_t)(r + base), ptr)) return false;+      w ^= t;+    }+    base += 64;+  }+  return true;+}++#if CROARING_IS_X64+#if CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX512+CROARING_ALLOW_UNALIGNED+static inline bool _avx512_bitset_container_equals(const bitset_container_t *container1, const bitset_container_t *container2) {+  const __m512i *ptr1 = (const __m512i*)container1->words;+  const __m512i *ptr2 = (const __m512i*)container2->words;+  for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS*sizeof(uint64_t)/64; i++) {+      __m512i r1 = _mm512_loadu_si512(ptr1+i);+      __m512i r2 = _mm512_loadu_si512(ptr2+i);+      __mmask64 mask = _mm512_cmpeq_epi8_mask(r1, r2);+      if ((uint64_t)mask != UINT64_MAX) {+          return false;+      }+  }+	return true;+}+CROARING_UNTARGET_AVX512+#endif // CROARING_COMPILER_SUPPORTS_AVX512+CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+static inline bool _avx2_bitset_container_equals(const bitset_container_t *container1, const bitset_container_t *container2) {+    const __m256i *ptr1 = (const __m256i*)container1->words;+    const __m256i *ptr2 = (const __m256i*)container2->words;+    for (size_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS*sizeof(uint64_t)/32; i++) {+      __m256i r1 = _mm256_loadu_si256(ptr1+i);+      __m256i r2 = _mm256_loadu_si256(ptr2+i);+      int mask = _mm256_movemask_epi8(_mm256_cmpeq_epi8(r1, r2));+      if ((uint32_t)mask != UINT32_MAX) {+          return false;+      }+  }+	return true;+}+CROARING_UNTARGET_AVX2+#endif // CROARING_IS_X64++CROARING_ALLOW_UNALIGNED+bool bitset_container_equals(const bitset_container_t *container1, const bitset_container_t *container2) {+  if((container1->cardinality != BITSET_UNKNOWN_CARDINALITY) && (container2->cardinality != BITSET_UNKNOWN_CARDINALITY)) {+    if(container1->cardinality != container2->cardinality) {+      return false;+    }+    if (container1->cardinality == INT32_C(0x10000)) {+      return true;+    }+  }+#if CROARING_IS_X64+  int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+  if( support & ROARING_SUPPORTS_AVX512 ) {+    return _avx512_bitset_container_equals(container1, container2);+  }+  else+#endif+  if( support & ROARING_SUPPORTS_AVX2 ) {+    return _avx2_bitset_container_equals(container1, container2);+  }+#endif+  return memcmp(container1->words,+                container2->words,+                BITSET_CONTAINER_SIZE_IN_WORDS*sizeof(uint64_t)) == 0;+}++CROARING_ALLOW_UNALIGNED+bool bitset_container_is_subset(const bitset_container_t *container1,+                          const bitset_container_t *container2) {+    if((container1->cardinality != BITSET_UNKNOWN_CARDINALITY) && (container2->cardinality != BITSET_UNKNOWN_CARDINALITY)) {+        if(container1->cardinality > container2->cardinality) {+            return false;+        }+    }+    for(int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i ) {+		if((container1->words[i] & container2->words[i]) != container1->words[i]) {+			return false;+		}+	}+	return true;+}++// Returns the position of the set bit of rank r (0-based) in w. Requires+// r < roaring_hamming(w). Narrowing with popcounts compiles to conditional+// moves, avoiding a mispredicted exit from a bit-by-bit loop.+static inline uint32_t bitset_select_in_word(uint64_t w, uint32_t r) {+    uint32_t pos = 0;+    uint32_t c = roaring_hamming(w & 0xFFFFFFFF);+    if (r >= c) {+        r -= c;+        w >>= 32;+        pos += 32;+    }+    c = roaring_hamming(w & 0xFFFF);+    if (r >= c) {+        r -= c;+        w >>= 16;+        pos += 16;+    }+    c = roaring_hamming(w & 0xFF);+    if (r >= c) {+        r -= c;+        w >>= 8;+        pos += 8;+    }+    for (; r > 0; r--) {+        w &= w - 1;+    }+    return pos + roaring_trailing_zeroes(w);+}++CROARING_ALLOW_UNALIGNED+bool bitset_container_select(const bitset_container_t *container,+                             uint32_t *start_rank, uint32_t rank,+                             uint32_t *element) {+    int card = bitset_container_cardinality(container);+    if (rank >= *start_rank + card) {+        *start_rank += card;+        return false;+    }+    const uint64_t *words = container->words;+    uint32_t r = rank - *start_rank;+    int32_t i = 0;+    // Independent popcounts over blocks of words pipeline well, and the+    // per-block branch is taken far less often than a per-word branch.+    for (; i + 8 <= BITSET_CONTAINER_SIZE_IN_WORDS; i += 8) {+        uint32_t c = roaring_hamming(words[i]) + roaring_hamming(words[i + 1]) ++                     roaring_hamming(words[i + 2]) ++                     roaring_hamming(words[i + 3]) ++                     roaring_hamming(words[i + 4]) ++                     roaring_hamming(words[i + 5]) ++                     roaring_hamming(words[i + 6]) ++                     roaring_hamming(words[i + 7]);+        if (r < c) {+            break;+        }+        r -= c;+    }+    for (; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) {+        uint32_t c = roaring_hamming(words[i]);+        if (r < c) {+            *element = (uint32_t)i * 64 + bitset_select_in_word(words[i], r);+            *start_rank = rank;+            return true;+        }+        r -= c;+    }+    // Only reachable if the stored cardinality is inconsistent.+    assert(false);+    *start_rank += card;+    return false;+}+++/* Returns the smallest value (assumes not empty) */+CROARING_ALLOW_UNALIGNED+uint16_t bitset_container_minimum(const bitset_container_t *container) {+  for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i ) {+    uint64_t w = container->words[i];+    if (w != 0) {+      int r = roaring_trailing_zeroes(w);+      return r + i * 64;+    }+  }+  return UINT16_MAX;+}++/* Returns the largest value (assumes not empty) */+CROARING_ALLOW_UNALIGNED+uint16_t bitset_container_maximum(const bitset_container_t *container) {+  for (int32_t i = BITSET_CONTAINER_SIZE_IN_WORDS - 1; i >= 0; --i ) {+    uint64_t w = container->words[i];+    if (w != 0) {+      int r = roaring_leading_zeroes(w);+      return i * 64 + 63  - r;+    }+  }+  return 0;+}++/* Returns the number of values equal or smaller than x */+CROARING_ALLOW_UNALIGNED+int bitset_container_rank(const bitset_container_t *container, uint16_t x) {+  // credit: aqrit+  int sum = 0;+  int i = 0;+  for (int end = x / 64; i < end; i++){+    sum += roaring_hamming(container->words[i]);+  }+  uint64_t lastword = container->words[i];+  uint64_t lastpos = UINT64_C(1) << (x % 64);+  uint64_t mask = lastpos + lastpos - 1; // smear right+  sum += roaring_hamming(lastword & mask);+  return sum;+}++CROARING_ALLOW_UNALIGNED+uint32_t bitset_container_rank_many(const bitset_container_t *container, uint64_t start_rank, const uint32_t* begin, const uint32_t* end, uint64_t* ans){+  const uint16_t high = (uint16_t)((*begin) >> 16);+  int i = 0;+  int sum = 0;+  const uint32_t* iter = begin;+  for(; iter != end; iter++) {+      uint32_t x = *iter;+      uint16_t xhigh = (uint16_t)(x >> 16);+      if(xhigh != high) return iter - begin; // stop at next container++      uint16_t xlow = (uint16_t)x;+      for(int count = xlow / 64; i < count; i++){+        sum += roaring_hamming(container->words[i]);+      }+      uint64_t lastword = container->words[i];+      uint64_t lastpos = UINT64_C(1) << (xlow % 64);+      uint64_t mask = lastpos + lastpos - 1; // smear right+      *(ans++) = start_rank + sum + roaring_hamming(lastword & mask);+  }+  return iter - begin;+}+++/* Returns the index of x , if not exsist return -1 */+CROARING_ALLOW_UNALIGNED+int bitset_container_get_index(const bitset_container_t *container, uint16_t x) {+  if (bitset_container_get(container, x)) {+    // credit: aqrit+    int sum = 0;+    int i = 0;+    for (int end = x / 64; i < end; i++){+      sum += roaring_hamming(container->words[i]);+    }+    uint64_t lastword = container->words[i];+    uint64_t lastpos = UINT64_C(1) << (x % 64);+    uint64_t mask = lastpos + lastpos - 1; // smear right+    sum += roaring_hamming(lastword & mask);+    return sum - 1;+  } else {+    return -1;+  }+}++/* Returns the index of the first value equal or larger than x, or -1 */+CROARING_ALLOW_UNALIGNED+int bitset_container_index_equalorlarger(const bitset_container_t *container, uint16_t x) {+  uint32_t x32 = x;+  uint32_t k = x32 / 64;+  uint64_t word = container->words[k];+  const int diff = x32 - k * 64; // in [0,64)+  word = (word >> diff) << diff; // a mask is faster, but we don't care+  while(word == 0) {+    k++;+    if(k == BITSET_CONTAINER_SIZE_IN_WORDS) return -1;+    word = container->words[k];+  }+  return k * 64 + roaring_trailing_zeroes(word);+}++#ifdef __cplusplus+} } }  // extern "C" { namespace roaring { namespace internal {+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif
+ cbits/CRoaring/src/containers/containers.c view
@@ -0,0 +1,1158 @@+#include <roaring/containers/containers.h>+#include <roaring/memory.h>++#ifdef __cplusplus+extern "C" {+// In Windows MSVC C++ compiler, (type){init} does not compile,+// it causes C4576: a parenthesized type followed by an initializer list is a+// non-standard explicit type conversion syntax The correct syntax is type{init}+#define ROARING_INIT_ROARING_CONTAINER_ITERATOR_T roaring_container_iterator_t+namespace roaring {+namespace internal {+#else+#define ROARING_INIT_ROARING_CONTAINER_ITERATOR_T (roaring_container_iterator_t)+#endif++static inline uint32_t minimum_uint32(uint32_t a, uint32_t b) {+    return (a < b) ? a : b;+}++extern inline const container_t *container_unwrap_shared(+    const container_t *candidate_shared_container, uint8_t *type);++extern inline container_t *container_mutable_unwrap_shared(+    container_t *candidate_shared_container, uint8_t *type);++extern inline int container_get_cardinality(const container_t *c,+                                            uint8_t typecode);++extern inline container_t *container_iand(container_t *c1, uint8_t type1,+                                          const container_t *c2, uint8_t type2,+                                          uint8_t *result_type);++extern inline container_t *container_ior(container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type);++extern inline container_t *container_ixor(container_t *c1, uint8_t type1,+                                          const container_t *c2, uint8_t type2,+                                          uint8_t *result_type);++extern inline container_t *container_iandnot(container_t *c1, uint8_t type1,+                                             const container_t *c2,+                                             uint8_t type2,+                                             uint8_t *result_type);++extern bool container_iterator_next(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value);++extern bool container_iterator_prev(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value);+extern bool container_contains(+    const container_t *c, uint16_t val,+    uint8_t typecode  // !!! should be second argument?+);++void container_free(container_t *c, uint8_t type) {+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            bitset_container_free(CAST_bitset(c));+            break;+        case ARRAY_CONTAINER_TYPE:+            array_container_free(CAST_array(c));+            break;+        case RUN_CONTAINER_TYPE:+            run_container_free(CAST_run(c));+            break;+        case SHARED_CONTAINER_TYPE:+            shared_container_free(CAST_shared(c));+            break;+        default:+            assert(false);+            roaring_unreachable;+    }+}++void container_printf(const container_t *c, uint8_t type) {+    c = container_unwrap_shared(c, &type);+    switch (type) {+        case BITSET_CONTAINER_TYPE:+            bitset_container_printf(const_CAST_bitset(c));+            return;+        case ARRAY_CONTAINER_TYPE:+            array_container_printf(const_CAST_array(c));+            return;+        case RUN_CONTAINER_TYPE:+            run_container_printf(const_CAST_run(c));+            return;+        default:+            roaring_unreachable;+    }+}++void container_printf_as_uint32_array(const container_t *c, uint8_t typecode,+                                      uint32_t base) {+    c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            bitset_container_printf_as_uint32_array(const_CAST_bitset(c), base);+            return;+        case ARRAY_CONTAINER_TYPE:+            array_container_printf_as_uint32_array(const_CAST_array(c), base);+            return;+        case RUN_CONTAINER_TYPE:+            run_container_printf_as_uint32_array(const_CAST_run(c), base);+            return;+        default:+            roaring_unreachable;+    }+}++bool container_internal_validate(const container_t *container, uint8_t typecode,+                                 const char **reason) {+    if (container == NULL) {+        *reason = "container is NULL";+        return false;+    }+    // Not using container_unwrap_shared because it asserts if shared containers+    // are nested+    if (typecode == SHARED_CONTAINER_TYPE) {+        const shared_container_t *shared_container =+            const_CAST_shared(container);+        if (croaring_refcount_get(&shared_container->counter) == 0) {+            *reason = "shared container has zero refcount";+            return false;+        }+        if (shared_container->typecode == SHARED_CONTAINER_TYPE) {+            *reason = "shared container is nested";+            return false;+        }+        if (shared_container->container == NULL) {+            *reason = "shared container has NULL container";+            return false;+        }+        container = shared_container->container;+        typecode = shared_container->typecode;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_validate(const_CAST_bitset(container),+                                             reason);+        case ARRAY_CONTAINER_TYPE:+            return array_container_validate(const_CAST_array(container),+                                            reason);+        case RUN_CONTAINER_TYPE:+            return run_container_validate(const_CAST_run(container), reason);+        default:+            *reason = "invalid typecode";+            return false;+    }+}++extern inline bool container_nonzero_cardinality(const container_t *c,+                                                 uint8_t typecode);++extern inline int container_to_uint32_array(uint32_t *output,+                                            const container_t *c,+                                            uint8_t typecode, uint32_t base);++extern inline container_t *container_add(container_t *c, uint16_t val,+                                         uint8_t typecode,  // !!! 2nd arg?+                                         uint8_t *new_typecode);++extern inline bool container_contains(const container_t *c, uint16_t val,+                                      uint8_t typecode);  // !!! 2nd arg?++extern inline container_t *container_and(const container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type);++extern inline container_t *container_or(const container_t *c1, uint8_t type1,+                                        const container_t *c2, uint8_t type2,+                                        uint8_t *result_type);++extern inline container_t *container_xor(const container_t *c1, uint8_t type1,+                                         const container_t *c2, uint8_t type2,+                                         uint8_t *result_type);++container_t *get_copy_of_container(container_t *c, uint8_t *typecode,+                                   bool copy_on_write) {+    if (copy_on_write) {+        shared_container_t *shared_container;+        if (*typecode == SHARED_CONTAINER_TYPE) {+            shared_container = CAST_shared(c);+            croaring_refcount_inc(&shared_container->counter);+            return shared_container;+        }+        assert(*typecode != SHARED_CONTAINER_TYPE);++        if ((shared_container = (shared_container_t *)roaring_malloc(+                 sizeof(shared_container_t))) == NULL) {+            return NULL;+        }++        shared_container->container = c;+        shared_container->typecode = *typecode;+        // At this point, we are creating new shared container+        // so there should be no other references, and setting+        // the counter to 2 - even non-atomically - is safe as+        // long as the value is set before the return statement.+        shared_container->counter = 2;+        *typecode = SHARED_CONTAINER_TYPE;++        return shared_container;+    }  // copy_on_write+    // otherwise, no copy on write...+    const container_t *actual_container = container_unwrap_shared(c, typecode);+    assert(*typecode != SHARED_CONTAINER_TYPE);+    return container_clone(actual_container, *typecode);+}++/**+ * Copies a container, requires a typecode. This allocates new memory, caller+ * is responsible for deallocation.+ */+container_t *container_clone(const container_t *c, uint8_t typecode) {+    // We do not want to allow cloning of shared containers.+    // c = container_unwrap_shared(c, &typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE:+            return bitset_container_clone(const_CAST_bitset(c));+        case ARRAY_CONTAINER_TYPE:+            return array_container_clone(const_CAST_array(c));+        case RUN_CONTAINER_TYPE:+            return run_container_clone(const_CAST_run(c));+        case SHARED_CONTAINER_TYPE:+            // Shared containers are not cloneable. Are you mixing COW and+            // non-COW bitmaps?+            return NULL;+        default:+            assert(false);+            roaring_unreachable;+            return NULL;+    }+}++container_t *shared_container_extract_copy(shared_container_t *sc,+                                           uint8_t *typecode) {+    assert(sc->typecode != SHARED_CONTAINER_TYPE);+    *typecode = sc->typecode;+    container_t *answer;+    if (croaring_refcount_is_unique(&sc->counter)) {+        answer = sc->container;+        sc->container = NULL;  // paranoid+        roaring_free(sc);+    } else {+        answer = container_clone(sc->container, *typecode);+        shared_container_free(sc);+    }+    assert(*typecode != SHARED_CONTAINER_TYPE);+    return answer;+}++void shared_container_free(shared_container_t *container) {+    if (croaring_refcount_dec(&container->counter)) {+        assert(container->typecode != SHARED_CONTAINER_TYPE);+        container_free(container->container, container->typecode);+        container->container = NULL;  // paranoid+        roaring_free(container);+    }+}++extern inline container_t *container_not(const container_t *c1, uint8_t type1,+                                         uint8_t *result_type);++extern inline container_t *container_not_range(const container_t *c1,+                                               uint8_t type1,+                                               uint32_t range_start,+                                               uint32_t range_end,+                                               uint8_t *result_type);++extern inline container_t *container_inot(container_t *c1, uint8_t type1,+                                          uint8_t *result_type);++extern inline container_t *container_inot_range(container_t *c1, uint8_t type1,+                                                uint32_t range_start,+                                                uint32_t range_end,+                                                uint8_t *result_type);++extern inline container_t *container_range_of_ones(uint32_t range_start,+                                                   uint32_t range_end,+                                                   uint8_t *result_type);++// where are the correponding things for union and intersection??+extern inline container_t *container_lazy_xor(const container_t *c1,+                                              uint8_t type1,+                                              const container_t *c2,+                                              uint8_t type2,+                                              uint8_t *result_type);++extern inline container_t *container_lazy_ixor(container_t *c1, uint8_t type1,+                                               const container_t *c2,+                                               uint8_t type2,+                                               uint8_t *result_type);++extern inline container_t *container_andnot(const container_t *c1,+                                            uint8_t type1,+                                            const container_t *c2,+                                            uint8_t type2,+                                            uint8_t *result_type);++CROARING_ALLOW_UNALIGNED+roaring_container_iterator_t container_init_iterator(const container_t *c,+                                                     uint8_t typecode,+                                                     uint16_t *value) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = 0;+            uint64_t word;+            while ((word = bc->words[wordindex]) == 0) {+                wordindex++;+            }+            // word is non-zero+            int32_t index = wordindex * 64 + roaring_trailing_zeroes(word);+            *value = index;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = index,+            };+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            *value = ac->array[0];+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = 0,+            };+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            *value = rc->runs[0].value;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = 0,+            };+        }+        default:+            assert(false);+            roaring_unreachable;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{0};+    }+}++CROARING_ALLOW_UNALIGNED+roaring_container_iterator_t container_init_iterator_last(const container_t *c,+                                                          uint8_t typecode,+                                                          uint16_t *value) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = BITSET_CONTAINER_SIZE_IN_WORDS - 1;+            uint64_t word;+            while ((word = bc->words[wordindex]) == 0) {+                wordindex--;+            }+            // word is non-zero+            int32_t index =+                wordindex * 64 + (63 - roaring_leading_zeroes(word));+            *value = index;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = index,+            };+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            int32_t index = ac->cardinality - 1;+            *value = ac->array[index];+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = index,+            };+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            int32_t run_index = rc->n_runs - 1;+            const rle16_t *last_run = &rc->runs[run_index];+            *value = last_run->value + last_run->length;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{+                .index = run_index,+            };+        }+        default:+            assert(false);+            roaring_unreachable;+            return ROARING_INIT_ROARING_CONTAINER_ITERATOR_T{0};+    }+}++bool container_iterator_lower_bound(const container_t *c, uint8_t typecode,+                                    roaring_container_iterator_t *it,+                                    uint16_t *value_out, uint16_t val) {+    if (val > container_maximum(c, typecode)) {+        return false;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            it->index = bitset_container_index_equalorlarger(bc, val);+            *value_out = it->index;+            return true;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            it->index = array_container_index_equalorlarger(ac, val);+            *value_out = ac->array[it->index];+            return true;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            it->index = run_container_index_equalorlarger(rc, val);+            if (rc->runs[it->index].value <= val) {+                *value_out = val;+            } else {+                *value_out = rc->runs[it->index].value;+            }+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+}++CROARING_ALLOW_UNALIGNED+bool container_iterator_read_into_uint32(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint32_t high16, uint32_t *buf,+                                         uint32_t count, uint32_t *consumed,+                                         uint16_t *value_out) {+    *consumed = 0;+    if (count == 0) {+        return false;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = it->index / 64;+            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX << (it->index % 64));+            do {+                // Read set bits.+                while (word != 0 && *consumed < count) {+                    *buf = high16 |+                           (wordindex * 64 + roaring_trailing_zeroes(word));+                    word = word & (word - 1);+                    buf++;+                    (*consumed)++;+                }+                // Skip unset bits.+                while (word == 0 &&+                       wordindex + 1 < BITSET_CONTAINER_SIZE_IN_WORDS) {+                    wordindex++;+                    word = bc->words[wordindex];+                }+            } while (word != 0 && *consumed < count);++            if (word != 0) {+                it->index = wordindex * 64 + roaring_trailing_zeroes(word);+                *value_out = it->index;+                return true;+            }+            return false;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint32_t num_values =+                minimum_uint32(ac->cardinality - it->index, count);+            // Hoist so GCC can vectorize the uint16->uint32 widen-or.+            const uint16_t *src = ac->array + it->index;+            for (uint32_t i = 0; i < num_values; i++) {+                buf[i] = high16 | src[i];+            }+            *consumed += num_values;+            it->index += num_values;+            if (it->index < ac->cardinality) {+                *value_out = ac->array[it->index];+                return true;+            }+            return false;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            do {+                uint32_t largest_run_value =+                    rc->runs[it->index].value + rc->runs[it->index].length;+                uint32_t num_values = minimum_uint32(+                    largest_run_value - *value_out + 1, count - *consumed);+                for (uint32_t i = 0; i < num_values; i++) {+                    buf[i] = high16 | (*value_out + i);+                }+                *value_out += num_values;+                buf += num_values;+                *consumed += num_values;++                // We check for `value == 0` because `it->value += num_values`+                // can overflow when `value == UINT16_MAX`, and `count >+                // length`. In this case `value` will overflow to 0.+                if (*value_out > largest_run_value || *value_out == 0) {+                    it->index++;+                    if (it->index < rc->n_runs) {+                        *value_out = rc->runs[it->index].value;+                    } else {+                        return false;+                    }+                }+            } while (*consumed < count);+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++CROARING_ALLOW_UNALIGNED+bool container_iterator_read_into_uint64(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint64_t high48, uint64_t *buf,+                                         uint32_t count, uint32_t *consumed,+                                         uint16_t *value_out) {+    *consumed = 0;+    if (count == 0) {+        return false;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = it->index / 64;+            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX << (it->index % 64));+            do {+                // Read set bits.+                while (word != 0 && *consumed < count) {+                    *buf = high48 |+                           (wordindex * 64 + roaring_trailing_zeroes(word));+                    word = word & (word - 1);+                    buf++;+                    (*consumed)++;+                }+                // Skip unset bits.+                while (word == 0 &&+                       wordindex + 1 < BITSET_CONTAINER_SIZE_IN_WORDS) {+                    wordindex++;+                    word = bc->words[wordindex];+                }+            } while (word != 0 && *consumed < count);++            if (word != 0) {+                it->index = wordindex * 64 + roaring_trailing_zeroes(word);+                *value_out = it->index;+                return true;+            }+            return false;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint32_t num_values =+                minimum_uint32(ac->cardinality - it->index, count);+            // Hoist so GCC can vectorize the uint16->uint64 widen-or.+            const uint16_t *src = ac->array + it->index;+            for (uint32_t i = 0; i < num_values; i++) {+                buf[i] = high48 | src[i];+            }+            *consumed += num_values;+            it->index += num_values;+            if (it->index < ac->cardinality) {+                *value_out = ac->array[it->index];+                return true;+            }+            return false;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            do {+                uint32_t largest_run_value =+                    rc->runs[it->index].value + rc->runs[it->index].length;+                uint32_t num_values = minimum_uint32(+                    largest_run_value - *value_out + 1, count - *consumed);+                for (uint32_t i = 0; i < num_values; i++) {+                    buf[i] = high48 | (*value_out + i);+                }+                *value_out += num_values;+                buf += num_values;+                *consumed += num_values;++                // We check for `value == 0` because `it->value += num_values`+                // can overflow when `value == UINT16_MAX`, and `count >+                // length`. In this case `value` will overflow to 0.+                if (*value_out > largest_run_value || *value_out == 0) {+                    it->index++;+                    if (it->index < rc->n_runs) {+                        *value_out = rc->runs[it->index].value;+                    } else {+                        return false;+                    }+                }+            } while (*consumed < count);+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++CROARING_ALLOW_UNALIGNED+bool container_iterator_read_backward_into_uint32(+    const container_t *c, uint8_t typecode, roaring_container_iterator_t *it,+    uint32_t high16, uint32_t *buf, uint32_t count, uint32_t *consumed,+    uint16_t *value_out) {+    *consumed = 0;+    if (count == 0) {+        return false;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = it->index / 64;+            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX >> (63 - (it->index % 64)));+            do {+                // Read set bits.+                while (word != 0 && *consumed < count) {+                    uint32_t bit = 63 - roaring_leading_zeroes(word);+                    *buf = high16 | (wordindex * 64 + bit);+                    word &= ~(UINT64_C(1) << bit);+                    buf++;+                    (*consumed)++;+                }+                // Skip unset bits.+                while (word == 0 && wordindex > 0) {+                    wordindex--;+                    word = bc->words[wordindex];+                }+            } while (word != 0 && *consumed < count);++            if (word != 0) {+                it->index =+                    wordindex * 64 + (63 - roaring_leading_zeroes(word));+                *value_out = it->index;+                return true;+            }+            return false;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint32_t num_values =+                minimum_uint32((uint32_t)(it->index + 1), count);+            // Walk backwards so GCC can vectorize the uint16->uint32 widen-or.+            const uint16_t *src = ac->array + it->index + 1;+            for (uint32_t i = 0; i < num_values; i++) {+                buf[i] = high16 | *--src;+            }+            *consumed += num_values;+            it->index -= num_values;+            if (it->index >= 0) {+                *value_out = ac->array[it->index];+                return true;+            }+            return false;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            do {+                uint32_t run_start = rc->runs[it->index].value;+                uint32_t num_values = minimum_uint32(*value_out - run_start + 1,+                                                     count - *consumed);+                for (uint32_t i = 0; i < num_values; i++) {+                    buf[i] = high16 | (*value_out - i);+                }+                *value_out -= num_values;+                buf += num_values;+                *consumed += num_values;++                // We check for `value == UINT16_MAX` because+                // `*value_out -= num_values` can underflow when+                // `value == 0` (run_start == 0). In this case `value`+                // will underflow to UINT16_MAX.+                if (*value_out < run_start || *value_out == UINT16_MAX) {+                    it->index--;+                    if (it->index >= 0) {+                        *value_out = rc->runs[it->index].value ++                                     rc->runs[it->index].length;+                    } else {+                        return false;+                    }+                }+            } while (*consumed < count);+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++CROARING_ALLOW_UNALIGNED+bool container_iterator_read_backward_into_uint64(+    const container_t *c, uint8_t typecode, roaring_container_iterator_t *it,+    uint64_t high48, uint64_t *buf, uint32_t count, uint32_t *consumed,+    uint16_t *value_out) {+    *consumed = 0;+    if (count == 0) {+        return false;+    }+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t wordindex = it->index / 64;+            uint64_t word =+                bc->words[wordindex] & (UINT64_MAX >> (63 - (it->index % 64)));+            do {+                // Read set bits.+                while (word != 0 && *consumed < count) {+                    uint32_t bit = 63 - roaring_leading_zeroes(word);+                    *buf = high48 | (wordindex * 64 + bit);+                    word &= ~(UINT64_C(1) << bit);+                    buf++;+                    (*consumed)++;+                }+                // Skip unset bits.+                while (word == 0 && wordindex > 0) {+                    wordindex--;+                    word = bc->words[wordindex];+                }+            } while (word != 0 && *consumed < count);++            if (word != 0) {+                it->index =+                    wordindex * 64 + (63 - roaring_leading_zeroes(word));+                *value_out = it->index;+                return true;+            }+            return false;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint32_t num_values =+                minimum_uint32((uint32_t)(it->index + 1), count);+            // Walk backwards so GCC can vectorize the uint16->uint64 widen-or.+            const uint16_t *src = ac->array + it->index + 1;+            for (uint32_t i = 0; i < num_values; i++) {+                buf[i] = high48 | *--src;+            }+            *consumed += num_values;+            it->index -= num_values;+            if (it->index >= 0) {+                *value_out = ac->array[it->index];+                return true;+            }+            return false;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            do {+                uint32_t run_start = rc->runs[it->index].value;+                uint32_t num_values = minimum_uint32(*value_out - run_start + 1,+                                                     count - *consumed);+                for (uint32_t i = 0; i < num_values; i++) {+                    buf[i] = high48 | (*value_out - i);+                }+                *value_out -= num_values;+                buf += num_values;+                *consumed += num_values;++                if (*value_out < run_start || *value_out == UINT16_MAX) {+                    it->index--;+                    if (it->index >= 0) {+                        *value_out = rc->runs[it->index].value ++                                     rc->runs[it->index].length;+                    } else {+                        return false;+                    }+                }+            } while (*consumed < count);+            return true;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++CROARING_ALLOW_UNALIGNED+bool container_iterator_skip(const container_t *c, uint8_t typecode,+                             roaring_container_iterator_t *it,+                             uint32_t skip_count, uint32_t *consumed_count,+                             uint16_t *value_out) {+    uint32_t actually_skipped;+    bool has_value;+    skip_count = minimum_uint32(skip_count, (uint32_t)UINT16_MAX + 1);+    switch (typecode) {+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            actually_skipped =+                minimum_uint32(ac->cardinality - it->index, skip_count);+            it->index += actually_skipped;+            has_value = it->index < ac->cardinality;+            if (has_value) {+                *value_out = ac->array[it->index];+            }+            break;+        }+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);++            uint32_t remaining_skip = skip_count;+            uint32_t current_index = it->index;+            uint64_t word_mask = UINT64_MAX << (current_index % 64);+            has_value = false;++            for (uint32_t word_index = current_index / 64;+                 word_index < BITSET_CONTAINER_SIZE_IN_WORDS; word_index++) {+                uint64_t word = bc->words[word_index] & word_mask;+                word_mask = ~0;  // Only apply mask for the first word++                uint32_t bits_in_word = roaring_hamming(word);+                if (bits_in_word > remaining_skip) {+                    // Unset the lowest bit `remaining_skip` times+                    for (; remaining_skip > 0; --remaining_skip) {+                        word &= word - 1;+                    }+                    has_value = true;+                    *value_out = it->index =+                        roaring_trailing_zeroes(word) + word_index * 64;+                    break;+                }+                // Skip all set bits in this word+                remaining_skip -= bits_in_word;+            }+            actually_skipped = skip_count - remaining_skip;+            break;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);++            uint16_t current_value = *value_out;+            uint32_t remaining_skip = skip_count;+            int32_t run_index;++            // Process skips by iterating through runs+            for (run_index = it->index;+                 remaining_skip > 0 && run_index < rc->n_runs; run_index++) {+                // max value (inclusive) in current run+                uint32_t run_max_inc =+                    rc->runs[run_index].value + rc->runs[run_index].length;+                // Max to skip in this run (we can skip from the current value+                // to the last value in the run, plus one to move past this run)+                uint32_t max_skip_this_run = run_max_inc - current_value + 1;+                uint32_t consume =+                    minimum_uint32(remaining_skip, max_skip_this_run);+                remaining_skip -= consume;+                if (consume < max_skip_this_run) {+                    current_value += consume;+                    break;+                }+                // Skip past the end of this run, to the next if there is one+                if (run_index + 1 < rc->n_runs) {+                    current_value = rc->runs[run_index + 1].value;+                }+            }++            // Update final state+            it->index = run_index;+            actually_skipped = skip_count - remaining_skip;+            has_value = run_index < rc->n_runs;+            if (has_value) {+                *value_out = current_value;+            }+            break;+        }+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+    *consumed_count = actually_skipped;+    return has_value;+}++bool container_iterator_skip_backward(const container_t *c, uint8_t typecode,+                                      roaring_container_iterator_t *it,+                                      uint32_t skip_count,+                                      uint32_t *consumed_count,+                                      uint16_t *value_out) {+    uint32_t actually_skipped;+    bool has_value;+    skip_count = minimum_uint32(skip_count, (uint32_t)UINT16_MAX + 1);+    switch (typecode) {+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            // Allow skipping back to -1+            actually_skipped = minimum_uint32(it->index + 1, skip_count);+            it->index -= actually_skipped;+            has_value = it->index >= 0;+            if (has_value) {+                *value_out = ac->array[it->index];+            }+            break;+        }+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);++            uint32_t remaining_skip = skip_count;+            uint32_t current_index = it->index;+            uint64_t word_mask = UINT64_MAX >> (63 - (current_index % 64));+            has_value = false;++            // Start from the word containing current index and go backwards+            for (int32_t word_index = current_index / 64; word_index >= 0;+                 word_index--) {+                uint64_t word = bc->words[word_index] & word_mask;+                word_mask = ~0;  // Only apply mask for the first word++                uint32_t bits_in_word = roaring_hamming(word);+                if (bits_in_word > remaining_skip) {+                    // Unset the highest bit `remaining_skip` times+                    for (; remaining_skip > 0; --remaining_skip) {+                        uint64_t high_bit =+                            UINT64_C(1) << (63 - roaring_leading_zeroes(word));+                        // Clear the highest set bit+                        word &= ~high_bit;+                    }+                    has_value = true;+                    *value_out = it->index =+                        (63 - roaring_leading_zeroes(word)) + word_index * 64;+                    break;+                }+                // Skip all set bits in this word+                remaining_skip -= bits_in_word;+            }+            actually_skipped = skip_count - remaining_skip;+            break;+        }+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);++            uint16_t current_value = *value_out;+            uint32_t remaining_skip = skip_count;+            int32_t run_index;++            // Process skips by iterating through runs backwards+            for (run_index = it->index; remaining_skip > 0 && run_index >= 0;+                 run_index--) {+                // min value (inclusive) in current run+                uint32_t run_min_inc = rc->runs[run_index].value;+                // Max to skip in this run (we can skip from the current value+                // back to the first value in the run, plus one to move before+                // this run)+                uint32_t max_skip_this_run = current_value - run_min_inc + 1;+                uint32_t consume =+                    minimum_uint32(remaining_skip, max_skip_this_run);+                remaining_skip -= consume;+                if (consume < max_skip_this_run) {+                    current_value -= consume;+                    break;+                }+                // Skip past the beginning of this run, to the previous if there+                // is one+                if (run_index - 1 >= 0) {+                    current_value = rc->runs[run_index - 1].value ++                                    rc->runs[run_index - 1].length;+                }+            }++            // Update final state+            it->index = run_index;+            actually_skipped = skip_count - remaining_skip;+            has_value = run_index >= 0;+            if (has_value) {+                *value_out = current_value;+            }+            break;+        }+        default:+            assert(false);+            roaring_unreachable;+            return false;+    }+    *consumed_count = actually_skipped;+    return has_value;+}++uint16_t container_iterator_find_run_end(const container_t *c, uint8_t typecode,+                                         roaring_container_iterator_t *it,+                                         uint16_t *value, bool *has_more) {+    switch (typecode) {+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            uint16_t run_end =+                rc->runs[it->index].value + rc->runs[it->index].length;+            it->index++;+            if (it->index < rc->n_runs) {+                *has_more = true;+                *value = rc->runs[it->index].value;+            } else {+                *has_more = false;+            }+            return run_end;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint16_t v = *value;+            while (it->index + 1 < ac->cardinality &&+                   ac->array[it->index + 1] == (uint16_t)(v + 1)) {+                it->index++;+                v++;+            }+            it->index++;+            if (it->index < ac->cardinality) {+                *has_more = true;+                *value = ac->array[it->index];+            } else {+                *has_more = false;+            }+            return v;+        }+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            uint32_t pos = (uint32_t)*value + 1;+            uint16_t run_end;+            if (pos >= (1 << 16)) {+                *has_more = false;+                return UINT16_MAX;+            }+            uint32_t wordindex = pos / 64;+            uint64_t word = ~bc->words[wordindex] & (UINT64_MAX << (pos % 64));+            while (word == 0 &&+                   wordindex + 1 < BITSET_CONTAINER_SIZE_IN_WORDS) {+                wordindex++;+                word = ~bc->words[wordindex];+            }+            if (word != 0) {+                run_end = (uint16_t)(wordindex * 64 ++                                     roaring_trailing_zeroes(word) - 1);+            } else {+                run_end = UINT16_MAX;+            }+            uint32_t next_pos = (uint32_t)run_end + 1;+            if (next_pos >= (1 << 16)) {+                *has_more = false;+            } else {+                wordindex = next_pos / 64;+                word = bc->words[wordindex] & (UINT64_MAX << (next_pos % 64));+                while (word == 0 &&+                       wordindex + 1 < BITSET_CONTAINER_SIZE_IN_WORDS) {+                    wordindex++;+                    word = bc->words[wordindex];+                }+                if (word != 0) {+                    *has_more = true;+                    it->index = wordindex * 64 + roaring_trailing_zeroes(word);+                    *value = (uint16_t)it->index;+                } else {+                    *has_more = false;+                }+            }+            return run_end;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++uint16_t container_iterator_find_run_start(const container_t *c,+                                           uint8_t typecode,+                                           roaring_container_iterator_t *it,+                                           uint16_t *value, bool *has_more) {+    switch (typecode) {+        case RUN_CONTAINER_TYPE: {+            const run_container_t *rc = const_CAST_run(c);+            uint16_t run_start = rc->runs[it->index].value;+            it->index--;+            if (it->index >= 0) {+                *has_more = true;+                *value = rc->runs[it->index].value + rc->runs[it->index].length;+            } else {+                *has_more = false;+            }+            return run_start;+        }+        case ARRAY_CONTAINER_TYPE: {+            const array_container_t *ac = const_CAST_array(c);+            uint16_t v = *value;+            while (it->index > 0 &&+                   ac->array[it->index - 1] == (uint16_t)(v - 1)) {+                it->index--;+                v--;+            }+            it->index--;+            if (it->index >= 0) {+                *has_more = true;+                *value = ac->array[it->index];+            } else {+                *has_more = false;+            }+            return v;+        }+        case BITSET_CONTAINER_TYPE: {+            const bitset_container_t *bc = const_CAST_bitset(c);+            if (*value == 0) {+                *has_more = false;+                return 0;+            }+            uint32_t pos = (uint32_t)*value - 1;+            int32_t wordindex = (int32_t)(pos / 64);+            uint64_t word =+                ~bc->words[wordindex] & (UINT64_MAX >> (63 - (pos % 64)));+            while (word == 0 && --wordindex >= 0) {+                word = ~bc->words[wordindex];+            }+            uint16_t run_start;+            if (word != 0) {+                run_start = (uint16_t)(wordindex * 64 ++                                       (63 - roaring_leading_zeroes(word)) + 1);+            } else {+                run_start = 0;+            }+            if (run_start == 0) {+                *has_more = false;+            } else {+                int32_t prev_pos = (int32_t)run_start - 1;+                wordindex = prev_pos / 64;+                word = bc->words[wordindex] &+                       (UINT64_MAX >> (63 - (prev_pos % 64)));+                while (word == 0 && --wordindex >= 0) {+                    word = bc->words[wordindex];+                }+                if (word != 0) {+                    *has_more = true;+                    it->index =+                        wordindex * 64 + (63 - roaring_leading_zeroes(word));+                    *value = (uint16_t)it->index;+                } else {+                    *has_more = false;+                }+            }+            return run_start;+        }+        default:+            assert(false);+            roaring_unreachable;+            return 0;+    }+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif++#undef ROARING_INIT_ROARING_CONTAINER_ITERATOR_T
+ cbits/CRoaring/src/containers/convert.c view
@@ -0,0 +1,363 @@+#include <stdio.h>++#include <roaring/bitset_util.h>+#include <roaring/containers/containers.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/perfparameters.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++// file contains grubby stuff that must know impl. details of all container+// types.+CROARING_ALLOW_UNALIGNED+bitset_container_t *bitset_container_from_array(const array_container_t *ac) {+    bitset_container_t *ans = bitset_container_create();+    int limit = array_container_cardinality(ac);+    for (int i = 0; i < limit; ++i) bitset_container_set(ans, ac->array[i]);+    return ans;+}++CROARING_ALLOW_UNALIGNED+bitset_container_t *bitset_container_from_run(const run_container_t *arr) {+    int card = run_container_cardinality(arr);+    bitset_container_t *answer = bitset_container_create();+    for (int rlepos = 0; rlepos < arr->n_runs; ++rlepos) {+        rle16_t vl = arr->runs[rlepos];+        bitset_set_lenrange(answer->words, vl.value, vl.length);+    }+    answer->cardinality = card;+    return answer;+}++array_container_t *array_container_from_run(const run_container_t *arr) {+    array_container_t *answer =+        array_container_create_given_capacity(run_container_cardinality(arr));+    answer->cardinality = 0;+    for (int rlepos = 0; rlepos < arr->n_runs; ++rlepos) {+        int run_start = arr->runs[rlepos].value;+        int run_end = run_start + arr->runs[rlepos].length;++        for (int run_value = run_start; run_value <= run_end; ++run_value) {+            answer->array[answer->cardinality++] = (uint16_t)run_value;+        }+    }+    return answer;+}++array_container_t *array_container_from_bitset(const bitset_container_t *bits) {+    array_container_t *result =+        array_container_create_given_capacity(bits->cardinality);+    result->cardinality = bits->cardinality;+#if CROARING_IS_X64+#if CROARING_COMPILER_SUPPORTS_AVX512+    // Every caller first checks that the cardinality fits an array container+    // (<= DEFAULT_MAX_SIZE), so the input here is sparse: under 6.25% density.+    // The vector decoder is still faster than the scalar loop at every+    // cardinality, from 16 values up, whether the values are spread out,+    // clustered or in runs: see microbenchmarks/bitset_to_array_bench.cpp.+    // Do not gate it on the cardinality without measuring there: a gate at+    // 1024 values made this conversion up to 2.3x slower below the gate.+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX512) {+        bitset_extract_setbits_avx512_uint16(+            bits->words, BITSET_CONTAINER_SIZE_IN_WORDS, result->array,+            bits->cardinality, 0);+    } else+#endif+    {+        //  sse version ends up being slower here+        // (bitset_extract_setbits_sse_uint16)+        // because of the sparsity of the data+        bitset_extract_setbits_uint16(+            bits->words, BITSET_CONTAINER_SIZE_IN_WORDS, result->array, 0);+    }+#else+    // If the system is not x64, then we have no accelerated function.+    bitset_extract_setbits_uint16(bits->words, BITSET_CONTAINER_SIZE_IN_WORDS,+                                  result->array, 0);+#endif++    return result;+}++/* assumes that container has adequate space.  Run from [s,e] (inclusive) */+static void add_run(run_container_t *rc, int s, int e) {+    rc->runs[rc->n_runs].value = s;+    rc->runs[rc->n_runs].length = e - s;+    rc->n_runs++;+}++run_container_t *run_container_from_array(const array_container_t *c) {+    int32_t n_runs = array_container_number_of_runs(c);+    run_container_t *answer = run_container_create_given_capacity(n_runs);+    int prev = -2;+    int run_start = -1;+    int32_t card = c->cardinality;+    if (card == 0) return answer;+    for (int i = 0; i < card; ++i) {+        const uint16_t cur_val = c->array[i];+        if (cur_val != prev + 1) {+            // new run starts; flush old one, if any+            if (run_start != -1) add_run(answer, run_start, prev);+            run_start = cur_val;+        }+        prev = c->array[i];+    }+    // now prev is the last seen value+    add_run(answer, run_start, prev);+    // assert(run_container_cardinality(answer) == c->cardinality);+    return answer;+}++/**+ * Convert the runcontainer to either a Bitmap or an Array Container, depending+ * on the cardinality.  Frees the container.+ * Allocates and returns new container, which caller is responsible for freeing.+ * It does not free the run container.+ */+container_t *convert_to_bitset_or_array_container(run_container_t *rc,+                                                  int32_t card,+                                                  uint8_t *resulttype) {+    if (card <= DEFAULT_MAX_SIZE) {+        array_container_t *answer = array_container_create_given_capacity(card);+        answer->cardinality = 0;+        for (int rlepos = 0; rlepos < rc->n_runs; ++rlepos) {+            uint16_t run_start = rc->runs[rlepos].value;+            uint16_t run_end = run_start + rc->runs[rlepos].length;+            for (uint16_t run_value = run_start; run_value < run_end;+                 ++run_value) {+                answer->array[answer->cardinality++] = run_value;+            }+            answer->array[answer->cardinality++] = run_end;+        }+        assert(card == answer->cardinality);+        *resulttype = ARRAY_CONTAINER_TYPE;+        // run_container_free(r);+        return answer;+    }+    bitset_container_t *answer = bitset_container_create();+    for (int rlepos = 0; rlepos < rc->n_runs; ++rlepos) {+        uint16_t run_start = rc->runs[rlepos].value;+        bitset_set_lenrange(answer->words, run_start, rc->runs[rlepos].length);+    }+    answer->cardinality = card;+    *resulttype = BITSET_CONTAINER_TYPE;+    // run_container_free(r);+    return answer;+}++/* Converts a run container to either an array or a bitset, IF it saves space.+ */+/* If a conversion occurs, the caller is responsible to free the original+ * container and+ * he becomes responsible to free the new one. */+container_t *convert_run_to_efficient_container(run_container_t *c,+                                                uint8_t *typecode_after) {+    int32_t size_as_run_container =+        run_container_serialized_size_in_bytes(c->n_runs);++    int32_t size_as_bitset_container =+        bitset_container_serialized_size_in_bytes();+    int32_t card = run_container_cardinality(c);+    int32_t size_as_array_container =+        array_container_serialized_size_in_bytes(card);++    int32_t min_size_non_run =+        size_as_bitset_container < size_as_array_container+            ? size_as_bitset_container+            : size_as_array_container;+    if (size_as_run_container <= min_size_non_run) {  // no conversion+        *typecode_after = RUN_CONTAINER_TYPE;+        return c;+    }+    if (card <= DEFAULT_MAX_SIZE) {+        // to array+        array_container_t *answer = array_container_create_given_capacity(card);+        answer->cardinality = 0;+        for (int rlepos = 0; rlepos < c->n_runs; ++rlepos) {+            int run_start = c->runs[rlepos].value;+            int run_end = run_start + c->runs[rlepos].length;++            for (int run_value = run_start; run_value <= run_end; ++run_value) {+                answer->array[answer->cardinality++] = (uint16_t)run_value;+            }+        }+        *typecode_after = ARRAY_CONTAINER_TYPE;+        return answer;+    }++    // else to bitset+    bitset_container_t *answer = bitset_container_create();++    for (int rlepos = 0; rlepos < c->n_runs; ++rlepos) {+        int start = c->runs[rlepos].value;+        int end = start + c->runs[rlepos].length;+        bitset_set_range(answer->words, start, end + 1);+    }+    answer->cardinality = card;+    *typecode_after = BITSET_CONTAINER_TYPE;+    return answer;+}++// like convert_run_to_efficient_container but frees the old result if needed+container_t *convert_run_to_efficient_container_and_free(+    run_container_t *c, uint8_t *typecode_after) {+    container_t *answer = convert_run_to_efficient_container(c, typecode_after);+    if (answer != c) run_container_free(c);+    return answer;+}++/* once converted, the original container is disposed here, rather than+   in roaring_array+*/++// TODO: split into run-  array-  and bitset-  subfunctions for sanity;+// a few function calls won't really matter.++container_t *convert_run_optimize(container_t *c, uint8_t typecode_original,+                                  uint8_t *typecode_after) {+    if (typecode_original == RUN_CONTAINER_TYPE) {+        container_t *newc =+            convert_run_to_efficient_container(CAST_run(c), typecode_after);+        if (newc != c) {+            container_free(c, typecode_original);+        }+        return newc;+    } else if (typecode_original == ARRAY_CONTAINER_TYPE) {+        // it might need to be converted to a run container.+        array_container_t *c_qua_array = CAST_array(c);+        int32_t n_runs = array_container_number_of_runs(c_qua_array);+        int32_t size_as_run_container =+            run_container_serialized_size_in_bytes(n_runs);+        int32_t card = array_container_cardinality(c_qua_array);+        int32_t size_as_array_container =+            array_container_serialized_size_in_bytes(card);++        if (size_as_run_container >= size_as_array_container) {+            *typecode_after = ARRAY_CONTAINER_TYPE;+            return c;+        }+        // else convert array to run container+        run_container_t *answer = run_container_create_given_capacity(n_runs);+        int prev = -2;+        int run_start = -1;++        assert(card > 0);+        for (int i = 0; i < card; ++i) {+            uint16_t cur_val = c_qua_array->array[i];+            if (cur_val != prev + 1) {+                // new run starts; flush old one, if any+                if (run_start != -1) add_run(answer, run_start, prev);+                run_start = cur_val;+            }+            prev = c_qua_array->array[i];+        }+        assert(run_start >= 0);+        // now prev is the last seen value+        add_run(answer, run_start, prev);+        *typecode_after = RUN_CONTAINER_TYPE;+        array_container_free(c_qua_array);+        return answer;+    } else if (typecode_original ==+               BITSET_CONTAINER_TYPE) {  // run conversions on bitset+        // does bitset need conversion to run?+        bitset_container_t *c_qua_bitset = CAST_bitset(c);+        int32_t n_runs = bitset_container_number_of_runs(c_qua_bitset);+        int32_t size_as_run_container =+            run_container_serialized_size_in_bytes(n_runs);+        int32_t size_as_bitset_container =+            bitset_container_serialized_size_in_bytes();++        if (size_as_bitset_container <= size_as_run_container) {+            // no conversion needed.+            *typecode_after = BITSET_CONTAINER_TYPE;+            return c;+        }+        // bitset to runcontainer (ported from Java  RunContainer(+        // BitmapContainer bc, int nbrRuns))+        assert(n_runs > 0);  // no empty bitmaps+        run_container_t *answer = run_container_create_given_capacity(n_runs);++        int long_ctr = 0;+        uint64_t cur_word = c_qua_bitset->words[0];+        while (true) {+            while (cur_word == UINT64_C(0) &&+                   long_ctr < BITSET_CONTAINER_SIZE_IN_WORDS - 1)+                cur_word = c_qua_bitset->words[++long_ctr];++            if (cur_word == UINT64_C(0)) {+                bitset_container_free(c_qua_bitset);+                *typecode_after = RUN_CONTAINER_TYPE;+                return answer;+            }++            int local_run_start = roaring_trailing_zeroes(cur_word);+            int run_start = local_run_start + 64 * long_ctr;+            uint64_t cur_word_with_1s = cur_word | (cur_word - 1);++            int run_end = 0;+            while (cur_word_with_1s == UINT64_C(0xFFFFFFFFFFFFFFFF) &&+                   long_ctr < BITSET_CONTAINER_SIZE_IN_WORDS - 1)+                cur_word_with_1s = c_qua_bitset->words[++long_ctr];++            if (cur_word_with_1s == UINT64_C(0xFFFFFFFFFFFFFFFF)) {+                run_end = 64 + long_ctr * 64;  // exclusive, I guess+                add_run(answer, run_start, run_end - 1);+                bitset_container_free(c_qua_bitset);+                *typecode_after = RUN_CONTAINER_TYPE;+                return answer;+            }+            int local_run_end = roaring_trailing_zeroes(~cur_word_with_1s);+            run_end = local_run_end + long_ctr * 64;+            add_run(answer, run_start, run_end - 1);+            cur_word = cur_word_with_1s & (cur_word_with_1s + 1);+        }+        return answer;+    } else {+        assert(false);+        roaring_unreachable;+        return NULL;+    }+}++container_t *container_from_run_range(const run_container_t *run, uint32_t min,+                                      uint32_t max, uint8_t *typecode_after) {+    // We expect most of the time to end up with a bitset container+    bitset_container_t *bitset = bitset_container_create();+    *typecode_after = BITSET_CONTAINER_TYPE;+    int32_t union_cardinality = 0;+    for (int32_t i = 0; i < run->n_runs; ++i) {+        uint32_t rle_min = run->runs[i].value;+        uint32_t rle_max = rle_min + run->runs[i].length;+        bitset_set_lenrange(bitset->words, rle_min, rle_max - rle_min);+        union_cardinality += run->runs[i].length + 1;+    }+    union_cardinality += max - min + 1;+    union_cardinality -=+        bitset_lenrange_cardinality(bitset->words, min, max - min);+    bitset_set_lenrange(bitset->words, min, max - min);+    bitset->cardinality = union_cardinality;+    if (bitset->cardinality <= DEFAULT_MAX_SIZE) {+        // we need to convert to an array container+        array_container_t *array = array_container_from_bitset(bitset);+        *typecode_after = ARRAY_CONTAINER_TYPE;+        bitset_container_free(bitset);+        return array;+    }+    return bitset;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_andnot.c view
@@ -0,0 +1,525 @@+/*+ * mixed_andnot.c.  More methods since operation is not symmetric,+ * except no "wide" andnot , so no lazy options motivated.+ */++#include <assert.h>+#include <string.h>++#include <roaring/array_util.h>+#include <roaring/bitset_util.h>+#include <roaring/containers/containers.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_andnot.h>+#include <roaring/containers/perfparameters.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst, a valid array container that could be the same as dst.*/+void array_bitset_container_andnot(const array_container_t *src_1,+                                   const bitset_container_t *src_2,+                                   array_container_t *dst) {+    // follows Java implementation as of June 2016+    if (dst->capacity < src_1->cardinality) {+        array_container_grow(dst, src_1->cardinality, false);+    }+    int32_t newcard = 0;+    const int32_t origcard = src_1->cardinality;+    const uint16_t *array = src_1->array;+    const uint64_t *words = src_2->words;+    uint16_t *out = dst->array;+    for (int i = 0; i < origcard; ++i) {+        uint16_t key = array[i];+        out[newcard] = key;+        newcard += 1 - ((words[key >> 6] >> (key & 63)) & 1);+    }+    dst->cardinality = newcard;+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * src_1 */++void array_bitset_container_iandnot(array_container_t *src_1,+                                    const bitset_container_t *src_2) {+    array_bitset_container_andnot(src_1, src_2, src_1);+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst, which does not initially have a valid container.+ * Return true for a bitset result; false for array+ */++bool bitset_array_container_andnot(const bitset_container_t *src_1,+                                   const array_container_t *src_2,+                                   container_t **dst) {+    // Java did this directly, but we have option of asm or avx+    bitset_container_t *result = bitset_container_create();+    bitset_container_copy(src_1, result);+    result->cardinality =+        (int32_t)bitset_clear_list(result->words, (uint64_t)result->cardinality,+                                   src_2->array, (uint64_t)src_2->cardinality);++    // do required type conversions.+    if (result->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(result);+        bitset_container_free(result);+        return false;+    }+    *dst = result;+    return true;+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_array_container_iandnot(bitset_container_t *src_1,+                                    const array_container_t *src_2,+                                    container_t **dst) {+    *dst = src_1;+    src_1->cardinality =+        (int32_t)bitset_clear_list(src_1->words, (uint64_t)src_1->cardinality,+                                   src_2->array, (uint64_t)src_2->cardinality);++    if (src_1->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(src_1);+        bitset_container_free(src_1);+        return false;  // not bitset+    } else+        return true;+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_andnot(const run_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst) {+    // follows the Java implementation as of June 2016+    int card = run_container_cardinality(src_1);+    if (card <= DEFAULT_MAX_SIZE) {+        // must be an array+        array_container_t *answer = array_container_create_given_capacity(card);+        answer->cardinality = 0;+        for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+            rle16_t rle = src_1->runs[rlepos];+            for (int run_value = rle.value; run_value <= rle.value + rle.length;+                 ++run_value) {+                if (!bitset_container_get(src_2, (uint16_t)run_value)) {+                    answer->array[answer->cardinality++] = (uint16_t)run_value;+                }+            }+        }+        *dst = answer;+        return false;+    } else {  // we guess it will be a bitset, though have to check guess when+              // done+        bitset_container_t *answer = bitset_container_clone(src_2);++        uint32_t last_pos = 0;+        for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+            rle16_t rle = src_1->runs[rlepos];++            uint32_t start = rle.value;+            uint32_t end = start + rle.length + 1;+            bitset_reset_range(answer->words, last_pos, start);+            bitset_flip_range(answer->words, start, end);+            last_pos = end;+        }+        bitset_reset_range(answer->words, last_pos, (uint32_t)(1 << 16));++        answer->cardinality = bitset_container_compute_cardinality(answer);++        if (answer->cardinality <= DEFAULT_MAX_SIZE) {+            *dst = array_container_from_bitset(answer);+            bitset_container_free(answer);+            return false;  // not bitset+        }+        *dst = answer;+        return true;  // bitset+    }+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_iandnot(run_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  container_t **dst) {+    // dummy implementation+    bool ans = run_bitset_container_andnot(src_1, src_2, dst);+    run_container_free(src_1);+    return ans;+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset").  dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool bitset_run_container_andnot(const bitset_container_t *src_1,+                                 const run_container_t *src_2,+                                 container_t **dst) {+    // follows Java implementation+    bitset_container_t *result = bitset_container_create();++    bitset_container_copy(src_1, result);+    for (int32_t rlepos = 0; rlepos < src_2->n_runs; ++rlepos) {+        rle16_t rle = src_2->runs[rlepos];+        bitset_reset_range(result->words, rle.value,+                           rle.value + rle.length + UINT32_C(1));+    }+    result->cardinality = bitset_container_compute_cardinality(result);++    if (result->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(result);+        bitset_container_free(result);+        return false;  // not bitset+    }+    *dst = result;+    return true;  // bitset+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_run_container_iandnot(bitset_container_t *src_1,+                                  const run_container_t *src_2,+                                  container_t **dst) {+    *dst = src_1;++    for (int32_t rlepos = 0; rlepos < src_2->n_runs; ++rlepos) {+        rle16_t rle = src_2->runs[rlepos];+        bitset_reset_range(src_1->words, rle.value,+                           rle.value + rle.length + UINT32_C(1));+    }+    src_1->cardinality = bitset_container_compute_cardinality(src_1);++    if (src_1->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(src_1);+        bitset_container_free(src_1);+        return false;  // not bitset+    } else+        return true;+}++/* helper. a_out must be a valid array container with adequate capacity.+ * Returns the cardinality of the output container. Partly Based on Java+ * implementation Util.unsignedDifference.+ *+ * TODO: Util.unsignedDifference does not use advanceUntil.  Is it cheaper+ * to avoid advanceUntil?+ */++static int run_array_array_subtract(const run_container_t *rc,+                                    const array_container_t *a_in,+                                    array_container_t *a_out) {+    int out_card = 0;+    int32_t in_array_pos =+        -1;  // since advanceUntil always assumes we start the search AFTER this++    for (int rlepos = 0; rlepos < rc->n_runs; rlepos++) {+        int32_t start = rc->runs[rlepos].value;+        int32_t end = start + rc->runs[rlepos].length + 1;++        in_array_pos = advanceUntil(a_in->array, in_array_pos,+                                    a_in->cardinality, (uint16_t)start);++        if (in_array_pos >= a_in->cardinality) {  // run has no items subtracted+            for (int32_t i = start; i < end; ++i)+                a_out->array[out_card++] = (uint16_t)i;+        } else {+            uint16_t next_nonincluded = a_in->array[in_array_pos];+            if (next_nonincluded >= end) {+                // another case when run goes unaltered+                for (int32_t i = start; i < end; ++i)+                    a_out->array[out_card++] = (uint16_t)i;+                in_array_pos--;  // ensure we see this item again if necessary+            } else {+                for (int32_t i = start; i < end; ++i)+                    if (i != next_nonincluded)+                        a_out->array[out_card++] = (uint16_t)i;+                    else  // 0 should ensure  we don't match+                        next_nonincluded =+                            (in_array_pos + 1 >= a_in->cardinality)+                                ? 0+                                : a_in->array[++in_array_pos];+                in_array_pos--;  // see again+            }+        }+    }+    return out_card;+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any type of container.+ */++int run_array_container_andnot(const run_container_t *src_1,+                               const array_container_t *src_2,+                               container_t **dst) {+    // follows the Java impl as of June 2016++    int card = run_container_cardinality(src_1);+    const int arbitrary_threshold = 32;++    if (card <= arbitrary_threshold) {+        if (src_2->cardinality == 0) {+            *dst = run_container_clone(src_1);+            return RUN_CONTAINER_TYPE;+        }+        // Java's "lazyandNot.toEfficientContainer" thing+        run_container_t *answer = run_container_create_given_capacity(+            card + array_container_cardinality(src_2));++        int rlepos = 0;+        int xrlepos = 0;  // "x" is src_2+        rle16_t rle = src_1->runs[rlepos];+        int32_t start = rle.value;+        int32_t end = start + rle.length + 1;+        int32_t xstart = src_2->array[xrlepos];++        while ((rlepos < src_1->n_runs) && (xrlepos < src_2->cardinality)) {+            if (end <= xstart) {+                // output the first run+                answer->runs[answer->n_runs++] =+                    CROARING_MAKE_RLE16(start, end - start - 1);+                rlepos++;+                if (rlepos < src_1->n_runs) {+                    start = src_1->runs[rlepos].value;+                    end = start + src_1->runs[rlepos].length + 1;+                }+            } else if (xstart + 1 <= start) {+                // exit the second run+                xrlepos++;+                if (xrlepos < src_2->cardinality) {+                    xstart = src_2->array[xrlepos];+                }+            } else {+                if (start < xstart) {+                    answer->runs[answer->n_runs++] =+                        CROARING_MAKE_RLE16(start, xstart - start - 1);+                }+                if (xstart + 1 < end) {+                    start = xstart + 1;+                } else {+                    rlepos++;+                    if (rlepos < src_1->n_runs) {+                        start = src_1->runs[rlepos].value;+                        end = start + src_1->runs[rlepos].length + 1;+                    }+                }+            }+        }+        if (rlepos < src_1->n_runs) {+            answer->runs[answer->n_runs++] =+                CROARING_MAKE_RLE16(start, end - start - 1);+            rlepos++;+            if (rlepos < src_1->n_runs) {+                memcpy(answer->runs + answer->n_runs, src_1->runs + rlepos,+                       (src_1->n_runs - rlepos) * sizeof(rle16_t));+                answer->n_runs += (src_1->n_runs - rlepos);+            }+        }+        uint8_t return_type;+        *dst = convert_run_to_efficient_container(answer, &return_type);+        if (answer != *dst) run_container_free(answer);+        return return_type;+    }+    // else it's a bitmap or array++    if (card <= DEFAULT_MAX_SIZE) {+        array_container_t *ac = array_container_create_given_capacity(card);+        // nb Java code used a generic iterator-based merge to compute+        // difference+        ac->cardinality = run_array_array_subtract(src_1, src_2, ac);+        *dst = ac;+        return ARRAY_CONTAINER_TYPE;+    }+    bitset_container_t *ans = bitset_container_from_run(src_1);+    bool result_is_bitset = bitset_array_container_iandnot(ans, src_2, dst);+    return (result_is_bitset ? BITSET_CONTAINER_TYPE : ARRAY_CONTAINER_TYPE);+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++int run_array_container_iandnot(run_container_t *src_1,+                                const array_container_t *src_2,+                                container_t **dst) {+    // dummy implementation same as June 2016 Java+    int ans = run_array_container_andnot(src_1, src_2, dst);+    run_container_free(src_1);+    return ans;+}++/* dst must be a valid array container, allowed to be src_1 */++void array_run_container_andnot(const array_container_t *src_1,+                                const run_container_t *src_2,+                                array_container_t *dst) {+    // basically following Java impl as of June 2016+    if (src_1->cardinality > dst->capacity) {+        array_container_grow(dst, src_1->cardinality, false);+    }++    if (src_2->n_runs == 0) {+        memmove(dst->array, src_1->array,+                sizeof(uint16_t) * src_1->cardinality);+        dst->cardinality = src_1->cardinality;+        return;+    }+    int32_t run_start = src_2->runs[0].value;+    int32_t run_end = run_start + src_2->runs[0].length;+    int which_run = 0;++    uint16_t val = 0;+    int dest_card = 0;+    for (int i = 0; i < src_1->cardinality; ++i) {+        val = src_1->array[i];+        if (val < run_start)+            dst->array[dest_card++] = val;+        else if (val <= run_end) {+            ;  // omitted item+        } else {+            do {+                if (which_run + 1 < src_2->n_runs) {+                    ++which_run;+                    run_start = src_2->runs[which_run].value;+                    run_end = run_start + src_2->runs[which_run].length;++                } else+                    run_start = run_end = (1 << 16) + 1;+            } while (val > run_end);+            --i;+        }+    }+    dst->cardinality = dest_card;+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++void array_run_container_iandnot(array_container_t *src_1,+                                 const run_container_t *src_2) {+    array_run_container_andnot(src_1, src_2, src_1);+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int run_run_container_andnot(const run_container_t *src_1,+                             const run_container_t *src_2, container_t **dst) {+    run_container_t *ans = run_container_create();+    run_container_andnot(src_1, src_2, ans);+    uint8_t typecode_after;+    *dst = convert_run_to_efficient_container_and_free(ans, &typecode_after);+    return typecode_after;+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++int run_run_container_iandnot(run_container_t *src_1,+                              const run_container_t *src_2, container_t **dst) {+    // following Java impl as of June 2016 (dummy)+    int ans = run_run_container_andnot(src_1, src_2, dst);+    run_container_free(src_1);+    return ans;+}++/*+ * dst is a valid array container and may be the same as src_1+ */++void array_array_container_andnot(const array_container_t *src_1,+                                  const array_container_t *src_2,+                                  array_container_t *dst) {+    array_container_andnot(src_1, src_2, dst);+}++/* inplace array-array andnot will always be able to reuse the space of+ * src_1 */+void array_array_container_iandnot(array_container_t *src_1,+                                   const array_container_t *src_2) {+    array_container_andnot(src_1, src_2, src_1);+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially). Return value is+ * "dst is a bitset"+ */++bool bitset_bitset_container_andnot(const bitset_container_t *src_1,+                                    const bitset_container_t *src_2,+                                    container_t **dst) {+    bitset_container_t *ans = bitset_container_create();+    int card = bitset_container_andnot(src_1, src_2, ans);+    if (card <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(ans);+        bitset_container_free(ans);+        return false;  // not bitset+    } else {+        *dst = ans;+        return true;+    }+}++/* Compute the andnot of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_bitset_container_iandnot(bitset_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     container_t **dst) {+    int card = bitset_container_andnot(src_1, src_2, src_1);+    if (card <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(src_1);+        bitset_container_free(src_1);+        return false;  // not bitset+    } else {+        *dst = src_1;+        return true;+    }+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_equal.c view
@@ -0,0 +1,88 @@+#include <roaring/containers/mixed_equal.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++bool array_container_equal_bitset(const array_container_t* container1,+                                  const bitset_container_t* container2) {+    if (container2->cardinality != BITSET_UNKNOWN_CARDINALITY) {+        if (container2->cardinality != container1->cardinality) {+            return false;+        }+    }+    int32_t pos = 0;+    for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+        uint64_t w = container2->words[i];+        while (w != 0) {+            uint64_t t = w & (~w + 1);+            uint16_t r = i * 64 + roaring_trailing_zeroes(w);+            if (pos >= container1->cardinality) {+                return false;+            }+            if (container1->array[pos] != r) {+                return false;+            }+            ++pos;+            w ^= t;+        }+    }+    return (pos == container1->cardinality);+}++bool run_container_equals_array(const run_container_t* container1,+                                const array_container_t* container2) {+    if (run_container_cardinality(container1) != container2->cardinality)+        return false;+    int32_t pos = 0;+    for (int i = 0; i < container1->n_runs; ++i) {+        const uint32_t run_start = container1->runs[i].value;+        const uint32_t le = container1->runs[i].length;++        if (container2->array[pos] != run_start) {+            return false;+        }++        if (container2->array[pos + le] != run_start + le) {+            return false;+        }++        pos += le + 1;+    }+    return true;+}++bool run_container_equals_bitset(const run_container_t* container1,+                                 const bitset_container_t* container2) {+    int run_card = run_container_cardinality(container1);+    int bitset_card = (container2->cardinality != BITSET_UNKNOWN_CARDINALITY)+                          ? container2->cardinality+                          : bitset_container_compute_cardinality(container2);+    if (bitset_card != run_card) {+        return false;+    }++    for (int32_t i = 0; i < container1->n_runs; i++) {+        uint32_t begin = container1->runs[i].value;+        if (container1->runs[i].length) {+            uint32_t end = begin + container1->runs[i].length + 1;+            if (!bitset_container_contains_range(container2, begin, end)) {+                return false;+            }+        } else {+            if (!bitset_container_contains(container2, begin)) {+                return false;+            }+        }+    }++    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_intersection.c view
@@ -0,0 +1,379 @@+/*+ * mixed_intersection.c+ *+ */++#include <roaring/array_util.h>+#include <roaring/bitset_util.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_intersection.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the intersection of src_1 and src_2 and write the result to+ * dst.  */+void array_bitset_container_intersection(const array_container_t *src_1,+                                         const bitset_container_t *src_2,+                                         array_container_t *dst) {+    if (dst->capacity < src_1->cardinality) {+        array_container_grow(dst, src_1->cardinality, false);+    }+    int32_t newcard = 0;  // dst could be src_1+    const int32_t origcard = src_1->cardinality;+    const uint16_t *array = src_1->array;+    const uint64_t *words = src_2->words;+    uint16_t *out = dst->array;+    for (int i = 0; i < origcard; ++i) {+        uint16_t key = array[i];+        // this branchless approach is much faster...+        out[newcard] = key;+        newcard += (words[key >> 6] >> (key & 63)) & 1;+        /**+         * we could do it this way instead...+         * if (bitset_container_contains(src_2, key)) {+         * dst->array[newcard++] = key;+         * }+         * but if the result is unpredictible, the processor generates+         * many mispredicted branches.+         * Difference can be huge (from 3 cycles when predictible all the way+         * to 16 cycles when unpredictible.+         * See+         * https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/blob/master/extra/bitset/c/arraybitsetintersection.c+         */+    }+    dst->cardinality = newcard;+}++/* Compute the size of the intersection of src_1 and src_2. */+int array_bitset_container_intersection_cardinality(+    const array_container_t *src_1, const bitset_container_t *src_2) {+    int32_t newcard = 0;+    const int32_t origcard = src_1->cardinality;+    for (int i = 0; i < origcard; ++i) {+        uint16_t key = src_1->array[i];+        newcard += bitset_container_contains(src_2, key);+    }+    return newcard;+}++bool array_bitset_container_intersect(const array_container_t *src_1,+                                      const bitset_container_t *src_2) {+    const int32_t origcard = src_1->cardinality;+    for (int i = 0; i < origcard; ++i) {+        uint16_t key = src_1->array[i];+        if (bitset_container_contains(src_2, key)) return true;+    }+    return false;+}++/* Compute the intersection of src_1 and src_2 and write the result to+ * dst. It is allowed for dst to be equal to src_1. We assume that dst is a+ * valid container. */+void array_run_container_intersection(const array_container_t *src_1,+                                      const run_container_t *src_2,+                                      array_container_t *dst) {+    if (run_container_is_full(src_2)) {+        if (dst != src_1) array_container_copy(src_1, dst);+        return;+    }+    if (dst->capacity < src_1->cardinality) {+        array_container_grow(dst, src_1->cardinality, false);+    }+    if (src_2->n_runs == 0) {+        return;+    }+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    rle16_t rle = src_2->runs[rlepos];+    int32_t newcard = 0;+    while (arraypos < src_1->cardinality) {+        const uint16_t arrayval = src_1->array[arraypos];+        while (rle.value + rle.length <+               arrayval) {  // this will frequently be false+            ++rlepos;+            if (rlepos == src_2->n_runs) {+                dst->cardinality = newcard;+                return;  // we are done+            }+            rle = src_2->runs[rlepos];+        }+        if (rle.value > arrayval) {+            arraypos = advanceUntil(src_1->array, arraypos, src_1->cardinality,+                                    rle.value);+        } else {+            dst->array[newcard] = arrayval;+            newcard++;+            arraypos++;+        }+    }+    dst->cardinality = newcard;+}++/* Compute the intersection of src_1 and src_2 and write the result to+ * *dst. If the result is true then the result is a bitset_container_t+ * otherwise is a array_container_t. If *dst ==  src_2, an in-place processing+ * is attempted.*/+bool run_bitset_container_intersection(const run_container_t *src_1,+                                       const bitset_container_t *src_2,+                                       container_t **dst) {+    if (run_container_is_full(src_1)) {+        if (*dst != src_2) *dst = bitset_container_clone(src_2);+        return true;+    }+    int32_t card = run_container_cardinality(src_1);+    if (card <= DEFAULT_MAX_SIZE) {+        // result can only be an array (assuming that we never make a+        // RunContainer)+        if (card > src_2->cardinality) {+            card = src_2->cardinality;+        }+        array_container_t *answer = array_container_create_given_capacity(card);+        *dst = answer;+        if (*dst == NULL) {+            return false;+        }+        const rle16_t *runs = src_1->runs;+        const int32_t n_runs = src_1->n_runs;+        const uint64_t *words = src_2->words;+        uint16_t *out = answer->array;+        int32_t newcard = 0;+        for (int32_t rlepos = 0; rlepos < n_runs; ++rlepos) {+            rle16_t rle = runs[rlepos];+            uint32_t endofrun = (uint32_t)rle.value + rle.length;+            for (uint32_t runValue = rle.value; runValue <= endofrun;+                 ++runValue) {+                out[newcard] = (uint16_t)runValue;+                newcard += (words[runValue >> 6] >> (runValue & 63)) & 1;+            }+        }+        answer->cardinality = newcard;+        return false;+    }+    if (*dst == src_2) {  // we attempt in-place+        bitset_container_t *answer = CAST_bitset(*dst);+        uint32_t start = 0;+        for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+            const rle16_t rle = src_1->runs[rlepos];+            uint32_t end = rle.value;+            bitset_reset_range(src_2->words, start, end);++            start = end + rle.length + 1;+        }+        bitset_reset_range(src_2->words, start, UINT32_C(1) << 16);+        answer->cardinality = bitset_container_compute_cardinality(answer);+        if (src_2->cardinality > DEFAULT_MAX_SIZE) {+            return true;+        } else {+            array_container_t *newanswer = array_container_from_bitset(src_2);+            if (newanswer == NULL) {+                *dst = NULL;+                return false;+            }+            *dst = newanswer;+            return false;+        }+    } else {  // no inplace+        // we expect the answer to be a bitmap (if we are lucky)+        bitset_container_t *answer = bitset_container_clone(src_2);++        *dst = answer;+        if (answer == NULL) {+            return true;+        }+        uint32_t start = 0;+        for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+            const rle16_t rle = src_1->runs[rlepos];+            uint32_t end = rle.value;+            bitset_reset_range(answer->words, start, end);+            start = end + rle.length + 1;+        }+        bitset_reset_range(answer->words, start, UINT32_C(1) << 16);+        answer->cardinality = bitset_container_compute_cardinality(answer);++        if (answer->cardinality > DEFAULT_MAX_SIZE) {+            return true;+        } else {+            array_container_t *newanswer = array_container_from_bitset(answer);+            bitset_container_free(CAST_bitset(*dst));+            if (newanswer == NULL) {+                *dst = NULL;+                return false;+            }+            *dst = newanswer;+            return false;+        }+    }+}++/* Compute the size of the intersection between src_1 and src_2 . */+int array_run_container_intersection_cardinality(const array_container_t *src_1,+                                                 const run_container_t *src_2) {+    if (run_container_is_full(src_2)) {+        return src_1->cardinality;+    }+    if (src_2->n_runs == 0) {+        return 0;+    }+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    rle16_t rle = src_2->runs[rlepos];+    int32_t newcard = 0;+    while (arraypos < src_1->cardinality) {+        const uint16_t arrayval = src_1->array[arraypos];+        while (rle.value + rle.length <+               arrayval) {  // this will frequently be false+            ++rlepos;+            if (rlepos == src_2->n_runs) {+                return newcard;  // we are done+            }+            rle = src_2->runs[rlepos];+        }+        if (rle.value > arrayval) {+            arraypos = advanceUntil(src_1->array, arraypos, src_1->cardinality,+                                    rle.value);+        } else {+            newcard++;+            arraypos++;+        }+    }+    return newcard;+}++/* Compute the intersection  between src_1 and src_2+ **/+int run_bitset_container_intersection_cardinality(+    const run_container_t *src_1, const bitset_container_t *src_2) {+    if (run_container_is_full(src_1)) {+        return bitset_container_cardinality(src_2);+    }+    int answer = 0;+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        answer +=+            bitset_lenrange_cardinality(src_2->words, rle.value, rle.length);+    }+    return answer;+}++bool array_run_container_intersect(const array_container_t *src_1,+                                   const run_container_t *src_2) {+    if (run_container_is_full(src_2)) {+        return !array_container_empty(src_1);+    }+    if (src_2->n_runs == 0) {+        return false;+    }+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    rle16_t rle = src_2->runs[rlepos];+    while (arraypos < src_1->cardinality) {+        const uint16_t arrayval = src_1->array[arraypos];+        while (rle.value + rle.length <+               arrayval) {  // this will frequently be false+            ++rlepos;+            if (rlepos == src_2->n_runs) {+                return false;  // we are done+            }+            rle = src_2->runs[rlepos];+        }+        if (rle.value > arrayval) {+            arraypos = advanceUntil(src_1->array, arraypos, src_1->cardinality,+                                    rle.value);+        } else {+            return true;+        }+    }+    return false;+}++/* Compute the intersection  between src_1 and src_2+ **/+bool run_bitset_container_intersect(const run_container_t *src_1,+                                    const bitset_container_t *src_2) {+    if (run_container_is_full(src_1)) {+        return !bitset_container_empty(src_2);+    }+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        if (!bitset_lenrange_empty(src_2->words, rle.value, rle.length))+            return true;+    }+    return false;+}++/*+ * Write the "card" common values of two bitset containers to "out", which+ * must hold exactly "card" values.+ */+static inline void extract_intersection_to_array(+    const bitset_container_t *src_1, const bitset_container_t *src_2,+    uint16_t *out, int32_t card) {+    // An empty result has no array to write to: "out" may be NULL.+    if (card == 0) return;+#if CROARING_IS_X64 && CROARING_COMPILER_SUPPORTS_AVX512+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX512) {+        bitset_extract_intersection_setbits_avx512_uint16(+            src_1->words, src_2->words, BITSET_CONTAINER_SIZE_IN_WORDS, out,+            (size_t)card, 0);+        return;+    }+#endif+    bitset_extract_intersection_setbits_uint16_with_card(+        src_1->words, src_2->words, BITSET_CONTAINER_SIZE_IN_WORDS, out,+        (size_t)card, 0);+}++/*+ * Compute the intersection between src_1 and src_2 and write the result+ * to *dst. If the return function is true, the result is a bitset_container_t+ * otherwise is a array_container_t.+ */+bool bitset_bitset_container_intersection(const bitset_container_t *src_1,+                                          const bitset_container_t *src_2,+                                          container_t **dst) {+    const int newCardinality = bitset_container_and_justcard(src_1, src_2);+    if (newCardinality > DEFAULT_MAX_SIZE) {+        *dst = bitset_container_create();+        if (*dst != NULL) {+            bitset_container_and_nocard(src_1, src_2, CAST_bitset(*dst));+            CAST_bitset(*dst)->cardinality = newCardinality;+        }+        return true;  // it is a bitset+    }+    *dst = array_container_create_given_capacity(newCardinality);+    if (*dst != NULL) {+        CAST_array(*dst)->cardinality = newCardinality;+        extract_intersection_to_array(src_1, src_2, CAST_array(*dst)->array,+                                      newCardinality);+    }+    return false;  // not a bitset+}++bool bitset_bitset_container_intersection_inplace(+    bitset_container_t *src_1, const bitset_container_t *src_2,+    container_t **dst) {+    const int newCardinality = bitset_container_and_justcard(src_1, src_2);+    if (newCardinality > DEFAULT_MAX_SIZE) {+        *dst = src_1;+        bitset_container_and_nocard(src_1, src_2, src_1);+        CAST_bitset(*dst)->cardinality = newCardinality;+        return true;  // it is a bitset+    }+    *dst = array_container_create_given_capacity(newCardinality);+    if (*dst != NULL) {+        CAST_array(*dst)->cardinality = newCardinality;+        extract_intersection_to_array(src_1, src_2, CAST_array(*dst)->array,+                                      newCardinality);+    }+    return false;  // not a bitset+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_negation.c view
@@ -0,0 +1,422 @@+/*+ * mixed_negation.c+ *+ */++#include <assert.h>+#include <string.h>++#include <roaring/array_util.h>+#include <roaring/bitset_util.h>+#include <roaring/containers/containers.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_negation.h>+#include <roaring/containers/run.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++// TODO: make simplified and optimized negation code across+// the full range.++/* Negation across the entire range of the container.+ * Compute the  negation of src  and write the result+ * to *dst. The complement of a+ * sufficiently sparse set will always be dense and a hence a bitmap+' * We assume that dst is pre-allocated and a valid bitset container+ * There can be no in-place version.+ */+void array_container_negation(const array_container_t *src,+                              bitset_container_t *dst) {+    uint64_t card = UINT64_C(1 << 16);+    bitset_container_set_all(dst);++    if (src->cardinality == 0) {+        return;+    }++    dst->cardinality = (int32_t)bitset_clear_list(dst->words, card, src->array,+                                                  (uint64_t)src->cardinality);+}++/* Negation across the entire range of the container+ * Compute the  negation of src  and write the result+ * to *dst.  A true return value indicates a bitset result,+ * otherwise the result is an array container.+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool bitset_container_negation(const bitset_container_t *src,+                               container_t **dst) {+    return bitset_container_negation_range(src, 0, (1 << 16), dst);+}++/* inplace version */+/*+ * Same as bitset_container_negation except that if the output is to+ * be a+ * bitset_container_t, then src is modified and no allocation is made.+ * If the output is to be an array_container_t, then caller is responsible+ * to free the container.+ * In all cases, the result is in *dst.+ */+bool bitset_container_negation_inplace(bitset_container_t *src,+                                       container_t **dst) {+    return bitset_container_negation_range_inplace(src, 0, (1 << 16), dst);+}++/* Negation across the entire range of container+ * Compute the  negation of src  and write the result+ * to *dst.  Return values are the *_TYPECODES as defined * in containers.h+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+int run_container_negation(const run_container_t *src, container_t **dst) {+    return run_container_negation_range(src, 0, (1 << 16), dst);+}++/*+ * Same as run_container_negation except that if the output is to+ * be a+ * run_container_t, and has the capacity to hold the result,+ * then src is modified and no allocation is made.+ * In all cases, the result is in *dst.+ */+int run_container_negation_inplace(run_container_t *src, container_t **dst) {+    return run_container_negation_range_inplace(src, 0, (1 << 16), dst);+}++/* Negation across a range of the container.+ * Compute the  negation of src  and write the result+ * to *dst. Returns true if the result is a bitset container+ * and false for an array container.  *dst is not preallocated.+ */+CROARING_ALLOW_UNALIGNED+bool array_container_negation_range(const array_container_t *src,+                                    const int range_start, const int range_end,+                                    container_t **dst) {+    /* close port of the Java implementation */+    if (range_start >= range_end) {+        *dst = array_container_clone(src);+        return false;+    }++    int32_t start_index =+        binarySearch(src->array, src->cardinality, (uint16_t)range_start);+    if (start_index < 0) start_index = -start_index - 1;++    int32_t last_index =+        binarySearch(src->array, src->cardinality, (uint16_t)(range_end - 1));+    if (last_index < 0) last_index = -last_index - 2;++    const int32_t current_values_in_range = last_index - start_index + 1;+    const int32_t span_to_be_flipped = range_end - range_start;+    const int32_t new_values_in_range =+        span_to_be_flipped - current_values_in_range;+    const int32_t cardinality_change =+        new_values_in_range - current_values_in_range;+    const int32_t new_cardinality = src->cardinality + cardinality_change;++    if (new_cardinality > DEFAULT_MAX_SIZE) {+        bitset_container_t *temp = bitset_container_from_array(src);+        bitset_flip_range(temp->words, (uint32_t)range_start,+                          (uint32_t)range_end);+        temp->cardinality = new_cardinality;+        *dst = temp;+        return true;+    }++    array_container_t *arr =+        array_container_create_given_capacity(new_cardinality);+    *dst = (container_t *)arr;+    if (new_cardinality == 0) {+        arr->cardinality = new_cardinality;+        return false;  // we are done.+    }+    // copy stuff before the active area+    memcpy(arr->array, src->array, start_index * sizeof(uint16_t));++    // work on the range+    int32_t out_pos = start_index, in_pos = start_index;+    int32_t val_in_range = range_start;+    for (; val_in_range < range_end && in_pos <= last_index; ++val_in_range) {+        if ((uint16_t)val_in_range != src->array[in_pos]) {+            arr->array[out_pos++] = (uint16_t)val_in_range;+        } else {+            ++in_pos;+        }+    }+    for (; val_in_range < range_end; ++val_in_range)+        arr->array[out_pos++] = (uint16_t)val_in_range;++    // content after the active range+    memcpy(arr->array + out_pos, src->array + (last_index + 1),+           (src->cardinality - (last_index + 1)) * sizeof(uint16_t));+    arr->cardinality = new_cardinality;+    return false;+}++/* Even when the result would fit, it is unclear how to make an+ * inplace version without inefficient copying.+ */++bool array_container_negation_range_inplace(array_container_t *src,+                                            const int range_start,+                                            const int range_end,+                                            container_t **dst) {+    bool ans = array_container_negation_range(src, range_start, range_end, dst);+    // TODO : try a real inplace version+    array_container_free(src);+    return ans;+}++// Appends the run [start, end] (inclusive) after the n runs, merging it with+// the last one when they touch. Returns the new number of runs.+static inline int32_t append_run(rle16_t *runs, int32_t n, uint32_t start,+                                 uint32_t end) {+    if (n > 0 &&+        (uint32_t)runs[n - 1].value + runs[n - 1].length + 1 == start) {+        runs[n - 1].length = (uint16_t)(end - runs[n - 1].value);+        return n;+    }+    runs[n].value = (uint16_t)start;+    runs[n].length = (uint16_t)(end - start);+    return n + 1;+}++CROARING_ALLOW_UNALIGNED+int array_container_negation_range_to_any(const array_container_t *src,+                                          const int range_start,+                                          const int range_end,+                                          container_t **dst) {+    // The result has more than DEFAULT_MAX_SIZE values, so it would be a+    // bitset, but it has at most cardinality + 1 runs: when these runs take+    // less room than a bitset, we build them directly. In the range, the+    // runs are the gaps between the values; outside, the values themselves.+    if (range_end - range_start - src->cardinality > DEFAULT_MAX_SIZE &&+        run_container_serialized_size_in_bytes(src->cardinality + 1) <+            BITSET_CONTAINER_SIZE_IN_WORDS * (int32_t)sizeof(uint64_t)) {+        run_container_t *ans =+            run_container_create_given_capacity(src->cardinality + 1);+        if (ans == NULL) {+            *dst = NULL;+            return RUN_CONTAINER_TYPE;+        }+        const uint16_t *array = src->array;+        const int32_t card = src->cardinality;+        rle16_t *runs = ans->runs;+        int32_t n = 0, k = 0;+        for (; k < card && array[k] < range_start; k++) {+            n = append_run(runs, n, array[k], array[k]);+        }+        uint32_t next = (uint32_t)range_start;  // first value not yet decided+        for (; k < card && array[k] < range_end; k++) {+            if (array[k] > next) n = append_run(runs, n, next, array[k] - 1u);+            next = array[k] + 1u;+        }+        if (next < (uint32_t)range_end) {+            n = append_run(runs, n, next, (uint32_t)range_end - 1);+        }+        for (; k < card; k++) {+            n = append_run(runs, n, array[k], array[k]);+        }+        ans->n_runs = n;+        *dst = ans;+        return RUN_CONTAINER_TYPE;+    }+    if (range_start == 0 && range_end == (1 << 16)) {+        // The negation of a whole array is always a bitset, built in one pass.+        bitset_container_t *ans = bitset_container_create();+        if (ans != NULL) array_container_negation(src, ans);+        *dst = ans;+        return BITSET_CONTAINER_TYPE;+    }+    return array_container_negation_range(src, range_start, range_end, dst)+               ? BITSET_CONTAINER_TYPE+               : ARRAY_CONTAINER_TYPE;+}++/* Negation across a range of the container+ * Compute the  negation of src  and write the result+ * to *dst.  A true return value indicates a bitset result,+ * otherwise the result is an array container.+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+bool bitset_container_negation_range(const bitset_container_t *src,+                                     const int range_start, const int range_end,+                                     container_t **dst) {+    // TODO maybe consider density-based estimate+    // and sometimes build result directly as array, with+    // conversion back to bitset if wrong.  Or determine+    // actual result cardinality, then go directly for the known final cont.++    // keep computation using bitsets as long as possible.+    bitset_container_t *t = bitset_container_clone(src);+    bitset_flip_range(t->words, (uint32_t)range_start, (uint32_t)range_end);+    t->cardinality = bitset_container_compute_cardinality(t);++    if (t->cardinality > DEFAULT_MAX_SIZE) {+        *dst = t;+        return true;+    } else {+        *dst = array_container_from_bitset(t);+        bitset_container_free(t);+        return false;+    }+}++/* inplace version */+/*+ * Same as bitset_container_negation except that if the output is to+ * be a+ * bitset_container_t, then src is modified and no allocation is made.+ * If the output is to be an array_container_t, then caller is responsible+ * to free the container.+ * In all cases, the result is in *dst.+ */+bool bitset_container_negation_range_inplace(bitset_container_t *src,+                                             const int range_start,+                                             const int range_end,+                                             container_t **dst) {+    bitset_flip_range(src->words, (uint32_t)range_start, (uint32_t)range_end);+    src->cardinality = bitset_container_compute_cardinality(src);+    if (src->cardinality > DEFAULT_MAX_SIZE) {+        *dst = src;+        return true;+    }+    *dst = array_container_from_bitset(src);+    bitset_container_free(src);+    return false;+}++/* Negation across a range of container+ * Compute the  negation of src  and write the result+ * to *dst. Return values are the *_TYPECODES as defined * in containers.h+ *  We assume that dst is not pre-allocated. In+ * case of failure, *dst will be NULL.+ */+CROARING_ALLOW_UNALIGNED+int run_container_negation_range(const run_container_t *src,+                                 const int range_start, const int range_end,+                                 container_t **dst) {+    uint8_t return_typecode;++    // follows the Java implementation+    if (range_end <= range_start) {+        *dst = run_container_clone(src);+        return RUN_CONTAINER_TYPE;+    }++    run_container_t *ans = run_container_create_given_capacity(+        src->n_runs + 1);  // src->n_runs + 1);+    int k = 0;+    for (; k < src->n_runs && src->runs[k].value < range_start; ++k) {+        ans->runs[k] = src->runs[k];+        ans->n_runs++;+    }++    run_container_smart_append_exclusive(+        ans, (uint16_t)range_start, (uint16_t)(range_end - range_start - 1));++    for (; k < src->n_runs; ++k) {+        run_container_smart_append_exclusive(ans, src->runs[k].value,+                                             src->runs[k].length);+    }++    *dst = convert_run_to_efficient_container(ans, &return_typecode);+    if (return_typecode != RUN_CONTAINER_TYPE) run_container_free(ans);++    return return_typecode;+}++/*+ * Same as run_container_negation except that if the output is to+ * be a+ * run_container_t, and has the capacity to hold the result,+ * then src is modified and no allocation is made.+ * In all cases, the result is in *dst.+ */+int run_container_negation_range_inplace(run_container_t *src,+                                         const int range_start,+                                         const int range_end,+                                         container_t **dst) {+    uint8_t return_typecode;++    if (range_end <= range_start) {+        *dst = src;+        return RUN_CONTAINER_TYPE;+    }++    // TODO: efficient special case when range is 0 to 65535 inclusive++    if (src->capacity == src->n_runs) {+        // no excess room.  More checking to see if result can fit+        bool last_val_before_range = false;+        bool first_val_in_range = false;+        bool last_val_in_range = false;+        bool first_val_past_range = false;++        if (range_start > 0)+            last_val_before_range =+                run_container_contains(src, (uint16_t)(range_start - 1));+        first_val_in_range = run_container_contains(src, (uint16_t)range_start);++        if (last_val_before_range == first_val_in_range) {+            last_val_in_range =+                run_container_contains(src, (uint16_t)(range_end - 1));+            if (range_end != 0x10000)+                first_val_past_range =+                    run_container_contains(src, (uint16_t)range_end);++            if (last_val_in_range ==+                first_val_past_range) {  // no space for inplace+                int ans = run_container_negation_range(src, range_start,+                                                       range_end, dst);+                run_container_free(src);+                return ans;+            }+        }+    }+    // all other cases: result will fit++    run_container_t *ans = src;+    int my_nbr_runs = src->n_runs;++    ans->n_runs = 0;+    int k = 0;+    for (; (k < my_nbr_runs) && (src->runs[k].value < range_start); ++k) {+        // ans->runs[k] = src->runs[k]; (would be self-copy)+        ans->n_runs++;+    }++    // as with Java implementation, use locals to give self a buffer of depth 1+    rle16_t buffered = CROARING_MAKE_RLE16(0, 0);+    rle16_t next = buffered;+    if (k < my_nbr_runs) buffered = src->runs[k];++    run_container_smart_append_exclusive(+        ans, (uint16_t)range_start, (uint16_t)(range_end - range_start - 1));++    for (; k < my_nbr_runs; ++k) {+        if (k + 1 < my_nbr_runs) next = src->runs[k + 1];++        run_container_smart_append_exclusive(ans, buffered.value,+                                             buffered.length);+        buffered = next;+    }++    *dst = convert_run_to_efficient_container(ans, &return_typecode);+    if (return_typecode != RUN_CONTAINER_TYPE) run_container_free(ans);++    return return_typecode;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_subset.c view
@@ -0,0 +1,145 @@+#include <roaring/array_util.h>+#include <roaring/containers/mixed_subset.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++bool array_container_is_subset_bitset(const array_container_t* container1,+                                      const bitset_container_t* container2) {+    if (container2->cardinality != BITSET_UNKNOWN_CARDINALITY) {+        if (container2->cardinality < container1->cardinality) {+            return false;+        }+    }+    for (int i = 0; i < container1->cardinality; ++i) {+        if (!bitset_container_contains(container2, container1->array[i])) {+            return false;+        }+    }+    return true;+}++bool run_container_is_subset_array(const run_container_t* container1,+                                   const array_container_t* container2) {+    if (run_container_cardinality(container1) > container2->cardinality)+        return false;+    int32_t start_pos = -1, stop_pos = -1;+    for (int i = 0; i < container1->n_runs; ++i) {+        int32_t start = container1->runs[i].value;+        int32_t stop = start + container1->runs[i].length;+        start_pos = advanceUntil(container2->array, stop_pos,+                                 container2->cardinality, start);+        stop_pos = advanceUntil(container2->array, stop_pos,+                                container2->cardinality, stop);+        if (stop_pos == container2->cardinality) {+            return false;+        } else if (stop_pos - start_pos != stop - start ||+                   container2->array[start_pos] != start ||+                   container2->array[stop_pos] != stop) {+            return false;+        }+    }+    return true;+}++bool array_container_is_subset_run(const array_container_t* container1,+                                   const run_container_t* container2) {+    if (container1->cardinality > run_container_cardinality(container2))+        return false;+    int i_array = 0, i_run = 0;+    while (i_array < container1->cardinality && i_run < container2->n_runs) {+        uint32_t start = container2->runs[i_run].value;+        uint32_t stop = start + container2->runs[i_run].length;+        if (container1->array[i_array] < start) {+            return false;+        } else if (container1->array[i_array] > stop) {+            i_run++;+        } else {  // the value of the array is in the run+            i_array++;+        }+    }+    if (i_array == container1->cardinality) {+        return true;+    } else {+        return false;+    }+}++bool run_container_is_subset_bitset(const run_container_t* container1,+                                    const bitset_container_t* container2) {+    // todo: this code could be much faster+    if (container2->cardinality != BITSET_UNKNOWN_CARDINALITY) {+        if (container2->cardinality < run_container_cardinality(container1)) {+            return false;+        }+    } else {+        int32_t card = bitset_container_compute_cardinality(+            container2);  // modify container2?+        if (card < run_container_cardinality(container1)) {+            return false;+        }+    }+    for (int i = 0; i < container1->n_runs; ++i) {+        uint32_t run_start = container1->runs[i].value;+        uint32_t le = container1->runs[i].length;+        for (uint32_t j = run_start; j <= run_start + le; ++j) {+            if (!bitset_container_contains(container2, j)) {+                return false;+            }+        }+    }+    return true;+}++bool bitset_container_is_subset_run(const bitset_container_t* container1,+                                    const run_container_t* container2) {+    // todo: this code could be much faster+    if (container1->cardinality != BITSET_UNKNOWN_CARDINALITY) {+        if (container1->cardinality > run_container_cardinality(container2)) {+            return false;+        }+    }+    int32_t i_bitset = 0, i_run = 0;+    while (i_bitset < BITSET_CONTAINER_SIZE_IN_WORDS &&+           i_run < container2->n_runs) {+        uint64_t w = container1->words[i_bitset];+        while (w != 0 && i_run < container2->n_runs) {+            uint32_t start = container2->runs[i_run].value;+            uint32_t stop = start + container2->runs[i_run].length;+            uint64_t t = w & (~w + 1);+            uint16_t r = i_bitset * 64 + roaring_trailing_zeroes(w);+            if (r < start) {+                return false;+            } else if (r > stop) {+                i_run++;+                continue;+            } else {+                w ^= t;+            }+        }+        if (w == 0) {+            i_bitset++;+        } else {+            return false;+        }+    }+    if (i_bitset < BITSET_CONTAINER_SIZE_IN_WORDS) {+        // terminated iterating on the run containers, check that rest of bitset+        // is empty+        for (; i_bitset < BITSET_CONTAINER_SIZE_IN_WORDS; i_bitset++) {+            if (container1->words[i_bitset] != 0) {+                return false;+            }+        }+    }+    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_union.c view
@@ -0,0 +1,374 @@+/*+ * mixed_union.c+ *+ */++#include <assert.h>+#include <string.h>++#include <roaring/bitset_util.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_union.h>+#include <roaring/containers/perfparameters.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the union of src_1 and src_2 and write the result to+ * dst.  */+void array_bitset_container_union(const array_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  bitset_container_t *dst) {+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    dst->cardinality = (int32_t)bitset_set_list_withcard(+        dst->words, dst->cardinality, src_1->array, src_1->cardinality);+}++/* Compute the union of src_1 and src_2 and write the result to+ * dst. It is allowed for src_2 to be dst.  This version does not+ * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY). */+void array_bitset_container_lazy_union(const array_container_t *src_1,+                                       const bitset_container_t *src_2,+                                       bitset_container_t *dst) {+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    bitset_set_list(dst->words, src_1->array, src_1->cardinality);+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;+}++void run_bitset_container_union(const run_container_t *src_1,+                                const bitset_container_t *src_2,+                                bitset_container_t *dst) {+    assert(!run_container_is_full(src_1));  // catch this case upstream+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        bitset_set_lenrange(dst->words, rle.value, rle.length);+    }+    dst->cardinality = bitset_container_compute_cardinality(dst);+}++void run_bitset_container_lazy_union(const run_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     bitset_container_t *dst) {+    assert(!run_container_is_full(src_1));  // catch this case upstream+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        bitset_set_lenrange(dst->words, rle.value, rle.length);+    }+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;+}++// why do we leave the result as a run container??+void array_run_container_union(const array_container_t *src_1,+                               const run_container_t *src_2,+                               run_container_t *dst) {+    if (run_container_is_full(src_2)) {+        run_container_copy(src_2, dst);+        return;+    }+    // TODO: see whether the "2*" is spurious+    run_container_grow(dst, 2 * (src_1->cardinality + src_2->n_runs), false);+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    rle16_t previousrle;+    if (src_2->runs[rlepos].value <= src_1->array[arraypos]) {+        previousrle = run_container_append_first(dst, src_2->runs[rlepos]);+        rlepos++;+    } else {+        previousrle =+            run_container_append_value_first(dst, src_1->array[arraypos]);+        arraypos++;+    }+    while ((rlepos < src_2->n_runs) && (arraypos < src_1->cardinality)) {+        if (src_2->runs[rlepos].value <= src_1->array[arraypos]) {+            run_container_append(dst, src_2->runs[rlepos], &previousrle);+            rlepos++;+        } else {+            run_container_append_value(dst, src_1->array[arraypos],+                                       &previousrle);+            arraypos++;+        }+    }+    if (arraypos < src_1->cardinality) {+        while (arraypos < src_1->cardinality) {+            run_container_append_value(dst, src_1->array[arraypos],+                                       &previousrle);+            arraypos++;+        }+    } else {+        while (rlepos < src_2->n_runs) {+            run_container_append(dst, src_2->runs[rlepos], &previousrle);+            rlepos++;+        }+    }+}++void array_run_container_inplace_union(const array_container_t *src_1,+                                       run_container_t *src_2) {+    if (run_container_is_full(src_2)) {+        return;+    }+    const int32_t maxoutput = src_1->cardinality + src_2->n_runs;+    const int32_t neededcapacity = maxoutput + src_2->n_runs;+    if (src_2->capacity < neededcapacity)+        run_container_grow(src_2, neededcapacity, true);+    memmove(src_2->runs + maxoutput, src_2->runs,+            src_2->n_runs * sizeof(rle16_t));+    rle16_t *inputsrc2 = src_2->runs + maxoutput;+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    int src2nruns = src_2->n_runs;+    src_2->n_runs = 0;++    rle16_t previousrle;++    if (inputsrc2[rlepos].value <= src_1->array[arraypos]) {+        previousrle = run_container_append_first(src_2, inputsrc2[rlepos]);+        rlepos++;+    } else {+        previousrle =+            run_container_append_value_first(src_2, src_1->array[arraypos]);+        arraypos++;+    }++    while ((rlepos < src2nruns) && (arraypos < src_1->cardinality)) {+        if (inputsrc2[rlepos].value <= src_1->array[arraypos]) {+            run_container_append(src_2, inputsrc2[rlepos], &previousrle);+            rlepos++;+        } else {+            run_container_append_value(src_2, src_1->array[arraypos],+                                       &previousrle);+            arraypos++;+        }+    }+    if (arraypos < src_1->cardinality) {+        while (arraypos < src_1->cardinality) {+            run_container_append_value(src_2, src_1->array[arraypos],+                                       &previousrle);+            arraypos++;+        }+    } else {+        while (rlepos < src2nruns) {+            run_container_append(src_2, inputsrc2[rlepos], &previousrle);+            rlepos++;+        }+    }+}++bool array_array_container_union(const array_container_t *src_1,+                                 const array_container_t *src_2,+                                 container_t **dst) {+    int totalCardinality = src_1->cardinality + src_2->cardinality;+    if (totalCardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_create_given_capacity(totalCardinality);+        if (*dst != NULL) {+            array_container_union(src_1, src_2, CAST_array(*dst));+        } else {+            return true;  // otherwise failure won't be caught+        }+        return false;  // not a bitset+    }+    *dst = bitset_container_create();+    bool returnval = true;  // expect a bitset+    if (*dst != NULL) {+        bitset_container_t *ourbitset = CAST_bitset(*dst);+        bitset_set_list(ourbitset->words, src_1->array, src_1->cardinality);+        ourbitset->cardinality = (int32_t)bitset_set_list_withcard(+            ourbitset->words, src_1->cardinality, src_2->array,+            src_2->cardinality);+        if (ourbitset->cardinality <= DEFAULT_MAX_SIZE) {+            // need to convert!+            *dst = array_container_from_bitset(ourbitset);+            bitset_container_free(ourbitset);+            returnval = false;  // not going to be a bitset+        }+    }+    return returnval;+}++bool array_array_container_inplace_union(array_container_t *src_1,+                                         const array_container_t *src_2,+                                         container_t **dst) {+    int totalCardinality = src_1->cardinality + src_2->cardinality;+    *dst = NULL;+    if (totalCardinality <= DEFAULT_MAX_SIZE) {+        if (src_1->capacity < totalCardinality) {+            *dst = array_container_create_given_capacity(+                2 * totalCardinality);  // be purposefully generous+            if (*dst != NULL) {+                array_container_union(src_1, src_2, CAST_array(*dst));+            } else {+                return true;  // otherwise failure won't be caught+            }+            return false;  // not a bitset+        } else {+            // fast_union_uint16 cannot work in place: its vectorized stores+            // could overwrite values not yet read.+            // https://github.com/RoaringBitmap/CRoaring/pull/452+            // See report https://github.com/RoaringBitmap/CRoaring/issues/476+            // So we copy src_1 aside first, which costs about as much as+            // moving it to the end of its array for the scalar union.+            uint16_t buffer[DEFAULT_MAX_SIZE];+            memcpy(buffer, src_1->array, src_1->cardinality * sizeof(uint16_t));+            src_1->cardinality = (int32_t)fast_union_uint16(+                buffer, src_1->cardinality, src_2->array, src_2->cardinality,+                src_1->array);+            return false;  // not a bitset+        }+    }+    *dst = bitset_container_create();+    bool returnval = true;  // expect a bitset+    if (*dst != NULL) {+        bitset_container_t *ourbitset = CAST_bitset(*dst);+        bitset_set_list(ourbitset->words, src_1->array, src_1->cardinality);+        ourbitset->cardinality = (int32_t)bitset_set_list_withcard(+            ourbitset->words, src_1->cardinality, src_2->array,+            src_2->cardinality);+        if (ourbitset->cardinality <= DEFAULT_MAX_SIZE) {+            // need to convert!+            if (src_1->capacity < ourbitset->cardinality) {+                array_container_grow(src_1, ourbitset->cardinality, false);+            }++            bitset_extract_setbits_uint16(ourbitset->words,+                                          BITSET_CONTAINER_SIZE_IN_WORDS,+                                          src_1->array, 0);+            src_1->cardinality = ourbitset->cardinality;+            *dst = src_1;+            bitset_container_free(ourbitset);+            returnval = false;  // not going to be a bitset+        }+    }+    return returnval;+}++bool array_array_container_lazy_union(const array_container_t *src_1,+                                      const array_container_t *src_2,+                                      container_t **dst) {+    int totalCardinality = src_1->cardinality + src_2->cardinality;+    //+    // We assume that operations involving bitset containers will be faster than+    // operations involving solely array containers, except maybe when array+    // containers are small. Indeed, for example, it is cheap to compute the+    // union between an array and a bitset container, generally more so than+    // between a large array and another array. So it is advantageous to favour+    // bitset containers during the computation. Of course, if we convert array+    // containers eagerly to bitset containers, we may later need to revert the+    // bitset containers to array containerr to satisfy the Roaring format+    // requirements, but such one-time conversions at the end may not be overly+    // expensive. We arrived to this design based on extensive benchmarking.+    //+    if (totalCardinality <= ARRAY_LAZY_LOWERBOUND) {+        *dst = array_container_create_given_capacity(totalCardinality);+        if (*dst != NULL) {+            array_container_union(src_1, src_2, CAST_array(*dst));+        } else {+            return true;  // otherwise failure won't be caught+        }+        return false;  // not a bitset+    }+    *dst = bitset_container_create();+    bool returnval = true;  // expect a bitset+    if (*dst != NULL) {+        bitset_container_t *ourbitset = CAST_bitset(*dst);+        bitset_set_list(ourbitset->words, src_1->array, src_1->cardinality);+        bitset_set_list(ourbitset->words, src_2->array, src_2->cardinality);+        ourbitset->cardinality = BITSET_UNKNOWN_CARDINALITY;+    }+    return returnval;+}++bool array_array_container_lazy_inplace_union(array_container_t *src_1,+                                              const array_container_t *src_2,+                                              container_t **dst) {+    int totalCardinality = src_1->cardinality + src_2->cardinality;+    *dst = NULL;+    //+    // We assume that operations involving bitset containers will be faster than+    // operations involving solely array containers, except maybe when array+    // containers are small. Indeed, for example, it is cheap to compute the+    // union between an array and a bitset container, generally more so than+    // between a large array and another array. So it is advantageous to favour+    // bitset containers during the computation. Of course, if we convert array+    // containers eagerly to bitset containers, we may later need to revert the+    // bitset containers to array containerr to satisfy the Roaring format+    // requirements, but such one-time conversions at the end may not be overly+    // expensive. We arrived to this design based on extensive benchmarking.+    //+    if (totalCardinality <= ARRAY_LAZY_LOWERBOUND) {+        if (src_1->capacity < totalCardinality) {+            *dst = array_container_create_given_capacity(+                2 * totalCardinality);  // be purposefully generous+            if (*dst != NULL) {+                array_container_union(src_1, src_2, CAST_array(*dst));+            } else {+                return true;  // otherwise failure won't be caught+            }+            return false;  // not a bitset+        } else {+            memmove(src_1->array + src_2->cardinality, src_1->array,+                    src_1->cardinality * sizeof(uint16_t));+            /*+              Next line is safe:++              We just need to focus on the reading and writing performed on+              array1. In `union_vector16`, both vectorized and scalar code still+              obey the basic rule: read from two inputs, do the union, and then+              write the output.++              Let's say the length(cardinality) of input2 is L2:+              ```+                  |<-  L2  ->|+              array1: [output--- |input 1---|---]+              array2: [input 2---]+              ```+              Let's define 3 __m128i pointers, `pos1` starts from `input1`,+              `pos2` starts from `input2`, these 2 point at the next byte to+              read, `out` starts from `output`, pointing at the next byte to+              overwrite.+              ```+              array1: [output--- |input 1---|---]+                          ^          ^+                      out        pos1+              array2: [input 2---]+                          ^+                          pos2+              ```+              The union output always contains less or equal number of elements+              than all inputs added, so we have:+              ```+              out <= pos1 + pos2+              ```+              therefore:+              ```+              out <= pos1 + L2+              ```+              which means you will not overwrite data beyond pos1, so the data+              haven't read is safe, and we don't care the data already read.+            */+            src_1->cardinality = (int32_t)fast_union_uint16(+                src_1->array + src_2->cardinality, src_1->cardinality,+                src_2->array, src_2->cardinality, src_1->array);+            return false;  // not a bitset+        }+    }+    *dst = bitset_container_create();+    bool returnval = true;  // expect a bitset+    if (*dst != NULL) {+        bitset_container_t *ourbitset = CAST_bitset(*dst);+        bitset_set_list(ourbitset->words, src_1->array, src_1->cardinality);+        bitset_set_list(ourbitset->words, src_2->array, src_2->cardinality);+        ourbitset->cardinality = BITSET_UNKNOWN_CARDINALITY;+    }+    return returnval;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/mixed_xor.c view
@@ -0,0 +1,385 @@+/*+ * mixed_xor.c+ */++#include <assert.h>+#include <string.h>++#include <roaring/bitset_util.h>+#include <roaring/containers/containers.h>+#include <roaring/containers/convert.h>+#include <roaring/containers/mixed_xor.h>+#include <roaring/containers/perfparameters.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially).+ * Result is true iff dst is a bitset  */+bool array_bitset_container_xor(const array_container_t *src_1,+                                const bitset_container_t *src_2,+                                container_t **dst) {+    bitset_container_t *result = bitset_container_create();+    bitset_container_copy(src_2, result);+    result->cardinality = (int32_t)bitset_flip_list_withcard(+        result->words, result->cardinality, src_1->array, src_1->cardinality);++    // do required type conversions.+    if (result->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(result);+        bitset_container_free(result);+        return false;  // not bitset+    }+    *dst = result;+    return true;  // bitset+}++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. It is allowed for src_2 to be dst.  This version does not+ * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY).+ */++void array_bitset_container_lazy_xor(const array_container_t *src_1,+                                     const bitset_container_t *src_2,+                                     bitset_container_t *dst) {+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    bitset_flip_list(dst->words, src_1->array, src_1->cardinality);+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;+}++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_xor(const run_container_t *src_1,+                              const bitset_container_t *src_2,+                              container_t **dst) {+    bitset_container_t *result = bitset_container_create();++    bitset_container_copy(src_2, result);+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        bitset_flip_range(result->words, rle.value,+                          rle.value + rle.length + UINT32_C(1));+    }+    result->cardinality = bitset_container_compute_cardinality(result);++    if (result->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(result);+        bitset_container_free(result);+        return false;  // not bitset+    }+    *dst = result;+    return true;  // bitset+}++/* lazy xor.  Dst is initialized and may be equal to src_2.+ *  Result is left as a bitset container, even if actual+ *  cardinality would dictate an array container.+ */++void run_bitset_container_lazy_xor(const run_container_t *src_1,+                                   const bitset_container_t *src_2,+                                   bitset_container_t *dst) {+    if (src_2 != dst) bitset_container_copy(src_2, dst);+    for (int32_t rlepos = 0; rlepos < src_1->n_runs; ++rlepos) {+        rle16_t rle = src_1->runs[rlepos];+        bitset_flip_range(dst->words, rle.value,+                          rle.value + rle.length + UINT32_C(1));+    }+    dst->cardinality = BITSET_UNKNOWN_CARDINALITY;+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int array_run_container_xor(const array_container_t *src_1,+                            const run_container_t *src_2, container_t **dst) {+    // semi following Java XOR implementation as of May 2016+    // the C OR implementation works quite differently and can return a run+    // container+    // TODO could optimize for full run containers.++    // use of lazy following Java impl.+    const int arbitrary_threshold = 32;+    if (src_1->cardinality < arbitrary_threshold) {+        run_container_t *ans = run_container_create();+        array_run_container_lazy_xor(src_1, src_2, ans);  // keeps runs.+        uint8_t typecode_after;+        *dst =+            convert_run_to_efficient_container_and_free(ans, &typecode_after);+        return typecode_after;+    }++    int card = run_container_cardinality(src_2);+    if (card <= DEFAULT_MAX_SIZE) {+        // Java implementation works with the array, xoring the run elements via+        // iterator+        array_container_t *temp = array_container_from_run(src_2);+        bool ret_is_bitset = array_array_container_xor(temp, src_1, dst);+        array_container_free(temp);+        return ret_is_bitset ? BITSET_CONTAINER_TYPE : ARRAY_CONTAINER_TYPE;++    } else {  // guess that it will end up as a bitset+        bitset_container_t *result = bitset_container_from_run(src_2);+        bool is_bitset = bitset_array_container_ixor(result, src_1, dst);+        // any necessary type conversion has been done by the ixor+        int retval = (is_bitset ? BITSET_CONTAINER_TYPE : ARRAY_CONTAINER_TYPE);+        return retval;+    }+}++/* Dst is a valid run container. (Can it be src_2? Let's say not.)+ * Leaves result as run container, even if other options are+ * smaller.+ */++void array_run_container_lazy_xor(const array_container_t *src_1,+                                  const run_container_t *src_2,+                                  run_container_t *dst) {+    run_container_grow(dst, src_1->cardinality + src_2->n_runs, false);+    int32_t rlepos = 0;+    int32_t arraypos = 0;+    dst->n_runs = 0;++    while ((rlepos < src_2->n_runs) && (arraypos < src_1->cardinality)) {+        if (src_2->runs[rlepos].value <= src_1->array[arraypos]) {+            run_container_smart_append_exclusive(dst, src_2->runs[rlepos].value,+                                                 src_2->runs[rlepos].length);+            rlepos++;+        } else {+            run_container_smart_append_exclusive(dst, src_1->array[arraypos],+                                                 0);+            arraypos++;+        }+    }+    while (arraypos < src_1->cardinality) {+        run_container_smart_append_exclusive(dst, src_1->array[arraypos], 0);+        arraypos++;+    }+    while (rlepos < src_2->n_runs) {+        run_container_smart_append_exclusive(dst, src_2->runs[rlepos].value,+                                             src_2->runs[rlepos].length);+        rlepos++;+    }+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int run_run_container_xor(const run_container_t *src_1,+                          const run_container_t *src_2, container_t **dst) {+    run_container_t *ans = run_container_create();+    run_container_xor(src_1, src_2, ans);+    uint8_t typecode_after;+    *dst = convert_run_to_efficient_container_and_free(ans, &typecode_after);+    return typecode_after;+}++/*+ * Java implementation (as of May 2016) for array_run, run_run+ * and  bitset_run don't do anything different for inplace.+ * Could adopt the mixed_union.c approach instead (ie, using+ * smart_append_exclusive)+ *+ */++bool array_array_container_xor(const array_container_t *src_1,+                               const array_container_t *src_2,+                               container_t **dst) {+    int totalCardinality =+        src_1->cardinality + src_2->cardinality;  // upper bound+    if (totalCardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_create_given_capacity(totalCardinality);+        array_container_xor(src_1, src_2, CAST_array(*dst));+        return false;  // not a bitset+    }+    *dst = bitset_container_from_array(src_1);+    bool returnval = true;  // expect a bitset+    bitset_container_t *ourbitset = CAST_bitset(*dst);+    ourbitset->cardinality = (uint32_t)bitset_flip_list_withcard(+        ourbitset->words, src_1->cardinality, src_2->array, src_2->cardinality);+    if (ourbitset->cardinality <= DEFAULT_MAX_SIZE) {+        // need to convert!+        *dst = array_container_from_bitset(ourbitset);+        bitset_container_free(ourbitset);+        returnval = false;  // not going to be a bitset+    }++    return returnval;+}++bool array_array_container_lazy_xor(const array_container_t *src_1,+                                    const array_container_t *src_2,+                                    container_t **dst) {+    int totalCardinality = src_1->cardinality + src_2->cardinality;+    //+    // We assume that operations involving bitset containers will be faster than+    // operations involving solely array containers, except maybe when array+    // containers are small. Indeed, for example, it is cheap to compute the+    // exclusive union between an array and a bitset container, generally more+    // so than between a large array and another array. So it is advantageous to+    // favour bitset containers during the computation. Of course, if we convert+    // array containers eagerly to bitset containers, we may later need to+    // revert the bitset containers to array containerr to satisfy the Roaring+    // format requirements, but such one-time conversions at the end may not be+    // overly expensive. We arrived to this design based on extensive+    // benchmarking on unions. For XOR/exclusive union, we simply followed the+    // heuristic used by the unions (see  mixed_union.c). Further tuning is+    // possible.+    //+    if (totalCardinality <= ARRAY_LAZY_LOWERBOUND) {+        *dst = array_container_create_given_capacity(totalCardinality);+        if (*dst != NULL) array_container_xor(src_1, src_2, CAST_array(*dst));+        return false;  // not a bitset+    }+    *dst = bitset_container_from_array(src_1);+    bool returnval = true;  // expect a bitset (maybe, for XOR??)+    if (*dst != NULL) {+        bitset_container_t *ourbitset = CAST_bitset(*dst);+        bitset_flip_list(ourbitset->words, src_2->array, src_2->cardinality);+        ourbitset->cardinality = BITSET_UNKNOWN_CARDINALITY;+    }+    return returnval;+}++/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially). Return value is+ * "dst is a bitset"+ */++bool bitset_bitset_container_xor(const bitset_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst) {+    bitset_container_t *ans = bitset_container_create();+    int card = bitset_container_xor(src_1, src_2, ans);+    if (card <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(ans);+        bitset_container_free(ans);+        return false;  // not bitset+    } else {+        *dst = ans;+        return true;+    }+}++/* Compute the xor of src_1 and src_2 and write the result to+ * dst (which has no container initially).  It will modify src_1+ * to be dst if the result is a bitset.  Otherwise, it will+ * free src_1 and dst will be a new array container.  In both+ * cases, the caller is responsible for deallocating dst.+ * Returns true iff dst is a bitset  */++bool bitset_array_container_ixor(bitset_container_t *src_1,+                                 const array_container_t *src_2,+                                 container_t **dst) {+    *dst = src_1;+    src_1->cardinality = (uint32_t)bitset_flip_list_withcard(+        src_1->words, src_1->cardinality, src_2->array, src_2->cardinality);++    if (src_1->cardinality <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(src_1);+        bitset_container_free(src_1);+        return false;  // not bitset+    } else+        return true;+}++/* a bunch of in-place, some of which may not *really* be inplace.+ * TODO: write actual inplace routine if efficiency warrants it+ * Anything inplace with a bitset is a good candidate+ */++bool bitset_bitset_container_ixor(bitset_container_t *src_1,+                                  const bitset_container_t *src_2,+                                  container_t **dst) {+    int card = bitset_container_xor(src_1, src_2, src_1);+    if (card <= DEFAULT_MAX_SIZE) {+        *dst = array_container_from_bitset(src_1);+        bitset_container_free(src_1);+        return false;  // not bitset+    } else {+        *dst = src_1;+        return true;+    }+}++bool array_bitset_container_ixor(array_container_t *src_1,+                                 const bitset_container_t *src_2,+                                 container_t **dst) {+    bool ans = array_bitset_container_xor(src_1, src_2, dst);+    array_container_free(src_1);+    return ans;+}++/* Compute the xor of src_1 and src_2 and write the result to+ * dst. Result may be either a bitset or an array container+ * (returns "result is bitset"). dst does not initially have+ * any container, but becomes either a bitset container (return+ * result true) or an array container.+ */++bool run_bitset_container_ixor(run_container_t *src_1,+                               const bitset_container_t *src_2,+                               container_t **dst) {+    bool ans = run_bitset_container_xor(src_1, src_2, dst);+    run_container_free(src_1);+    return ans;+}++bool bitset_run_container_ixor(bitset_container_t *src_1,+                               const run_container_t *src_2,+                               container_t **dst) {+    bool ans = run_bitset_container_xor(src_2, src_1, dst);+    bitset_container_free(src_1);+    return ans;+}++/* dst does not indicate a valid container initially.  Eventually it+ * can become any kind of container.+ */++int array_run_container_ixor(array_container_t *src_1,+                             const run_container_t *src_2, container_t **dst) {+    int ans = array_run_container_xor(src_1, src_2, dst);+    array_container_free(src_1);+    return ans;+}++int run_array_container_ixor(run_container_t *src_1,+                             const array_container_t *src_2,+                             container_t **dst) {+    int ans = array_run_container_xor(src_2, src_1, dst);+    run_container_free(src_1);+    return ans;+}++bool array_array_container_ixor(array_container_t *src_1,+                                const array_container_t *src_2,+                                container_t **dst) {+    bool ans = array_array_container_xor(src_1, src_2, dst);+    array_container_free(src_1);+    return ans;+}++int run_run_container_ixor(run_container_t *src_1, const run_container_t *src_2,+                           container_t **dst) {+    int ans = run_run_container_xor(src_1, src_2, dst);+    run_container_free(src_1);+    return ans;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/containers/run.c view
@@ -0,0 +1,1143 @@+#include <stdio.h>+#include <stdlib.h>++#include <roaring/containers/run.h>+#include <roaring/memory.h>+#include <roaring/portability.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wuninitialized"+#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"+#endif+#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++extern inline uint16_t run_container_minimum(const run_container_t *run);+extern inline uint16_t run_container_maximum(const run_container_t *run);+extern inline int32_t interleavedBinarySearch(const rle16_t *array,+                                              int32_t lenarray, uint16_t ikey);+extern inline bool run_container_contains(const run_container_t *run,+                                          uint16_t pos);+extern inline int run_container_index_equalorlarger(const run_container_t *arr,+                                                    uint16_t x);+extern inline bool run_container_is_full(const run_container_t *run);+extern inline bool run_container_nonzero_cardinality(const run_container_t *rc);+extern inline int32_t run_container_serialized_size_in_bytes(int32_t num_runs);+extern inline run_container_t *run_container_create_range(uint32_t start,+                                                          uint32_t stop);+extern inline int run_container_cardinality(const run_container_t *run);++bool run_container_add(run_container_t *run, uint16_t pos) {+    int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos);+    if (index >= 0) return false;  // already there+    index = -index - 2;            // points to preceding value, possibly -1+    if (index >= 0) {              // possible match+        int32_t offset = pos - run->runs[index].value;+        int32_t le = run->runs[index].length;+        if (offset <= le) return false;  // already there+        if (offset == le + 1) {+            // we may need to fuse+            if (index + 1 < run->n_runs) {+                if (run->runs[index + 1].value == pos + 1) {+                    // indeed fusion is needed+                    run->runs[index].length = run->runs[index + 1].value ++                                              run->runs[index + 1].length -+                                              run->runs[index].value;+                    recoverRoomAtIndex(run, (uint16_t)(index + 1));+                    return true;+                }+            }+            run->runs[index].length++;+            return true;+        }+        if (index + 1 < run->n_runs) {+            // we may need to fuse+            if (run->runs[index + 1].value == pos + 1) {+                // indeed fusion is needed+                run->runs[index + 1].value = pos;+                run->runs[index + 1].length = run->runs[index + 1].length + 1;+                return true;+            }+        }+    }+    if (index == -1) {+        // we may need to extend the first run+        if (0 < run->n_runs) {+            if (run->runs[0].value == pos + 1) {+                run->runs[0].length++;+                run->runs[0].value--;+                return true;+            }+        }+    }+    makeRoomAtIndex(run, (uint16_t)(index + 1));+    run->runs[index + 1].value = pos;+    run->runs[index + 1].length = 0;+    return true;+}++/* Create a new run container. Return NULL in case of failure. */+run_container_t *run_container_create_given_capacity(int32_t size) {+    run_container_t *run;+    /* Allocate the run container itself. */+    if ((run = (run_container_t *)roaring_malloc(sizeof(run_container_t))) ==+        NULL) {+        return NULL;+    }+    if (size <= 0) {  // we don't want to rely on malloc(0)+        run->runs = NULL;+    } else if ((run->runs = (rle16_t *)roaring_malloc(sizeof(rle16_t) *+                                                      size)) == NULL) {+        roaring_free(run);+        return NULL;+    }+    run->capacity = size;+    run->n_runs = 0;+    return run;+}++int run_container_shrink_to_fit(run_container_t *src) {+    int32_t n_runs = src->n_runs;+    int32_t capacity = src->capacity;+    if (n_runs == capacity) return 0;  // nothing to do+    bool shrunk;+    src->runs = (rle16_t *)croaring_realloc_shrink(+        src->runs, n_runs * sizeof(rle16_t), &shrunk);+    if (!shrunk) return 0;+    src->capacity = n_runs;+    return (capacity - n_runs) * (int)sizeof(rle16_t);+}+/* Create a new run container. Return NULL in case of failure. */+run_container_t *run_container_create(void) {+    return run_container_create_given_capacity(RUN_DEFAULT_INIT_SIZE);+}++CROARING_ALLOW_UNALIGNED+run_container_t *run_container_clone(const run_container_t *src) {+    run_container_t *run = run_container_create_given_capacity(src->capacity);+    if (run == NULL) return NULL;+    run->capacity = src->capacity;+    run->n_runs = src->n_runs;+    memcpy(run->runs, src->runs, src->n_runs * sizeof(rle16_t));+    return run;+}++void run_container_offset(const run_container_t *c, container_t **loc,+                          container_t **hic, uint16_t offset) {+    run_container_t *lo = NULL, *hi = NULL;++    bool split;+    unsigned int lo_cap, hi_cap;+    int top, pivot;++    top = (1 << 16) - offset;+    pivot = run_container_index_equalorlarger(c, top);+    // pivot is the index of the first run that is >= top or -1 if no such run++    if (pivot >= 0) {+        split = c->runs[pivot].value < top;+        lo_cap = pivot + (split ? 1 : 0);+        hi_cap = c->n_runs - pivot;+    } else {+        // here pivot < 0+        split = false;+        lo_cap = c->n_runs;+        hi_cap = 0;+    }++    if (loc && lo_cap) {+        lo = run_container_create_given_capacity(lo_cap);+        memcpy(lo->runs, c->runs, lo_cap * sizeof(rle16_t));+        lo->n_runs = lo_cap;+        for (unsigned int i = 0; i < lo_cap; ++i) {+            lo->runs[i].value += offset;+        }+        *loc = (container_t *)lo;+    }++    if (hic && hi_cap) {+        hi = run_container_create_given_capacity(hi_cap);+        memcpy(hi->runs, c->runs + pivot, hi_cap * sizeof(rle16_t));+        hi->n_runs = hi_cap;+        for (unsigned int i = 0; i < hi_cap; ++i) {+            hi->runs[i].value += offset;+        }+        *hic = (container_t *)hi;+    }++    // Fix the split.+    if (split) {+        if (lo != NULL) {+            // Add the missing run to 'lo', exhausting length.+            lo->runs[lo->n_runs - 1].length =+                (1 << 16) - lo->runs[lo->n_runs - 1].value - 1;+        }++        if (hi != NULL) {+            // Fix the first run in 'hi'.+            hi->runs[0].length -= UINT16_MAX - hi->runs[0].value + 1;+            hi->runs[0].value = 0;+        }+    }+}++/* Free memory. */+void run_container_free(run_container_t *run) {+    if (run == NULL) return;+    roaring_free(run->runs);+    roaring_free(run);+}++void run_container_grow(run_container_t *run, int32_t min, bool copy) {+    int32_t newCapacity = (run->capacity == 0)   ? RUN_DEFAULT_INIT_SIZE+                          : run->capacity < 64   ? run->capacity * 2+                          : run->capacity < 1024 ? run->capacity * 3 / 2+                                                 : run->capacity * 5 / 4;+    if (newCapacity < min) newCapacity = min;+    run->capacity = newCapacity;+    assert(run->capacity >= min);+    if (copy) {+        rle16_t *oldruns = run->runs;+        run->runs = (rle16_t *)roaring_realloc(oldruns,+                                               run->capacity * sizeof(rle16_t));+        if (run->runs == NULL) roaring_free(oldruns);+    } else {+        roaring_free(run->runs);+        run->runs = (rle16_t *)roaring_malloc(run->capacity * sizeof(rle16_t));+    }+    // We may have run->runs == NULL.+}++/* copy one container into another */+void run_container_copy(const run_container_t *src, run_container_t *dst) {+    const int32_t n_runs = src->n_runs;+    if (src->n_runs > dst->capacity) {+        run_container_grow(dst, n_runs, false);+    }+    dst->n_runs = n_runs;+    memcpy(dst->runs, src->runs, sizeof(rle16_t) * n_runs);+}++/* Compute the union of `src_1' and `src_2' and write the result to `dst'+ * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */+CROARING_ALLOW_UNALIGNED+void run_container_union(const run_container_t *src_1,+                         const run_container_t *src_2, run_container_t *dst) {+    // TODO: this could be a lot more efficient++    // we start out with inexpensive checks+    const bool if1 = run_container_is_full(src_1);+    const bool if2 = run_container_is_full(src_2);+    if (if1 || if2) {+        if (if1) {+            run_container_copy(src_1, dst);+            return;+        }+        if (if2) {+            run_container_copy(src_2, dst);+            return;+        }+    }+    const int32_t neededcapacity = src_1->n_runs + src_2->n_runs;+    if (dst->capacity < neededcapacity)+        run_container_grow(dst, neededcapacity, false);+    dst->n_runs = 0;+    int32_t rlepos = 0;+    int32_t xrlepos = 0;++    rle16_t previousrle;+    if (src_1->runs[rlepos].value <= src_2->runs[xrlepos].value) {+        previousrle = run_container_append_first(dst, src_1->runs[rlepos]);+        rlepos++;+    } else {+        previousrle = run_container_append_first(dst, src_2->runs[xrlepos]);+        xrlepos++;+    }++    while ((xrlepos < src_2->n_runs) && (rlepos < src_1->n_runs)) {+        rle16_t newrl;+        if (src_1->runs[rlepos].value <= src_2->runs[xrlepos].value) {+            newrl = src_1->runs[rlepos];+            rlepos++;+        } else {+            newrl = src_2->runs[xrlepos];+            xrlepos++;+        }+        run_container_append(dst, newrl, &previousrle);+    }+    while (xrlepos < src_2->n_runs) {+        run_container_append(dst, src_2->runs[xrlepos], &previousrle);+        xrlepos++;+    }+    while (rlepos < src_1->n_runs) {+        run_container_append(dst, src_1->runs[rlepos], &previousrle);+        rlepos++;+    }+}++/* Compute the union of `src_1' and `src_2' and write the result to `src_1'+ */+CROARING_ALLOW_UNALIGNED+void run_container_union_inplace(run_container_t *src_1,+                                 const run_container_t *src_2) {+    // TODO: this could be a lot more efficient++    // we start out with inexpensive checks+    const bool if1 = run_container_is_full(src_1);+    const bool if2 = run_container_is_full(src_2);+    if (if1 || if2) {+        if (if1) {+            return;+        }+        if (if2) {+            run_container_copy(src_2, src_1);+            return;+        }+    }+    // we move the data to the end of the current array+    const int32_t maxoutput = src_1->n_runs + src_2->n_runs;+    const int32_t neededcapacity = maxoutput + src_1->n_runs;+    if (src_1->capacity < neededcapacity)+        run_container_grow(src_1, neededcapacity, true);+    memmove(src_1->runs + maxoutput, src_1->runs,+            src_1->n_runs * sizeof(rle16_t));+    rle16_t *inputsrc1 = src_1->runs + maxoutput;+    const int32_t input1nruns = src_1->n_runs;+    src_1->n_runs = 0;+    int32_t rlepos = 0;+    int32_t xrlepos = 0;++    rle16_t previousrle;+    if (inputsrc1[rlepos].value <= src_2->runs[xrlepos].value) {+        previousrle = run_container_append_first(src_1, inputsrc1[rlepos]);+        rlepos++;+    } else {+        previousrle = run_container_append_first(src_1, src_2->runs[xrlepos]);+        xrlepos++;+    }+    while ((xrlepos < src_2->n_runs) && (rlepos < input1nruns)) {+        rle16_t newrl;+        if (inputsrc1[rlepos].value <= src_2->runs[xrlepos].value) {+            newrl = inputsrc1[rlepos];+            rlepos++;+        } else {+            newrl = src_2->runs[xrlepos];+            xrlepos++;+        }+        run_container_append(src_1, newrl, &previousrle);+    }+    while (xrlepos < src_2->n_runs) {+        run_container_append(src_1, src_2->runs[xrlepos], &previousrle);+        xrlepos++;+    }+    while (rlepos < input1nruns) {+        run_container_append(src_1, inputsrc1[rlepos], &previousrle);+        rlepos++;+    }+}++/* Compute the symmetric difference of `src_1' and `src_2' and write the result+ * to `dst'+ * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */+CROARING_ALLOW_UNALIGNED+void run_container_xor(const run_container_t *src_1,+                       const run_container_t *src_2, run_container_t *dst) {+    // don't bother to convert xor with full range into negation+    // since negation is implemented similarly++    const int32_t neededcapacity = src_1->n_runs + src_2->n_runs;+    if (dst->capacity < neededcapacity)+        run_container_grow(dst, neededcapacity, false);++    rle16_t *out = dst->runs;+    int32_t n_out = 0;+    int32_t pos1 = 0;+    int32_t pos2 = 0;++    while ((pos1 < src_1->n_runs) && (pos2 < src_2->n_runs)) {+        if (src_1->runs[pos1].value <= src_2->runs[pos2].value) {+            n_out = run_append_exclusive(out, n_out, src_1->runs[pos1].value,+                                         src_1->runs[pos1].length);+            pos1++;+        } else {+            n_out = run_append_exclusive(out, n_out, src_2->runs[pos2].value,+                                         src_2->runs[pos2].length);+            pos2++;+        }+    }+    while (pos1 < src_1->n_runs) {+        n_out = run_append_exclusive(out, n_out, src_1->runs[pos1].value,+                                     src_1->runs[pos1].length);+        pos1++;+    }++    while (pos2 < src_2->n_runs) {+        n_out = run_append_exclusive(out, n_out, src_2->runs[pos2].value,+                                     src_2->runs[pos2].length);+        pos2++;+    }+    dst->n_runs = n_out;+}++/* Compute the intersection of src_1 and src_2 and write the result to+ * dst. It is assumed that dst is distinct from both src_1 and src_2. */+CROARING_ALLOW_UNALIGNED+void run_container_intersection(const run_container_t *src_1,+                                const run_container_t *src_2,+                                run_container_t *dst) {+    const bool if1 = run_container_is_full(src_1);+    const bool if2 = run_container_is_full(src_2);+    if (if1 || if2) {+        if (if1) {+            run_container_copy(src_2, dst);+            return;+        }+        if (if2) {+            run_container_copy(src_1, dst);+            return;+        }+    }+    // TODO: this could be a lot more efficient, could use SIMD optimizations+    const int32_t neededcapacity = src_1->n_runs + src_2->n_runs;+    if (dst->capacity < neededcapacity)+        run_container_grow(dst, neededcapacity, false);+    dst->n_runs = 0;+    int32_t rlepos = 0;+    int32_t xrlepos = 0;+    int32_t start = src_1->runs[rlepos].value;+    int32_t end = start + src_1->runs[rlepos].length + 1;+    int32_t xstart = src_2->runs[xrlepos].value;+    int32_t xend = xstart + src_2->runs[xrlepos].length + 1;+    while ((rlepos < src_1->n_runs) && (xrlepos < src_2->n_runs)) {+        if (end <= xstart) {+            ++rlepos;+            if (rlepos < src_1->n_runs) {+                start = src_1->runs[rlepos].value;+                end = start + src_1->runs[rlepos].length + 1;+            }+        } else if (xend <= start) {+            ++xrlepos;+            if (xrlepos < src_2->n_runs) {+                xstart = src_2->runs[xrlepos].value;+                xend = xstart + src_2->runs[xrlepos].length + 1;+            }+        } else {  // they overlap+            const int32_t lateststart = start > xstart ? start : xstart;+            int32_t earliestend;+            if (end == xend) {  // improbable+                earliestend = end;+                rlepos++;+                xrlepos++;+                if (rlepos < src_1->n_runs) {+                    start = src_1->runs[rlepos].value;+                    end = start + src_1->runs[rlepos].length + 1;+                }+                if (xrlepos < src_2->n_runs) {+                    xstart = src_2->runs[xrlepos].value;+                    xend = xstart + src_2->runs[xrlepos].length + 1;+                }+            } else if (end < xend) {+                earliestend = end;+                rlepos++;+                if (rlepos < src_1->n_runs) {+                    start = src_1->runs[rlepos].value;+                    end = start + src_1->runs[rlepos].length + 1;+                }++            } else {  // end > xend+                earliestend = xend;+                xrlepos++;+                if (xrlepos < src_2->n_runs) {+                    xstart = src_2->runs[xrlepos].value;+                    xend = xstart + src_2->runs[xrlepos].length + 1;+                }+            }+            dst->runs[dst->n_runs].value = (uint16_t)lateststart;+            dst->runs[dst->n_runs].length =+                (uint16_t)(earliestend - lateststart - 1);+            dst->n_runs++;+        }+    }+}++/* Compute the size of the intersection of src_1 and src_2 . */+CROARING_ALLOW_UNALIGNED+int run_container_intersection_cardinality(const run_container_t *src_1,+                                           const run_container_t *src_2) {+    const bool if1 = run_container_is_full(src_1);+    const bool if2 = run_container_is_full(src_2);+    if (if1 || if2) {+        if (if1) {+            return run_container_cardinality(src_2);+        }+        if (if2) {+            return run_container_cardinality(src_1);+        }+    }+    int answer = 0;+    int32_t rlepos = 0;+    int32_t xrlepos = 0;+    int32_t start = src_1->runs[rlepos].value;+    int32_t end = start + src_1->runs[rlepos].length + 1;+    int32_t xstart = src_2->runs[xrlepos].value;+    int32_t xend = xstart + src_2->runs[xrlepos].length + 1;+    while ((rlepos < src_1->n_runs) && (xrlepos < src_2->n_runs)) {+        if (end <= xstart) {+            ++rlepos;+            if (rlepos < src_1->n_runs) {+                start = src_1->runs[rlepos].value;+                end = start + src_1->runs[rlepos].length + 1;+            }+        } else if (xend <= start) {+            ++xrlepos;+            if (xrlepos < src_2->n_runs) {+                xstart = src_2->runs[xrlepos].value;+                xend = xstart + src_2->runs[xrlepos].length + 1;+            }+        } else {  // they overlap+            const int32_t lateststart = start > xstart ? start : xstart;+            int32_t earliestend;+            if (end == xend) {  // improbable+                earliestend = end;+                rlepos++;+                xrlepos++;+                if (rlepos < src_1->n_runs) {+                    start = src_1->runs[rlepos].value;+                    end = start + src_1->runs[rlepos].length + 1;+                }+                if (xrlepos < src_2->n_runs) {+                    xstart = src_2->runs[xrlepos].value;+                    xend = xstart + src_2->runs[xrlepos].length + 1;+                }+            } else if (end < xend) {+                earliestend = end;+                rlepos++;+                if (rlepos < src_1->n_runs) {+                    start = src_1->runs[rlepos].value;+                    end = start + src_1->runs[rlepos].length + 1;+                }++            } else {  // end > xend+                earliestend = xend;+                xrlepos++;+                if (xrlepos < src_2->n_runs) {+                    xstart = src_2->runs[xrlepos].value;+                    xend = xstart + src_2->runs[xrlepos].length + 1;+                }+            }+            answer += earliestend - lateststart;+        }+    }+    return answer;+}++CROARING_ALLOW_UNALIGNED+bool run_container_intersect(const run_container_t *src_1,+                             const run_container_t *src_2) {+    const bool if1 = run_container_is_full(src_1);+    const bool if2 = run_container_is_full(src_2);+    if (if1 || if2) {+        if (if1) {+            return !run_container_empty(src_2);+        }+        if (if2) {+            return !run_container_empty(src_1);+        }+    }+    int32_t rlepos = 0;+    int32_t xrlepos = 0;+    int32_t start = src_1->runs[rlepos].value;+    int32_t end = start + src_1->runs[rlepos].length + 1;+    int32_t xstart = src_2->runs[xrlepos].value;+    int32_t xend = xstart + src_2->runs[xrlepos].length + 1;+    while ((rlepos < src_1->n_runs) && (xrlepos < src_2->n_runs)) {+        if (end <= xstart) {+            ++rlepos;+            if (rlepos < src_1->n_runs) {+                start = src_1->runs[rlepos].value;+                end = start + src_1->runs[rlepos].length + 1;+            }+        } else if (xend <= start) {+            ++xrlepos;+            if (xrlepos < src_2->n_runs) {+                xstart = src_2->runs[xrlepos].value;+                xend = xstart + src_2->runs[xrlepos].length + 1;+            }+        } else {  // they overlap+            return true;+        }+    }+    return false;+}++/* Compute the difference of src_1 and src_2 and write the result to+ * dst. It is assumed that dst is distinct from both src_1 and src_2. */+CROARING_ALLOW_UNALIGNED+void run_container_andnot(const run_container_t *src_1,+                          const run_container_t *src_2, run_container_t *dst) {+    // following Java implementation as of June 2016++    if (dst->capacity < src_1->n_runs + src_2->n_runs)+        run_container_grow(dst, src_1->n_runs + src_2->n_runs, false);++    int32_t n_out = 0;++    int rlepos1 = 0;+    int rlepos2 = 0;+    int32_t start = src_1->runs[rlepos1].value;+    int32_t end = start + src_1->runs[rlepos1].length + 1;+    int32_t start2 = src_2->runs[rlepos2].value;+    int32_t end2 = start2 + src_2->runs[rlepos2].length + 1;++    while ((rlepos1 < src_1->n_runs) && (rlepos2 < src_2->n_runs)) {+        if (end <= start2) {+            // output the first run+            dst->runs[n_out++] = CROARING_MAKE_RLE16(start, end - start - 1);+            rlepos1++;+            if (rlepos1 < src_1->n_runs) {+                start = src_1->runs[rlepos1].value;+                end = start + src_1->runs[rlepos1].length + 1;+            }+        } else if (end2 <= start) {+            // exit the second run+            rlepos2++;+            if (rlepos2 < src_2->n_runs) {+                start2 = src_2->runs[rlepos2].value;+                end2 = start2 + src_2->runs[rlepos2].length + 1;+            }+        } else {+            if (start < start2) {+                dst->runs[n_out++] =+                    CROARING_MAKE_RLE16(start, start2 - start - 1);+            }+            if (end2 < end) {+                start = end2;+            } else {+                rlepos1++;+                if (rlepos1 < src_1->n_runs) {+                    start = src_1->runs[rlepos1].value;+                    end = start + src_1->runs[rlepos1].length + 1;+                }+            }+        }+    }+    if (rlepos1 < src_1->n_runs) {+        dst->runs[n_out++] = CROARING_MAKE_RLE16(start, end - start - 1);+        rlepos1++;+        if (rlepos1 < src_1->n_runs) {+            memcpy(dst->runs + n_out, src_1->runs + rlepos1,+                   sizeof(rle16_t) * (src_1->n_runs - rlepos1));+            n_out += src_1->n_runs - rlepos1;+        }+    }+    dst->n_runs = n_out;+}++/*+ * Print this container using printf (useful for debugging).+ */+void run_container_printf(const run_container_t *cont) {+    for (int i = 0; i < cont->n_runs; ++i) {+        uint16_t run_start = cont->runs[i].value;+        uint16_t le = cont->runs[i].length;+        printf("[%d,%d]", run_start, run_start + le);+    }+}++/*+ * Print this container using printf as a comma-separated list of 32-bit+ * integers starting at base.+ */+void run_container_printf_as_uint32_array(const run_container_t *cont,+                                          uint32_t base) {+    if (cont->n_runs == 0) return;+    {+        uint32_t run_start = base + cont->runs[0].value;+        uint16_t le = cont->runs[0].length;+        printf("%u", run_start);+        for (uint32_t j = 1; j <= le; ++j) printf(",%u", run_start + j);+    }+    for (int32_t i = 1; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;+        for (uint32_t j = 0; j <= le; ++j) printf(",%u", run_start + j);+    }+}++/*+ * Validate the container. Returns true if valid.+ */+CROARING_ALLOW_UNALIGNED+bool run_container_validate(const run_container_t *run, const char **reason) {+    if (run->n_runs < 0) {+        *reason = "negative run count";+        return false;+    }+    if (run->capacity < 0) {+        *reason = "negative run capacity";+        return false;+    }+    if (run->capacity < run->n_runs) {+        *reason = "capacity less than run count";+        return false;+    }++    if (run->n_runs == 0) {+        *reason = "zero run count";+        return false;+    }+    if (run->runs == NULL) {+        *reason = "NULL runs";+        return false;+    }++    // Use uint32_t to avoid overflow issues on ranges that contain UINT16_MAX.+    uint32_t last_end = 0;+    for (int i = 0; i < run->n_runs; ++i) {+        uint32_t start = run->runs[i].value;+        uint32_t end = start + run->runs[i].length + 1;+        if (end <= start) {+            *reason = "run start + length overflow";+            return false;+        }+        if (end > (1 << 16)) {+            *reason = "run start + length too large";+            return false;+        }+        if (start < last_end) {+            *reason = "run start less than last end";+            return false;+        }+        if (start == last_end && last_end != 0) {+            *reason = "run start equal to last end, should have combined";+            return false;+        }+        last_end = end;+    }+    return true;+}++CROARING_ALLOW_UNALIGNED+int32_t run_container_write(const run_container_t *container, char *buf) {+    uint16_t cast_16 = container->n_runs;+    uint16_t n_runs_le = croaring_htole16(cast_16);+    memcpy(buf, &n_runs_le, sizeof(uint16_t));+#if CROARING_IS_BIG_ENDIAN+    char *out = buf + sizeof(uint16_t);+    for (int32_t i = 0; i < container->n_runs; ++i) {+        uint16_t v_le = croaring_htole16(container->runs[i].value);+        uint16_t l_le = croaring_htole16(container->runs[i].length);+        memcpy(out, &v_le, sizeof(uint16_t));+        memcpy(out + sizeof(uint16_t), &l_le, sizeof(uint16_t));+        out += sizeof(rle16_t);+    }+#else+    memcpy(buf + sizeof(uint16_t), container->runs,+           container->n_runs * sizeof(rle16_t));+#endif+    return run_container_size_in_bytes(container);+}++int32_t run_container_read(int32_t cardinality, run_container_t *container,+                           const char *buf) {+    (void)cardinality;+    uint16_t cast_16;+    memcpy(&cast_16, buf, sizeof(uint16_t));+    container->n_runs = croaring_letoh16(cast_16);+    if (container->n_runs > container->capacity)+        run_container_grow(container, container->n_runs, false);+    if (container->n_runs > 0) {+#if CROARING_IS_BIG_ENDIAN+        const char *in = buf + sizeof(uint16_t);+        for (int32_t i = 0; i < container->n_runs; ++i) {+            uint16_t v_le, l_le;+            memcpy(&v_le, in, sizeof(uint16_t));+            memcpy(&l_le, in + sizeof(uint16_t), sizeof(uint16_t));+            container->runs[i].value = croaring_letoh16(v_le);+            container->runs[i].length = croaring_letoh16(l_le);+            in += sizeof(rle16_t);+        }+#else+        memcpy(container->runs, buf + sizeof(uint16_t),+               container->n_runs * sizeof(rle16_t));+#endif+    }+    return run_container_size_in_bytes(container);+}++CROARING_ALLOW_UNALIGNED+bool run_container_iterate(const run_container_t *cont, uint32_t base,+                           roaring_iterator iterator, void *ptr) {+    for (int i = 0; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;++        for (int j = 0; j <= le; ++j)+            if (!iterator(run_start + j, ptr)) return false;+    }+    return true;+}++CROARING_ALLOW_UNALIGNED+bool run_container_iterate64(const run_container_t *cont, uint32_t base,+                             roaring_iterator64 iterator, uint64_t high_bits,+                             void *ptr) {+    for (int i = 0; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;++        for (int j = 0; j <= le; ++j)+            if (!iterator(high_bits | (uint64_t)(run_start + j), ptr))+                return false;+    }+    return true;+}++CROARING_ALLOW_UNALIGNED+bool run_container_is_subset(const run_container_t *container1,+                             const run_container_t *container2) {+    int i1 = 0, i2 = 0;+    while (i1 < container1->n_runs && i2 < container2->n_runs) {+        int start1 = container1->runs[i1].value;+        int stop1 = start1 + container1->runs[i1].length;+        int start2 = container2->runs[i2].value;+        int stop2 = start2 + container2->runs[i2].length;+        if (start1 < start2) {+            return false;+        } else {  // start1 >= start2+            if (stop1 < stop2) {+                i1++;+            } else if (stop1 == stop2) {+                i1++;+                i2++;+            } else {  // stop1 > stop2+                i2++;+            }+        }+    }+    if (i1 == container1->n_runs) {+        return true;+    } else {+        return false;+    }+}++// TODO: write smart_append_exclusive version to match the overloaded 1 param+// Java version (or  is it even used?)++void run_container_smart_append_exclusive(run_container_t *src,+                                          const uint16_t start,+                                          const uint16_t length) {+    src->n_runs = run_append_exclusive(src->runs, src->n_runs, start, length);+}++CROARING_ALLOW_UNALIGNED+bool run_container_select(const run_container_t *container,+                          uint32_t *start_rank, uint32_t rank,+                          uint32_t *element) {+    for (int i = 0; i < container->n_runs; i++) {+        uint16_t length = container->runs[i].length;+        if (rank <= *start_rank + length) {+            uint16_t value = container->runs[i].value;+            *element = value + rank - (*start_rank);+            return true;+        } else+            *start_rank += length + 1;+    }+    return false;+}++CROARING_ALLOW_UNALIGNED+int run_container_rank(const run_container_t *container, uint16_t x) {+    int sum = 0;+    uint32_t x32 = x;+    for (int i = 0; i < container->n_runs; i++) {+        uint32_t startpoint = container->runs[i].value;+        uint32_t length = container->runs[i].length;+        uint32_t endpoint = length + startpoint;+        if (x <= endpoint) {+            if (x < startpoint) break;+            return sum + (x32 - startpoint) + 1;+        } else {+            sum += length + 1;+        }+    }+    return sum;+}+CROARING_ALLOW_UNALIGNED+uint32_t run_container_rank_many(const run_container_t *container,+                                 uint64_t start_rank, const uint32_t *begin,+                                 const uint32_t *end, uint64_t *ans) {+    const uint16_t high = (uint16_t)((*begin) >> 16);+    const uint32_t *iter = begin;+    int sum = 0;+    int i = 0;+    for (; iter != end; iter++) {+        uint32_t x = *iter;+        uint16_t xhigh = (uint16_t)(x >> 16);+        if (xhigh != high) return iter - begin;  // stop at next container++        uint32_t x32 = x & 0xFFFF;+        while (i < container->n_runs) {+            uint32_t startpoint = container->runs[i].value;+            uint32_t length = container->runs[i].length;+            uint32_t endpoint = length + startpoint;+            if (x32 <= endpoint) {+                if (x32 < startpoint) {+                    *(ans++) = start_rank + sum;+                } else {+                    *(ans++) = start_rank + sum + (x32 - startpoint) + 1;+                }+                break;+            } else {+                sum += length + 1;+                i++;+            }+        }+        if (i >= container->n_runs) *(ans++) = start_rank + sum;+    }++    return iter - begin;+}++CROARING_ALLOW_UNALIGNED+int run_container_get_index(const run_container_t *container, uint16_t x) {+    if (run_container_contains(container, x)) {+        int sum = 0;+        uint32_t x32 = x;+        for (int i = 0; i < container->n_runs; i++) {+            uint32_t startpoint = container->runs[i].value;+            uint32_t length = container->runs[i].length;+            uint32_t endpoint = length + startpoint;+            if (x <= endpoint) {+                if (x < startpoint) break;+                return sum + (x32 - startpoint);+            } else {+                sum += length + 1;+            }+        }+        return sum - 1;+    } else {+        return -1;+    }+}+#define CROARING_ENABLE_AVX512_RUN_CONTAINER_CARDINALITY 0++#if defined(CROARING_IS_X64) && CROARING_COMPILER_SUPPORTS_AVX512++#if CROARING_ENABLE_AVX512_RUN_CONTAINER_CARDINALITY+CROARING_TARGET_AVX512+CROARING_ALLOW_UNALIGNED+/* Get the cardinality of `run'. Requires an actual computation. */+static inline int _avx512_run_container_cardinality(+    const run_container_t *run) {+    const int32_t n_runs = run->n_runs;+    const rle16_t *runs = run->runs;++    int32_t k = 0;+    const int32_t step512 = sizeof(__m512i) / sizeof(rle16_t);+    __m512i total = _mm512_setzero_si512();+    for (; k + step512 <= n_runs; k += step512) {+        __m512i ymm1 = _mm512_loadu_si512((const __m512i *)(runs + k));+        __m512i justlengths = _mm512_srli_epi32(ymm1, 16);+        total = _mm512_add_epi32(total, justlengths);+    }+    if (k < n_runs) {+        __m512i ymm1 = _mm512_maskz_loadu_epi32((1 << (n_runs - k)) - 1,+                                                (const __m512i *)(runs + k));+        __m512i justlengths = _mm512_srli_epi32(ymm1, 16);+        total = _mm512_add_epi32(total, justlengths);+    }+    return _mm512_reduce_add_epi32(total) + n_runs;+}++CROARING_UNTARGET_AVX512+#endif  // CROARING_ENABLE_AVX512_RUN_CONTAINER_CARDINALITY++CROARING_TARGET_AVX2+CROARING_ALLOW_UNALIGNED+/* Get the cardinality of `run'. Requires an actual computation. */+static inline int _avx2_run_container_cardinality(const run_container_t *run) {+    const int32_t n_runs = run->n_runs;+    const rle16_t *runs = run->runs;++    /* by initializing with n_runs, we omit counting the +1 for each pair. */+    int sum = n_runs;+    int32_t k = 0;+    const int32_t step = sizeof(__m256i) / sizeof(rle16_t);+    if (n_runs > step) {+        __m256i total = _mm256_setzero_si256();+        for (; k + step <= n_runs; k += step) {+            __m256i ymm1 = _mm256_loadu_si256((const __m256i *)(runs + k));+            __m256i justlengths = _mm256_srli_epi32(ymm1, 16);+            total = _mm256_add_epi32(total, justlengths);+        }+        // a store might be faster than extract?+        uint32_t buffer[sizeof(__m256i) / sizeof(rle16_t)];+        _mm256_storeu_si256((__m256i *)buffer, total);+        sum += (buffer[0] + buffer[1]) + (buffer[2] + buffer[3]) ++               (buffer[4] + buffer[5]) + (buffer[6] + buffer[7]);+    }+    for (; k < n_runs; ++k) {+        sum += runs[k].length;+    }++    return sum;+}++CROARING_ALLOW_UNALIGNED+int _avx2_run_container_to_uint32_array(void *vout, const run_container_t *cont,+                                        uint32_t base) {+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;++    for (int i = 0; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;+        if (le < 8) {+            for (int j = 0; j <= le; ++j) {+                uint32_t val = run_start + j;+                memcpy(out + outpos, &val,+                       sizeof(uint32_t));  // should be compiled as a MOV on x64+                outpos++;+            }+        } else {+            int j = 0;+            __m256i run_start_v = _mm256_set1_epi32(run_start);+            // [8,8,8,8....]+            __m256i inc = _mm256_set1_epi32(8);+            // used for generate sequence:+            // [0, 1, 2, 3...], [8, 9, 10,...]+            __m256i delta = _mm256_setr_epi32(0, 1, 2, 3, 4, 5, 6, 7);+            for (j = 0; j + 8 <= le; j += 8) {+                __m256i val_v = _mm256_add_epi32(run_start_v, delta);+                _mm256_storeu_si256((__m256i *)(out + outpos), val_v);+                delta = _mm256_add_epi32(inc, delta);+                outpos += 8;+            }+            for (; j <= le; ++j) {+                uint32_t val = run_start + j;+                memcpy(out + outpos, &val,+                       sizeof(uint32_t));  // should be compiled as a MOV on x64+                outpos++;+            }+        }+    }+    return outpos;+}++CROARING_UNTARGET_AVX2++/* Get the cardinality of `run'. Requires an actual computation. */+CROARING_ALLOW_UNALIGNED+static inline int _scalar_run_container_cardinality(+    const run_container_t *run) {+    const int32_t n_runs = run->n_runs;+    const rle16_t *runs = run->runs;++    /* by initializing with n_runs, we omit counting the +1 for each pair. */+    int sum = n_runs;+    for (int k = 0; k < n_runs; ++k) {+        sum += runs[k].length;+    }++    return sum;+}++int run_container_cardinality(const run_container_t *run) {+    // Empirically AVX-512 is not always faster than AVX2+#if CROARING_COMPILER_SUPPORTS_AVX512 && \+    CROARING_ENABLE_AVX512_RUN_CONTAINER_CARDINALITY+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX512) {+        return _avx512_run_container_cardinality(run);+    } else+#endif+        if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+        return _avx2_run_container_cardinality(run);+    } else {+        return _scalar_run_container_cardinality(run);+    }+}++CROARING_ALLOW_UNALIGNED+int _scalar_run_container_to_uint32_array(void *vout,+                                          const run_container_t *cont,+                                          uint32_t base) {+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;+    for (int i = 0; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;+        for (int j = 0; j <= le; ++j) {+            uint32_t val = run_start + j;+            memcpy(out + outpos, &val,+                   sizeof(uint32_t));  // should be compiled as a MOV on x64+            outpos++;+        }+    }+    return outpos;+}++int run_container_to_uint32_array(void *vout, const run_container_t *cont,+                                  uint32_t base) {+    if (croaring_hardware_support() & ROARING_SUPPORTS_AVX2) {+        return _avx2_run_container_to_uint32_array(vout, cont, base);+    } else {+        return _scalar_run_container_to_uint32_array(vout, cont, base);+    }+}++#else++/* Get the cardinality of `run'. Requires an actual computation. */+CROARING_ALLOW_UNALIGNED+int run_container_cardinality(const run_container_t *run) {+    const int32_t n_runs = run->n_runs;+    const rle16_t *runs = run->runs;++    /* by initializing with n_runs, we omit counting the +1 for each pair. */+    int sum = n_runs;+    for (int k = 0; k < n_runs; ++k) {+        sum += runs[k].length;+    }++    return sum;+}++CROARING_ALLOW_UNALIGNED+int run_container_to_uint32_array(void *vout, const run_container_t *cont,+                                  uint32_t base) {+    int outpos = 0;+    uint32_t *out = (uint32_t *)vout;+    for (int i = 0; i < cont->n_runs; ++i) {+        uint32_t run_start = base + cont->runs[i].value;+        uint16_t le = cont->runs[i].length;+        for (int j = 0; j <= le; ++j) {+            uint32_t val = run_start + j;+            memcpy(out + outpos, &val,+                   sizeof(uint32_t));  // should be compiled as a MOV on x64+            outpos++;+        }+    }+    return outpos;+}++#endif++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif+#if defined(__GNUC__) && !defined(__clang__)+#pragma GCC diagnostic pop+#endif
+ cbits/CRoaring/src/isadetection.c view
@@ -0,0 +1,447 @@++/* From+https://github.com/endorno/pytorch/blob/master/torch/lib/TH/generic/simd/simd.h+Highly modified.++Copyright (c) 2016-     Facebook, Inc            (Adam Paszke)+Copyright (c) 2014-     Facebook, Inc            (Soumith Chintala)+Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)+Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)+Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)+Copyright (c) 2011-2013 NYU                      (Clement Farabet)+Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou,+Iain Melvin, Jason Weston) Copyright (c) 2006      Idiap Research Institute+(Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert,+Samy Bengio, Johnny Mariethoz)++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1. Redistributions of source code must retain the above copyright+   notice, this list of conditions and the following disclaimer.++2. Redistributions in binary form must reproduce the above copyright+   notice, this list of conditions and the following disclaimer in the+   documentation and/or other materials provided with the distribution.++3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories+America and IDIAP Research Institute nor the names of its contributors may be+   used to endorse or promote products derived from this software without+   specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE+POSSIBILITY OF SUCH DAMAGE.+*/++#include <stdbool.h>+#include <stdint.h>+#include <stdlib.h>++// Binaries produced by Visual Studio 19.38 with solely AVX2 routines+// can compile to AVX-512 thus causing crashes on non-AVX-512 systems.+// This appears to affect VS 17.8 and 17.9. We disable AVX-512 and AVX2+// on these systems. It seems that ClangCL is not affected.+// https://github.com/RoaringBitmap/CRoaring/pull/603+#ifndef __clang__+#if _MSC_VER == 1938+#define ROARING_DISABLE_AVX 1+#endif  // _MSC_VER == 1938+#endif  // __clang__++#ifdef __FILC__+#include <stdfil.h>+#endif++// We need portability.h to be included first, see+// https://github.com/RoaringBitmap/CRoaring/issues/394+#include <roaring/portability.h>+#if CROARING_REGULAR_VISUAL_STUDIO+#include <intrin.h>+#elif CROARING_IS_X64 &&                                            \+    ((defined(HAVE_GCC_GET_CPUID) && defined(USE_GCC_GET_CPUID)) || \+     defined(__FILC__))+#include <cpuid.h>+#endif  // CROARING_REGULAR_VISUAL_STUDIO+#include <roaring/isadetection.h>++#if CROARING_IS_X64+#ifndef CROARING_COMPILER_SUPPORTS_AVX512+#error "CROARING_COMPILER_SUPPORTS_AVX512 needs to be defined."+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+#endif++#if CROARING_IS_ARM64 && defined(__linux__)+#include <sys/auxv.h>+// The kernel advertises SVE in AT_HWCAP and SVE2 in AT_HWCAP2. Older+// headers may not define these constants, so we provide the kernel's values+// (we deliberately do not include <asm/hwcap.h>, which is not available on+// all toolchains, e.g., musl without linux-headers).+#ifndef AT_HWCAP2+#define AT_HWCAP2 26+#endif+#ifndef HWCAP_SVE+#define HWCAP_SVE (1 << 22)+#endif+#ifndef HWCAP2_SVE2+#define HWCAP2_SVE2 (1 << 1)+#endif+#endif  // CROARING_IS_ARM64 && defined(__linux__)++#if CROARING_IS_ARM64 && defined(_WIN32)+#ifndef _WINDOWS_+// We avoid including <windows.h> (macro pollution); this matches the+// declaration in the Windows SDK (BOOL WINAPI+// IsProcessorFeaturePresent(DWORD)).+#ifdef __cplusplus+#define CROARING_EXTERN_C extern "C"+#else+#define CROARING_EXTERN_C+#endif+CROARING_EXTERN_C __declspec(dllimport) int __stdcall IsProcessorFeaturePresent(+    unsigned long ProcessorFeature);+#endif  // _WINDOWS_+// Only recent Windows SDKs define these processor features.+#ifndef PF_ARM_SVE_INSTRUCTIONS_AVAILABLE+#define PF_ARM_SVE_INSTRUCTIONS_AVAILABLE 46+#endif+#ifndef PF_ARM_SVE2_INSTRUCTIONS_AVAILABLE+#define PF_ARM_SVE2_INSTRUCTIONS_AVAILABLE 47+#endif+#endif  // CROARING_IS_ARM64 && defined(_WIN32)++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif+enum croaring_instruction_set {+    CROARING_DEFAULT = 0x0,+    CROARING_NEON = 0x1,+    CROARING_AVX2 = 0x4,+    CROARING_SSE42 = 0x8,+    CROARING_PCLMULQDQ = 0x10,+    CROARING_BMI1 = 0x20,+    CROARING_BMI2 = 0x40,+    CROARING_ALTIVEC = 0x80,+    CROARING_AVX512F = 0x100,+    CROARING_AVX512DQ = 0x200,+    CROARING_AVX512BW = 0x400,+    CROARING_AVX512VBMI2 = 0x800,+    CROARING_AVX512BITALG = 0x1000,+    CROARING_AVX512VPOPCNTDQ = 0x2000,+    CROARING_SVE = 0x4000,+    CROARING_UNINITIALIZED = 0x8000,+    CROARING_SVE2 = 0x10000+};++#if CROARING_COMPILER_SUPPORTS_AVX512+unsigned int CROARING_AVX512_REQUIRED =+    (CROARING_AVX512F | CROARING_AVX512DQ | CROARING_AVX512BW |+     CROARING_AVX512VBMI2 | CROARING_AVX512BITALG | CROARING_AVX512VPOPCNTDQ);+#endif++#if CROARING_IS_X64  // x64++static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx,+                         uint32_t *edx) {+#if CROARING_REGULAR_VISUAL_STUDIO+    int cpu_info[4];+    __cpuidex(cpu_info, *eax, *ecx);+    *eax = cpu_info[0];+    *ebx = cpu_info[1];+    *ecx = cpu_info[2];+    *edx = cpu_info[3];+#elif (defined(HAVE_GCC_GET_CPUID) && defined(USE_GCC_GET_CPUID)) || \+    defined(__FILC__)+    uint32_t level = *eax;+    __get_cpuid(level, eax, ebx, ecx, edx);+#else+    uint32_t a = *eax, b, c = *ecx, d;+    __asm__("cpuid\n\t" : "+a"(a), "=b"(b), "+c"(c), "=d"(d));+    *eax = a;+    *ebx = b;+    *ecx = c;+    *edx = d;+#endif+}++static inline uint64_t xgetbv(void) {+#if defined(_MSC_VER)+    return _xgetbv(0);+#elif defined(__FILC__)+    return zxgetbv();+#else+    uint32_t xcr0_lo, xcr0_hi;+    __asm__("xgetbv\n\t" : "=a"(xcr0_lo), "=d"(xcr0_hi) : "c"(0));+    return xcr0_lo | ((uint64_t)xcr0_hi << 32);+#endif+}++/**+ * This is a relatively expensive function but it will get called at most+ * *once* per compilation units. Normally, the CRoaring library is built+ * as one compilation unit.+ */+static inline uint32_t dynamic_croaring_detect_supported_architectures(void) {+    uint32_t eax, ebx, ecx, edx;+    uint32_t host_isa = 0x0;+    // Can be found on Intel ISA Reference for CPUID+    static uint32_t cpuid_avx2_bit =+        1 << 5;  ///< @private Bit 5 of EBX for EAX=0x7+    static uint32_t cpuid_bmi1_bit =+        1 << 3;  ///< @private bit 3 of EBX for EAX=0x7+    static uint32_t cpuid_bmi2_bit =+        1 << 8;  ///< @private bit 8 of EBX for EAX=0x7+    static uint32_t cpuid_avx512f_bit =+        1 << 16;  ///< @private bit 16 of EBX for EAX=0x7+    static uint32_t cpuid_avx512dq_bit =+        1 << 17;  ///< @private bit 17 of EBX for EAX=0x7+    static uint32_t cpuid_avx512bw_bit =+        1 << 30;  ///< @private bit 30 of EBX for EAX=0x7+    static uint32_t cpuid_avx512vbmi2_bit =+        1 << 6;  ///< @private bit 6 of ECX for EAX=0x7+    static uint32_t cpuid_avx512bitalg_bit =+        1 << 12;  ///< @private bit 12 of ECX for EAX=0x7+    static uint32_t cpuid_avx512vpopcntdq_bit =+        1 << 14;  ///< @private bit 14 of ECX for EAX=0x7+    static uint64_t cpuid_avx256_saved = 1 << 2;  ///< @private bit 2 = AVX+    static uint64_t cpuid_avx512_saved =+        7 << 5;  ///< @private bits 5,6,7 = opmask, ZMM_hi256, hi16_ZMM+    static uint32_t cpuid_sse42_bit =+        1 << 20;  ///< @private bit 20 of ECX for EAX=0x1+    static uint32_t cpuid_osxsave =+        (1 << 26) | (1 << 27);  ///< @private bits 26+27 of ECX for EAX=0x1+    static uint32_t cpuid_pclmulqdq_bit =+        1 << 1;  ///< @private bit  1 of ECX for EAX=0x1++    // EBX for EAX=0x1+    eax = 0x1;+    ecx = 0x0;+    cpuid(&eax, &ebx, &ecx, &edx);++    if (ecx & cpuid_sse42_bit) {+        host_isa |= CROARING_SSE42;+    } else {+        return host_isa;  // everything after is redundant+    }++    if (ecx & cpuid_pclmulqdq_bit) {+        host_isa |= CROARING_PCLMULQDQ;+    }++    if ((ecx & cpuid_osxsave) != cpuid_osxsave) {+        return host_isa;+    }++    // xgetbv for checking if the OS saves registers+    uint64_t xcr0 = xgetbv();++    if ((xcr0 & cpuid_avx256_saved) == 0) {+        return host_isa;+    }++    // ECX for EAX=0x7+    eax = 0x7;+    ecx = 0x0;+    cpuid(&eax, &ebx, &ecx, &edx);+    if (ebx & cpuid_avx2_bit) {+        host_isa |= CROARING_AVX2;+    }+    if (ebx & cpuid_bmi1_bit) {+        host_isa |= CROARING_BMI1;+    }++    if (ebx & cpuid_bmi2_bit) {+        host_isa |= CROARING_BMI2;+    }++    if (!((xcr0 & cpuid_avx512_saved) == cpuid_avx512_saved)) {+        return host_isa;+    }++    if (ebx & cpuid_avx512f_bit) {+        host_isa |= CROARING_AVX512F;+    }++    if (ebx & cpuid_avx512bw_bit) {+        host_isa |= CROARING_AVX512BW;+    }++    if (ebx & cpuid_avx512dq_bit) {+        host_isa |= CROARING_AVX512DQ;+    }++    if (ecx & cpuid_avx512vbmi2_bit) {+        host_isa |= CROARING_AVX512VBMI2;+    }++    if (ecx & cpuid_avx512bitalg_bit) {+        host_isa |= CROARING_AVX512BITALG;+    }++    if (ecx & cpuid_avx512vpopcntdq_bit) {+        host_isa |= CROARING_AVX512VPOPCNTDQ;+    }++    return host_isa;+}++#endif  // end SIMD extension detection code++#if CROARING_IS_ARM64++static inline uint32_t dynamic_croaring_detect_supported_architectures(void) {+    // NEON is mandatory on AArch64.+    uint32_t host_isa = CROARING_NEON;+#if defined(__linux__)+    unsigned long hwcap = getauxval(AT_HWCAP);+    unsigned long hwcap2 = getauxval(AT_HWCAP2);+    if (hwcap & HWCAP_SVE) {+        host_isa |= CROARING_SVE;+        // We only claim SVE2 when SVE is also present. Before Linux 6.14, the+        // kernel set HWCAP2_SVE2 on processors implementing SME(2) but not+        // SVE, because SVE2 instructions are available in streaming mode. Our+        // SVE2 code runs in non-streaming mode and needs actual SVE.+        if (hwcap2 & HWCAP2_SVE2) {+            host_isa |= CROARING_SVE2;+        }+    }+#elif defined(_WIN32)+    if (IsProcessorFeaturePresent(PF_ARM_SVE_INSTRUCTIONS_AVAILABLE)) {+        host_isa |= CROARING_SVE;+        // As on Linux, require SVE before claiming SVE2.+        if (IsProcessorFeaturePresent(PF_ARM_SVE2_INSTRUCTIONS_AVAILABLE)) {+            host_isa |= CROARING_SVE2;+        }+    }+#endif+    // On other systems (e.g., macOS, where Apple Silicon has no SVE), we only+    // report NEON.+    return host_isa;+}++#endif  // CROARING_IS_ARM64++#if CROARING_IS_X64 || CROARING_IS_ARM64++#if CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_CPP+static inline uint32_t croaring_detect_supported_architectures(void) {+    // thread-safe as per the C++11 standard.+    static uint32_t buffer = dynamic_croaring_detect_supported_architectures();+    return buffer;+}+#elif CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_C+static uint32_t croaring_detect_supported_architectures(void) {+    // we use an atomic for thread safety+    static _Atomic uint32_t buffer = CROARING_UNINITIALIZED;+    if (buffer == CROARING_UNINITIALIZED) {+        // atomicity is sufficient+        buffer = dynamic_croaring_detect_supported_architectures();+    }+    return buffer;+}+#else+// If we do not have atomics, we do the best we can.+static inline uint32_t croaring_detect_supported_architectures(void) {+    static uint32_t buffer = CROARING_UNINITIALIZED;+    if (buffer == CROARING_UNINITIALIZED) {+        buffer = dynamic_croaring_detect_supported_architectures();+    }+    return buffer;+}+#endif  // CROARING_C_ATOMIC++#endif  // CROARING_IS_X64 || CROARING_IS_ARM64++// The cached result of croaring_hardware_support(), which threads calling it+// for the first time all write.+#if CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_CPP+typedef std::atomic<int> croaring_atomic_int;+#define CROARING_SUPPORT_UNKNOWN \+    { 0xFFFFFFF }  // no copy before C++17+#elif CROARING_ATOMIC_IMPL == CROARING_ATOMIC_IMPL_C+typedef _Atomic int croaring_atomic_int;+#define CROARING_SUPPORT_UNKNOWN 0xFFFFFFF+#else+typedef int croaring_atomic_int;+#define CROARING_SUPPORT_UNKNOWN 0xFFFFFFF+#endif++#if CROARING_IS_X64  // x64++#ifdef ROARING_DISABLE_AVX++int croaring_hardware_support(void) { return 0; }++#elif defined(__AVX512F__) && defined(__AVX512DQ__) &&   \+    defined(__AVX512BW__) && defined(__AVX512VBMI2__) && \+    defined(__AVX512BITALG__) && defined(__AVX512VPOPCNTDQ__)+int croaring_hardware_support(void) {+    return ROARING_SUPPORTS_AVX2 | ROARING_SUPPORTS_AVX512;+}+#elif defined(__AVX2__)++int croaring_hardware_support(void) {+    static croaring_atomic_int support = CROARING_SUPPORT_UNKNOWN;+    if (support == 0xFFFFFFF) {+        bool avx512_support = false;+#if CROARING_COMPILER_SUPPORTS_AVX512+        avx512_support =+            ((croaring_detect_supported_architectures() &+              CROARING_AVX512_REQUIRED) == CROARING_AVX512_REQUIRED);+#endif+        support = ROARING_SUPPORTS_AVX2 |+                  (avx512_support ? ROARING_SUPPORTS_AVX512 : 0);+    }+    return support;+}+#else++int croaring_hardware_support(void) {+    static croaring_atomic_int support = CROARING_SUPPORT_UNKNOWN;+    if (support == 0xFFFFFFF) {+        bool has_avx2 = (croaring_detect_supported_architectures() &+                         CROARING_AVX2) == CROARING_AVX2;+        bool has_avx512 = false;+#if CROARING_COMPILER_SUPPORTS_AVX512+        has_avx512 = (croaring_detect_supported_architectures() &+                      CROARING_AVX512_REQUIRED) == CROARING_AVX512_REQUIRED;+#endif  // CROARING_COMPILER_SUPPORTS_AVX512+        support = (has_avx2 ? ROARING_SUPPORTS_AVX2 : 0) |+                  (has_avx512 ? ROARING_SUPPORTS_AVX512 : 0);+    }+    return support;+}+#endif++#endif  // CROARING_IS_X64 // x64++#if CROARING_IS_ARM64++int croaring_hardware_support(void) {+    static croaring_atomic_int support = CROARING_SUPPORT_UNKNOWN;+    if (support == 0xFFFFFFF) {+        uint32_t isa = croaring_detect_supported_architectures();+        support = ((isa & CROARING_SVE) ? ROARING_SUPPORTS_SVE : 0) |+                  ((isa & CROARING_SVE2) ? ROARING_SUPPORTS_SVE2 : 0);+    }+    return support;+}++#endif  // CROARING_IS_ARM64+#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/license-comment.h view
@@ -0,0 +1,52 @@+/*+ * The CRoaring project is under a dual license (Apache/MIT).+ * Users of the library may choose one or the other license.+ */+/*+ * Copyright 2016-2022 The CRoaring authors+ *+ * Licensed under the Apache License, Version 2.0 (the "License");+ * you may not use this file except in compliance with the License.+ * You may obtain a copy of the License at+ *+ *    http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+/*+ * MIT License+ *+ * Copyright 2016-2022 The CRoaring authors+ *+ * Permission is hereby granted, free of charge, to any+ * person obtaining a copy of this software and associated+ * documentation files (the "Software"), to deal in the+ * Software without restriction, including without+ * limitation the rights to use, copy, modify, merge,+ * publish, distribute, sublicense, and/or sell copies of+ * the Software, and to permit persons to whom the Software+ * is furnished to do so, subject to the following+ * conditions:+ *+ * The above copyright notice and this permission notice+ * shall be included in all copies or substantial portions+ * of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF+ * ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED+ * TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A+ * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT+ * SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY+ * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION+ * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR+ * IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER+ * DEALINGS IN THE SOFTWARE.+ *+ * SPDX-License-Identifier: MIT+ */
+ cbits/CRoaring/src/memory.c view
@@ -0,0 +1,64 @@+#include <stdlib.h>++#include <roaring/memory.h>++// without the following, we get lots of warnings about posix_memalign+#ifndef __cplusplus+extern int posix_memalign(void** __memptr, size_t __alignment, size_t __size);+#endif  //__cplusplus // C++ does not have a well defined signature++// portable version of  posix_memalign+static void* roaring_bitmap_aligned_malloc(size_t alignment, size_t size) {+    void* p;+#ifdef _MSC_VER+    p = _aligned_malloc(size, alignment);+#elif defined(__MINGW32__) || defined(__MINGW64__)+    p = __mingw_aligned_malloc(size, alignment);+#else+    // somehow, if this is used before including "x86intrin.h", it creates an+    // implicit defined warning.+    if (posix_memalign(&p, alignment, size) != 0) return NULL;+#endif+    return p;+}++static void roaring_bitmap_aligned_free(void* memblock) {+#ifdef _MSC_VER+    _aligned_free(memblock);+#elif defined(__MINGW32__) || defined(__MINGW64__)+    __mingw_aligned_free(memblock);+#else+    free(memblock);+#endif+}++static roaring_memory_t global_memory_hook = {+    .malloc = malloc,+    .realloc = realloc,+    .calloc = calloc,+    .free = free,+    .aligned_malloc = roaring_bitmap_aligned_malloc,+    .aligned_free = roaring_bitmap_aligned_free,+};++void roaring_init_memory_hook(roaring_memory_t memory_hook) {+    global_memory_hook = memory_hook;+}++void* roaring_malloc(size_t n) { return global_memory_hook.malloc(n); }++void* roaring_realloc(void* p, size_t new_sz) {+    return global_memory_hook.realloc(p, new_sz);+}++void* roaring_calloc(size_t n_elements, size_t element_size) {+    return global_memory_hook.calloc(n_elements, element_size);+}++void roaring_free(void* p) { global_memory_hook.free(p); }++void* roaring_aligned_malloc(size_t alignment, size_t size) {+    return global_memory_hook.aligned_malloc(alignment, size);+}++void roaring_aligned_free(void* p) { global_memory_hook.aligned_free(p); }
+ cbits/CRoaring/src/roaring.c view
@@ -0,0 +1,3856 @@+#include <assert.h>+#include <inttypes.h>+#include <limits.h>+#include <stdarg.h>+#include <stdint.h>+#include <stdio.h>+#include <string.h>++#include <roaring/roaring.h>++// Include after roaring.h+#include <roaring/array_util.h>+#include <roaring/bitset_util.h>+#include <roaring/containers/containers.h>+#include <roaring/roaring_array.h>++#ifdef __cplusplus+using namespace ::roaring::internal;++extern "C" {+namespace roaring {+namespace api {+#endif++#define CROARING_SERIALIZATION_ARRAY_UINT32 1+#define CROARING_SERIALIZATION_CONTAINER 2+extern inline bool roaring_bitmap_contains(const roaring_bitmap_t *r,+                                           uint32_t val);+extern inline int roaring_trailing_zeroes(unsigned long long input_num);+extern inline int roaring_leading_zeroes(unsigned long long input_num);+extern inline void roaring_bitmap_init_cleared(roaring_bitmap_t *r);+extern inline bool roaring_bitmap_get_copy_on_write(const roaring_bitmap_t *r);+extern inline void roaring_bitmap_set_copy_on_write(roaring_bitmap_t *r,+                                                    bool cow);+extern inline roaring_bitmap_t *roaring_bitmap_create(void);+extern inline void roaring_bitmap_add_range(roaring_bitmap_t *r, uint64_t min,+                                            uint64_t max);+extern inline void roaring_bitmap_remove_range(roaring_bitmap_t *r,+                                               uint64_t min, uint64_t max);++static inline bool is_cow(const roaring_bitmap_t *r) {+    return r->high_low_container.flags & ROARING_FLAG_COW;+}+static inline bool is_frozen(const roaring_bitmap_t *r) {+    return r->high_low_container.flags & ROARING_FLAG_FROZEN;+}++// this is like roaring_bitmap_add, but it populates pointer arguments in such a+// way+// that we can recover the container touched, which, in turn can be used to+// accelerate some functions (when you repeatedly need to add to the same+// container)+static inline container_t *containerptr_roaring_bitmap_add(roaring_bitmap_t *r,+                                                           uint32_t val,+                                                           uint8_t *type,+                                                           int *index) {+    roaring_array_t *ra = &r->high_low_container;++    uint16_t hb = val >> 16;+    const int i = ra_get_index(ra, hb);+    if (i >= 0) {+        ra_unshare_container_at_index(ra, (uint16_t)i);+        container_t *c = ra_get_container_at_index(ra, (uint16_t)i, type);+        uint8_t new_type = *type;+        container_t *c2 = container_add(c, val & 0xFFFF, *type, &new_type);+        *index = i;+        if (c2 != c) {+            container_free(c, *type);+            ra_set_container_at_index(ra, i, c2, new_type);+            *type = new_type;+            return c2;+        } else {+            return c;+        }+    } else {+        array_container_t *new_ac = array_container_create();+        container_t *c =+            container_add(new_ac, val & 0xFFFF, ARRAY_CONTAINER_TYPE, type);+        // we could just assume that it stays an array container+        ra_insert_new_key_value_at(ra, -i - 1, hb, c, *type);+        *index = -i - 1;+        return c;+    }+}++roaring_bitmap_t *roaring_bitmap_create_with_capacity(uint32_t cap) {+    roaring_bitmap_t *ans =+        (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t));+    if (!ans) {+        return NULL;+    }+    bool is_ok = ra_init_with_capacity(&ans->high_low_container, cap);+    if (!is_ok) {+        roaring_free(ans);+        return NULL;+    }+    return ans;+}++bool roaring_bitmap_init_with_capacity(roaring_bitmap_t *r, uint32_t cap) {+    return ra_init_with_capacity(&r->high_low_container, cap);+}++static inline void add_bulk_impl(roaring_bitmap_t *r,+                                 roaring_bulk_context_t *context,+                                 uint32_t val) {+    uint16_t key = val >> 16;+    if (context->container == NULL || context->key != key) {+        uint8_t typecode;+        int idx;+        context->container =+            containerptr_roaring_bitmap_add(r, val, &typecode, &idx);+        context->typecode = typecode;+        context->idx = idx;+        context->key = key;+    } else {+        // no need to seek the container, it is at hand+        // because we already have the container at hand, we can do the+        // insertion directly, bypassing the roaring_bitmap_add call+        uint8_t new_typecode;+        container_t *container2 = container_add(+            context->container, val & 0xFFFF, context->typecode, &new_typecode);+        if (container2 != context->container) {+            // rare instance when we need to change the container type+            container_free(context->container, context->typecode);+            ra_set_container_at_index(&r->high_low_container, context->idx,+                                      container2, new_typecode);+            context->typecode = new_typecode;+            context->container = container2;+        }+    }+}++void roaring_bitmap_add_many(roaring_bitmap_t *r, size_t n_args,+                             const uint32_t *vals) {+    uint32_t val;+    const uint32_t *start = vals;+    const uint32_t *end = vals + n_args;+    const uint32_t *current_val = start;++    if (n_args == 0) {+        return;+    }++    uint8_t typecode;+    int idx;+    container_t *container;+    val = *current_val;+    container = containerptr_roaring_bitmap_add(r, val, &typecode, &idx);+    roaring_bulk_context_t context = {container, idx, (uint16_t)(val >> 16),+                                      typecode};++    for (; current_val != end; current_val++) {+        memcpy(&val, current_val, sizeof(val));+        add_bulk_impl(r, &context, val);+    }+}++void roaring_bitmap_add_bulk(roaring_bitmap_t *r,+                             roaring_bulk_context_t *context, uint32_t val) {+    add_bulk_impl(r, context, val);+}++bool roaring_bitmap_contains_bulk(const roaring_bitmap_t *r,+                                  roaring_bulk_context_t *context,+                                  uint32_t val) {+    uint16_t key = val >> 16;+    if (context->container == NULL || context->key != key) {+        int32_t start_idx = -1;+        if (context->container != NULL && context->key < key) {+            start_idx = context->idx;+        }+        int idx = ra_advance_until(&r->high_low_container, key, start_idx);+        if (idx == ra_get_size(&r->high_low_container)) {+            return false;+        }+        uint8_t typecode;+        context->container = ra_get_container_at_index(+            &r->high_low_container, (uint16_t)idx, &typecode);+        context->typecode = typecode;+        context->idx = idx;+        context->key =+            ra_get_key_at_index(&r->high_low_container, (uint16_t)idx);+        // ra_advance_until finds the next key >= the target, we found a later+        // container.+        if (context->key != key) {+            return false;+        }+    }+    // context is now set up+    return container_contains(context->container, val & 0xFFFF,+                              context->typecode);+}++roaring_bitmap_t *roaring_bitmap_of_ptr(size_t n_args, const uint32_t *vals) {+    roaring_bitmap_t *answer = roaring_bitmap_create();+    roaring_bitmap_add_many(answer, n_args, vals);+    return answer;+}++roaring_bitmap_t *roaring_bitmap_of(size_t n_args, ...) {+    // todo: could be greatly optimized but we do not expect this call to ever+    // include long lists+    roaring_bitmap_t *answer = roaring_bitmap_create();+    roaring_bulk_context_t context = CROARING_ZERO_INITIALIZER;+    va_list ap;+    va_start(ap, n_args);+    for (size_t i = 0; i < n_args; i++) {+        uint32_t val = va_arg(ap, uint32_t);+        roaring_bitmap_add_bulk(answer, &context, val);+    }+    va_end(ap);+    return answer;+}++static inline uint64_t minimum_uint64(uint64_t a, uint64_t b) {+    return (a < b) ? a : b;+}++roaring_bitmap_t *roaring_bitmap_from_range(uint64_t min, uint64_t max,+                                            uint32_t step) {+    if (max >= UINT64_C(0x100000000)) {+        max = UINT64_C(0x100000000);+    }+    if (step == 0) return NULL;+    if (max <= min) return NULL;+    roaring_bitmap_t *answer = roaring_bitmap_create();+    if (step >= (1 << 16)) {+        // 64-bit counter: max may be 2^32, which a uint32_t never reaches+        for (uint64_t value = min; value < max; value += step) {+            roaring_bitmap_add(answer, (uint32_t)value);+        }+        return answer;+    }+    uint64_t min_tmp = min;+    do {+        uint32_t key = (uint32_t)min_tmp >> 16;+        uint32_t container_min = min_tmp & 0xFFFF;+        uint32_t container_max =+            (uint32_t)minimum_uint64(max - (key << 16), 1 << 16);+        uint8_t type;+        container_t *container = container_from_range(+            &type, container_min, container_max, (uint16_t)step);+        ra_append(&answer->high_low_container, (uint16_t)key, container, type);+        uint32_t gap = container_max - container_min + step - 1;+        min_tmp += gap - (gap % step);+    } while (min_tmp < max);+    // cardinality of bitmap will be ((uint64_t) max - min + step - 1 ) / step+    return answer;+}++void roaring_bitmap_add_range_closed(roaring_bitmap_t *r, uint32_t min,+                                     uint32_t max) {+    if (min > max) {+        return;+    }++    roaring_array_t *ra = &r->high_low_container;++    uint32_t min_key = min >> 16;+    uint32_t max_key = max >> 16;++    int32_t num_required_containers = max_key - min_key + 1;+    int32_t suffix_length =+        count_greater(ra->keys, ra->size, (uint16_t)max_key);+    int32_t prefix_length =+        count_less(ra->keys, ra->size - suffix_length, (uint16_t)min_key);+    int32_t common_length = ra->size - prefix_length - suffix_length;++    if (num_required_containers > common_length) {+        ra_shift_tail(ra, suffix_length,+                      num_required_containers - common_length);+    }++    int32_t src = prefix_length + common_length - 1;+    int32_t dst = ra->size - suffix_length - 1;+    for (uint32_t key = max_key; key != min_key - 1;+         key--) {  // beware of min_key==0+        uint32_t container_min = (min_key == key) ? (min & 0xffff) : 0;+        uint32_t container_max = (max_key == key) ? (max & 0xffff) : 0xffff;+        container_t *new_container;+        uint8_t new_type;++        if (src >= 0 && ra->keys[src] == key) {+            ra_unshare_container_at_index(ra, (uint16_t)src);+            new_container =+                container_add_range(ra->containers[src], ra->typecodes[src],+                                    container_min, container_max, &new_type);+            if (new_container != ra->containers[src]) {+                container_free(ra->containers[src], ra->typecodes[src]);+            }+            src--;+        } else {+            new_container = container_from_range(&new_type, container_min,+                                                 container_max + 1, 1);+        }+        ra_replace_key_and_container_at_index(ra, dst, (uint16_t)key,+                                              new_container, new_type);+        dst--;+    }+}++void roaring_bitmap_remove_range_closed(roaring_bitmap_t *r, uint32_t min,+                                        uint32_t max) {+    if (min > max) {+        return;+    }++    roaring_array_t *ra = &r->high_low_container;++    uint32_t min_key = min >> 16;+    uint32_t max_key = max >> 16;++    int32_t src = count_less(ra->keys, ra->size, (uint16_t)min_key);+    int32_t dst = src;+    while (src < ra->size && ra->keys[src] <= max_key) {+        uint32_t container_min =+            (min_key == ra->keys[src]) ? (min & 0xffff) : 0;+        uint32_t container_max =+            (max_key == ra->keys[src]) ? (max & 0xffff) : 0xffff;+        ra_unshare_container_at_index(ra, (uint16_t)src);+        container_t *new_container;+        uint8_t new_type;+        new_container =+            container_remove_range(ra->containers[src], ra->typecodes[src],+                                   container_min, container_max, &new_type);+        if (new_container != ra->containers[src]) {+            container_free(ra->containers[src], ra->typecodes[src]);+        }+        if (new_container) {+            ra_replace_key_and_container_at_index(ra, dst, ra->keys[src],+                                                  new_container, new_type);+            dst++;+        }+        src++;+    }+    if (src > dst) {+        ra_shift_tail(ra, ra->size - src, dst - src);+    }+}++void roaring_bitmap_printf(const roaring_bitmap_t *r) {+    const roaring_array_t *ra = &r->high_low_container;++    printf("{");+    for (int i = 0; i < ra->size; ++i) {+        container_printf_as_uint32_array(ra->containers[i], ra->typecodes[i],+                                         ((uint32_t)ra->keys[i]) << 16);++        if (i + 1 < ra->size) {+            printf(",");+        }+    }+    printf("}");+}++void roaring_bitmap_printf_describe(const roaring_bitmap_t *r) {+    const roaring_array_t *ra = &r->high_low_container;++    printf("{");+    for (int i = 0; i < ra->size; ++i) {+        printf("%d: %s (%d)", ra->keys[i],+               get_full_container_name(ra->containers[i], ra->typecodes[i]),+               container_get_cardinality(ra->containers[i], ra->typecodes[i]));+        if (ra->typecodes[i] == SHARED_CONTAINER_TYPE) {+            printf("(shared count = %" PRIu32 " )",+                   croaring_refcount_get(+                       &(CAST_shared(ra->containers[i])->counter)));+        }++        if (i + 1 < ra->size) {+            printf(", ");+        }+    }+    printf("}");+}++/**+ *  (For advanced users.)+ * Collect statistics about the bitmap+ */+void roaring_bitmap_statistics(const roaring_bitmap_t *r,+                               roaring_statistics_t *stat) {+    const roaring_array_t *ra = &r->high_low_container;++    memset(stat, 0, sizeof(*stat));+    stat->n_containers = ra->size;+    stat->min_value = roaring_bitmap_minimum(r);+    stat->max_value = roaring_bitmap_maximum(r);++    for (int i = 0; i < ra->size; ++i) {+        uint8_t truetype =+            get_container_type(ra->containers[i], ra->typecodes[i]);+        uint32_t card =+            container_get_cardinality(ra->containers[i], ra->typecodes[i]);+        uint32_t sbytes =+            container_size_in_bytes(ra->containers[i], ra->typecodes[i]);+        stat->cardinality += card;+        switch (truetype) {+            case BITSET_CONTAINER_TYPE:+                stat->n_bitset_containers++;+                stat->n_values_bitset_containers += card;+                stat->n_bytes_bitset_containers += sbytes;+                break;+            case ARRAY_CONTAINER_TYPE:+                stat->n_array_containers++;+                stat->n_values_array_containers += card;+                stat->n_bytes_array_containers += sbytes;+                break;+            case RUN_CONTAINER_TYPE:+                stat->n_run_containers++;+                stat->n_values_run_containers += card;+                stat->n_bytes_run_containers += sbytes;+                break;+            default:+                assert(false);+                roaring_unreachable;+        }+    }+}++bool roaring_contains_shared(const roaring_bitmap_t *r) {+    const roaring_array_t *ra = &r->high_low_container;+    for (int i = 0; i < ra->size; ++i) {+        if (ra->typecodes[i] == SHARED_CONTAINER_TYPE) {+            return true;+        }+    }+    return false;+}++bool roaring_unshare_all(roaring_bitmap_t *r) {+    const roaring_array_t *ra = &r->high_low_container;+    bool unshared = false;+    for (int i = 0; i < ra->size; ++i) {+        uint8_t typecode = ra->typecodes[i];+        if (typecode == SHARED_CONTAINER_TYPE) {+            ra->containers[i] = get_writable_copy_if_shared(ra->containers[i],+                                                            &ra->typecodes[i]);+            unshared = true;+        }+    }+    return unshared;+}++/*+ * Checks that:+ * - Array containers are sorted and contain no duplicates+ * - Range containers are sorted and contain no overlapping ranges+ * - Roaring containers are sorted by key and there are no duplicate keys+ * - The correct container type is use for each container (e.g. bitmaps aren't+ * used for small containers)+ * - Shared containers are only used when the bitmap is COW+ */+bool roaring_bitmap_internal_validate(const roaring_bitmap_t *r,+                                      const char **reason) {+    const char *reason_local;+    if (reason == NULL) {+        // Always allow assigning through *reason+        reason = &reason_local;+    }+    *reason = NULL;+    const roaring_array_t *ra = &r->high_low_container;+    if (ra->size < 0) {+        *reason = "negative size";+        return false;+    }+    if (ra->allocation_size < 0) {+        *reason = "negative allocation size";+        return false;+    }+    if (ra->size > ra->allocation_size) {+        *reason = "more containers than allocated space";+        return false;+    }+    if (ra->flags & ~(ROARING_FLAG_COW | ROARING_FLAG_FROZEN)) {+        *reason = "invalid flags";+        return false;+    }+    if (ra->size == 0) {+        return true;+    }++    if (ra->keys == NULL) {+        *reason = "keys is NULL";+        return false;+    }+    if (ra->typecodes == NULL) {+        *reason = "typecodes is NULL";+        return false;+    }+    if (ra->containers == NULL) {+        *reason = "containers is NULL";+        return false;+    }++    uint32_t prev_key = ra->keys[0];+    for (int32_t i = 1; i < ra->size; ++i) {+        if (ra->keys[i] <= prev_key) {+            *reason = "keys not strictly increasing";+            return false;+        }+        prev_key = ra->keys[i];+    }++    bool cow = roaring_bitmap_get_copy_on_write(r);++    for (int32_t i = 0; i < ra->size; ++i) {+        if (ra->typecodes[i] == SHARED_CONTAINER_TYPE && !cow) {+            *reason = "shared container in non-COW bitmap";+            return false;+        }+        if (!container_internal_validate(ra->containers[i], ra->typecodes[i],+                                         reason)) {+            // reason should already be set+            if (*reason == NULL) {+                *reason = "container failed to validate but no reason given";+            }+            return false;+        }+    }++    return true;+}++roaring_bitmap_t *roaring_bitmap_copy(const roaring_bitmap_t *r) {+    roaring_bitmap_t *ans =+        (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t));+    if (!ans) {+        return NULL;+    }+    if (!ra_init_with_capacity(  // allocation of list of containers can fail+            &ans->high_low_container, r->high_low_container.size)) {+        roaring_free(ans);+        return NULL;+    }+    if (!ra_overwrite(  // memory allocation of individual containers may fail+            &r->high_low_container, &ans->high_low_container, is_cow(r))) {+        roaring_bitmap_free(ans);  // overwrite should leave in freeable state+        return NULL;+    }+    roaring_bitmap_set_copy_on_write(ans, is_cow(r));+    return ans;+}++bool roaring_bitmap_overwrite(roaring_bitmap_t *dest,+                              const roaring_bitmap_t *src) {+    // ra_overwrite() frees dest's containers, shared or not. Setting the flag+    // with roaring_bitmap_set_copy_on_write() would first copy the shared ones.+    if (is_cow(src)) {+        dest->high_low_container.flags |= ROARING_FLAG_COW;+    } else {+        dest->high_low_container.flags &= ~ROARING_FLAG_COW;+    }+    return ra_overwrite(&src->high_low_container, &dest->high_low_container,+                        is_cow(src));+}++void roaring_bitmap_free(const roaring_bitmap_t *r) {+    if (r == NULL) {+        return;+    }+    if (!is_frozen(r)) {+        ra_clear((roaring_array_t *)&r->high_low_container);+    }+    roaring_free((roaring_bitmap_t *)r);+}++void roaring_bitmap_clear(roaring_bitmap_t *r) {+    ra_reset(&r->high_low_container);+}++void roaring_bitmap_add(roaring_bitmap_t *r, uint32_t val) {+    roaring_array_t *ra = &r->high_low_container;++    const uint16_t hb = val >> 16;+    const int i = ra_get_index(ra, hb);+    uint8_t typecode;+    if (i >= 0) {+        ra_unshare_container_at_index(ra, (uint16_t)i);+        container_t *container =+            ra_get_container_at_index(ra, (uint16_t)i, &typecode);+        uint8_t newtypecode = typecode;+        container_t *container2 =+            container_add(container, val & 0xFFFF, typecode, &newtypecode);+        if (container2 != container) {+            container_free(container, typecode);+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+        }+    } else {+        array_container_t *newac = array_container_create();+        container_t *container =+            container_add(newac, val & 0xFFFF, ARRAY_CONTAINER_TYPE, &typecode);+        // we could just assume that it stays an array container+        ra_insert_new_key_value_at(&r->high_low_container, -i - 1, hb,+                                   container, typecode);+    }+}++bool roaring_bitmap_add_checked(roaring_bitmap_t *r, uint32_t val) {+    const uint16_t hb = val >> 16;+    const int i = ra_get_index(&r->high_low_container, hb);+    uint8_t typecode;+    bool result = false;+    if (i >= 0) {+        ra_unshare_container_at_index(&r->high_low_container, (uint16_t)i);+        container_t *container = ra_get_container_at_index(+            &r->high_low_container, (uint16_t)i, &typecode);++        const int oldCardinality =+            container_get_cardinality(container, typecode);++        uint8_t newtypecode = typecode;+        container_t *container2 =+            container_add(container, val & 0xFFFF, typecode, &newtypecode);+        if (container2 != container) {+            container_free(container, typecode);+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+            result = true;+        } else {+            const int newCardinality =+                container_get_cardinality(container, newtypecode);++            result = oldCardinality != newCardinality;+        }+    } else {+        array_container_t *newac = array_container_create();+        container_t *container =+            container_add(newac, val & 0xFFFF, ARRAY_CONTAINER_TYPE, &typecode);+        // we could just assume that it stays an array container+        ra_insert_new_key_value_at(&r->high_low_container, -i - 1, hb,+                                   container, typecode);+        result = true;+    }++    return result;+}++void roaring_bitmap_remove(roaring_bitmap_t *r, uint32_t val) {+    const uint16_t hb = val >> 16;+    const int i = ra_get_index(&r->high_low_container, hb);+    uint8_t typecode;+    if (i >= 0) {+        ra_unshare_container_at_index(&r->high_low_container, (uint16_t)i);+        container_t *container = ra_get_container_at_index(+            &r->high_low_container, (uint16_t)i, &typecode);+        uint8_t newtypecode = typecode;+        container_t *container2 =+            container_remove(container, val & 0xFFFF, typecode, &newtypecode);+        if (container2 != container) {+            container_free(container, typecode);+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+        }+        if (container_nonzero_cardinality(container2, newtypecode)) {+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+        } else {+            ra_remove_at_index_and_free(&r->high_low_container, i);+        }+    }+}++bool roaring_bitmap_remove_checked(roaring_bitmap_t *r, uint32_t val) {+    const uint16_t hb = val >> 16;+    const int i = ra_get_index(&r->high_low_container, hb);+    uint8_t typecode;+    bool result = false;+    if (i >= 0) {+        ra_unshare_container_at_index(&r->high_low_container, (uint16_t)i);+        container_t *container = ra_get_container_at_index(+            &r->high_low_container, (uint16_t)i, &typecode);++        const int oldCardinality =+            container_get_cardinality(container, typecode);++        uint8_t newtypecode = typecode;+        container_t *container2 =+            container_remove(container, val & 0xFFFF, typecode, &newtypecode);+        if (container2 != container) {+            container_free(container, typecode);+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+        }++        const int newCardinality =+            container_get_cardinality(container2, newtypecode);++        if (newCardinality != 0) {+            ra_set_container_at_index(&r->high_low_container, i, container2,+                                      newtypecode);+        } else {+            ra_remove_at_index_and_free(&r->high_low_container, i);+        }++        result = oldCardinality != newCardinality;+    }+    return result;+}++void roaring_bitmap_remove_many(roaring_bitmap_t *r, size_t n_args,+                                const uint32_t *vals) {+    if (n_args == 0 || r->high_low_container.size == 0) {+        return;+    }+    int32_t pos =+        -1;  // position of the container used in the previous iteration+    for (size_t i = 0; i < n_args; i++) {+        uint16_t key = (uint16_t)(vals[i] >> 16);+        if (pos < 0 || key != r->high_low_container.keys[pos]) {+            pos = ra_get_index(&r->high_low_container, key);+            if (pos >= 0) {+                ra_unshare_container_at_index(&r->high_low_container,+                                              (uint16_t)pos);+            }+        }+        if (pos >= 0) {+            uint8_t new_typecode;+            container_t *new_container;+            new_container = container_remove(+                r->high_low_container.containers[pos], vals[i] & 0xffff,+                r->high_low_container.typecodes[pos], &new_typecode);+            if (new_container != r->high_low_container.containers[pos]) {+                container_free(r->high_low_container.containers[pos],+                               r->high_low_container.typecodes[pos]);+                ra_replace_key_and_container_at_index(&r->high_low_container,+                                                      pos, key, new_container,+                                                      new_typecode);+            }+            if (!container_nonzero_cardinality(new_container, new_typecode)) {+                container_free(new_container, new_typecode);+                ra_remove_at_index(&r->high_low_container, pos);+                pos = -1;+            }+        }+    }+}++// there should be some SIMD optimizations possible here+roaring_bitmap_t *roaring_bitmap_and(const roaring_bitmap_t *x1,+                                     const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    uint32_t neededcap = length1 > length2 ? length2 : length1;+    roaring_bitmap_t *answer = roaring_bitmap_create_with_capacity(neededcap);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));++    int pos1 = 0, pos2 = 0;++    while (pos1 < length1 && pos2 < length2) {+        const uint16_t s1 =+            ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+        const uint16_t s2 =+            ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        if (s1 == s2) {+            uint8_t type1, type2;+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c = container_and(c1, type1, c2, type2, &result_type);++            if (container_nonzero_cardinality(c, result_type)) {+                ra_append(&answer->high_low_container, s1, c, result_type);+            } else {+                container_free(c, result_type);  // otherwise: memory leak!+            }+            ++pos1;+            ++pos2;+        } else if (s1 < s2) {  // s1 < s2+            pos1 = ra_advance_until(&x1->high_low_container, s2, pos1);+        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+        }+    }+    return answer;+}++/**+ * Compute the union of 'number' bitmaps.+ */+roaring_bitmap_t *roaring_bitmap_or_many(size_t number,+                                         const roaring_bitmap_t **x) {+    if (number == 0) {+        return roaring_bitmap_create();+    }+    if (number == 1) {+        return roaring_bitmap_copy(x[0]);+    }+    roaring_bitmap_t *answer =+        roaring_bitmap_lazy_or(x[0], x[1], CROARING_LAZY_OR_BITSET_CONVERSION);+    for (size_t i = 2; i < number; i++) {+        roaring_bitmap_lazy_or_inplace(answer, x[i],+                                       CROARING_LAZY_OR_BITSET_CONVERSION);+    }+    roaring_bitmap_repair_after_lazy(answer);+    return answer;+}++/**+ * Compute the xor of 'number' bitmaps.+ */+roaring_bitmap_t *roaring_bitmap_xor_many(size_t number,+                                          const roaring_bitmap_t **x) {+    if (number == 0) {+        return roaring_bitmap_create();+    }+    if (number == 1) {+        return roaring_bitmap_copy(x[0]);+    }+    roaring_bitmap_t *answer = roaring_bitmap_lazy_xor(x[0], x[1]);+    for (size_t i = 2; i < number; i++) {+        roaring_bitmap_lazy_xor_inplace(answer, x[i]);+    }+    roaring_bitmap_repair_after_lazy(answer);+    return answer;+}++// inplace and (modifies its first argument).+void roaring_bitmap_and_inplace(roaring_bitmap_t *x1,+                                const roaring_bitmap_t *x2) {+    if (x1 == x2) return;+    int pos1 = 0, pos2 = 0, intersection_size = 0;+    const int length1 = ra_get_size(&x1->high_low_container);+    const int length2 = ra_get_size(&x2->high_low_container);++    // any skipped-over or newly emptied containers in x1+    // have to be freed.+    while (pos1 < length1 && pos2 < length2) {+        const uint16_t s1 =+            ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+        const uint16_t s2 =+            ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        if (s1 == s2) {+            uint8_t type1, type2, result_type;+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);++            // We do the computation "in place" only when c1 is not a shared+            // container. Rationale: using a shared container safely with in+            // place computation would require making a copy and then doing the+            // computation in place which is likely less efficient than avoiding+            // in place entirely and always generating a new container.+            container_t *c =+                (type1 == SHARED_CONTAINER_TYPE)+                    ? container_and(c1, type1, c2, type2, &result_type)+                    : container_iand(c1, type1, c2, type2, &result_type);++            if (c != c1) {  // in this instance a new container was created, and+                            // we need to free the old one+                container_free(c1, type1);+            }+            if (container_nonzero_cardinality(c, result_type)) {+                ra_replace_key_and_container_at_index(&x1->high_low_container,+                                                      intersection_size, s1, c,+                                                      result_type);+                intersection_size++;+            } else {+                container_free(c, result_type);+            }+            ++pos1;+            ++pos2;+        } else if (s1 < s2) {+            pos1 = ra_advance_until_freeing(&x1->high_low_container, s2, pos1);+        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+        }+    }++    // if we ended early because x2 ran out, then all remaining in x1 should be+    // freed+    while (pos1 < length1) {+        container_free(x1->high_low_container.containers[pos1],+                       x1->high_low_container.typecodes[pos1]);+        ++pos1;+    }++    // all containers after this have either been copied or freed+    ra_downsize(&x1->high_low_container, intersection_size);+}++roaring_bitmap_t *roaring_bitmap_or(const roaring_bitmap_t *x1,+                                    const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    if (0 == length1) {+        return roaring_bitmap_copy(x2);+    }+    if (0 == length2) {+        return roaring_bitmap_copy(x1);+    }+    roaring_bitmap_t *answer =+        roaring_bitmap_create_with_capacity(length1 + length2);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c = container_or(c1, type1, c2, type2, &result_type);++            // since we assume that the initial containers are non-empty, the+            // result here+            // can only be non-empty+            ra_append(&answer->high_low_container, s1, c, result_type);+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            // c1 = container_clone(c1, type1);+            c1 = get_copy_of_container(c1, &type1, is_cow(x1));+            if (is_cow(x1)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c1,+                                          type1);+            }+            ra_append(&answer->high_low_container, s1, c1, type1);+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            // c2 = container_clone(c2, type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_append(&answer->high_low_container, s2, c2, type2);+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&answer->high_low_container,+                             &x2->high_low_container, pos2, length2,+                             is_cow(x2));+    } else if (pos2 == length2) {+        ra_append_copy_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1,+                             is_cow(x1));+    }+    return answer;+}++static void roaring_inplace_merge_bulk(roaring_bitmap_t *x1,+                                       const roaring_bitmap_t *x2, int dst,+                                       int left, int right, bool is_xor) {+    roaring_array_t *ra1 = &x1->high_low_container;+    const roaring_array_t *ra2 = &x2->high_low_container;+    const bool cow2 = is_cow(x2);+    const int length1 = ra1->size;+    const int length2 = ra2->size;++    int distinct = 0;+    {+        int l = left, r = right;+        while (l < length1 && r < length2) {+            uint16_t k1 = ra1->keys[l];+            uint16_t k2 = ra2->keys[r];+            if (k1 < k2) {+                l++;+            } else if (k1 > k2) {+                r++;+            } else {+                l++;+                r++;+            }+            distinct++;+        }+        distinct += (length1 - l) + (length2 - r);+    }+    const int total = dst + distinct;++    roaring_array_t merged;+    ra_init_with_capacity(&merged, total > 0 ? (uint32_t)total : 1);++    for (int i = 0; i < dst; i++) {+        ra_append(&merged, ra1->keys[i], ra1->containers[i], ra1->typecodes[i]);+    }++    uint8_t result_type = 0;+    while (left < length1 && right < length2) {+        uint16_t k1 = ra1->keys[left];+        uint16_t k2 = ra2->keys[right];+        if (k1 < k2) {+            ra_append(&merged, k1, ra1->containers[left], ra1->typecodes[left]);+            left++;+        } else if (k1 > k2) {+            uint8_t type2 = ra2->typecodes[right];+            container_t *c2 =+                get_copy_of_container(ra2->containers[right], &type2, cow2);+            if (cow2) {+                ra_set_container_at_index(ra2, right, c2, type2);+            }+            ra_append(&merged, k2, c2, type2);+            right++;+        } else {+            uint8_t type1 = ra1->typecodes[left];+            container_t *c1 = ra1->containers[left];+            uint8_t type2 = ra2->typecodes[right];+            container_t *c2 = ra2->containers[right];+            if (is_xor) {+                container_t *c;+                if (type1 == SHARED_CONTAINER_TYPE) {+                    c = container_xor(c1, type1, c2, type2, &result_type);+                    shared_container_free(CAST_shared(c1));+                } else {+                    c = container_ixor(c1, type1, c2, type2, &result_type);+                }+                if (container_nonzero_cardinality(c, result_type)) {+                    ra_append(&merged, k1, c, result_type);+                } else {+                    container_free(c, result_type);+                }+            } else {+                if (container_is_full(c1, type1)) {+                    ra_append(&merged, k1, c1, type1);+                } else {+                    container_t *c =+                        (type1 == SHARED_CONTAINER_TYPE)+                            ? container_or(c1, type1, c2, type2, &result_type)+                            : container_ior(c1, type1, c2, type2, &result_type);+                    if (c != c1) {+                        container_free(c1, type1);+                    }+                    ra_append(&merged, k1, c, result_type);+                }+            }+            left++;+            right++;+        }+    }+    for (; left < length1; left++) {+        ra_append(&merged, ra1->keys[left], ra1->containers[left],+                  ra1->typecodes[left]);+    }+    for (; right < length2; right++) {+        uint8_t type2 = ra2->typecodes[right];+        container_t *c2 =+            get_copy_of_container(ra2->containers[right], &type2, cow2);+        if (cow2) {+            ra_set_container_at_index(ra2, right, c2, type2);+        }+        ra_append(&merged, ra2->keys[right], c2, type2);+    }++    merged.flags = ra1->flags;+    ra_clear_without_containers(ra1);+    *ra1 = merged;+}++// Union of x2 into x1 when x2 has few containers compared to x1. We find each+// key of x2 by galloping, and we insert the new containers by moving the+// containers of x1 once, block by block, from the back.+static void roaring_inplace_union_few(roaring_bitmap_t *x1,+                                      const roaring_bitmap_t *x2) {+    roaring_array_t *ra1 = &x1->high_low_container;+    const roaring_array_t *ra2 = &x2->high_low_container;+    const bool cow2 = is_cow(x2);+    const int32_t length1 = ra1->size;+    const int32_t length2 = ra2->size;+    uint8_t result_type = 0;++    // Union of the containers with a key in both, counting the other ones.+    int32_t added = 0;+    int32_t pos1 = -1;+    for (int32_t pos2 = 0; pos2 < length2; pos2++) {+        const uint16_t key = ra2->keys[pos2];+        pos1 = ra_advance_until(ra1, key, pos1);+        if (pos1 == length1 || ra1->keys[pos1] != key) {+            added++;+            pos1--;  // The next key may be at the same position.+            continue;+        }+        uint8_t type1 = ra1->typecodes[pos1];+        container_t *c1 = ra1->containers[pos1];+        if (container_is_full(c1, type1)) continue;+        container_t *c2 = ra2->containers[pos2];+        uint8_t type2 = ra2->typecodes[pos2];+        container_t *c =+            (type1 == SHARED_CONTAINER_TYPE)+                ? container_or(c1, type1, c2, type2, &result_type)+                : container_ior(c1, type1, c2, type2, &result_type);+        if (c != c1) {+            container_free(c1, type1);+        }+        ra_set_container_at_index(ra1, pos1, c, result_type);+    }+    if (added == 0) return;++    // Insertion of the other containers, from the back: the containers of x1+    // from index 'end' on are already in place.+    extend_array(ra1, added);+    int32_t end = length1;+    int32_t dst = length1 + added;+    for (int32_t pos2 = length2 - 1; pos2 >= 0 && dst > end; pos2--) {+        const uint16_t key = ra2->keys[pos2];+        int32_t i = binarySearch(ra1->keys, end, key);+        if (i >= 0) continue;+        i = -i - 1;+        const int32_t n = end - i;+        dst -= n;+        memmove(ra1->keys + dst, ra1->keys + i, n * sizeof(uint16_t));+        memmove(ra1->containers + dst, ra1->containers + i,+                n * sizeof(container_t *));+        memmove(ra1->typecodes + dst, ra1->typecodes + i, n * sizeof(uint8_t));+        dst--;+        uint8_t type2 = ra2->typecodes[pos2];+        container_t *c2 =+            get_copy_of_container(ra2->containers[pos2], &type2, cow2);+        if (cow2) {+            ra_set_container_at_index(ra2, pos2, c2, type2);+        }+        ra1->keys[dst] = key;+        ra1->containers[dst] = c2;+        ra1->typecodes[dst] = type2;+        end = i;+    }+    ra1->size = length1 + added;+}++// inplace or (modifies its first argument).+void roaring_bitmap_or_inplace(roaring_bitmap_t *x1,+                               const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    int length1 = x1->high_low_container.size;+    const int length2 = x2->high_low_container.size;++    if (0 == length2) return;++    if (0 == length1) {+        roaring_bitmap_overwrite(x1, x2);+        return;+    }+    // Merging would visit all the containers of x1.+    if ((int64_t)length2 * 32 < length1) {+        roaring_inplace_union_few(x1, x2);+        return;+    }+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            if (!container_is_full(c1, type1)) {+                container_t *c2 = ra_get_container_at_index(+                    &x2->high_low_container, (uint16_t)pos2, &type2);+                container_t *c =+                    (type1 == SHARED_CONTAINER_TYPE)+                        ? container_or(c1, type1, c2, type2, &result_type)+                        : container_ior(c1, type1, c2, type2, &result_type);++                if (c != c1) {  // in this instance a new container was created,+                                // and we need to free the old one+                    container_free(c1, type1);+                }+                ra_set_container_at_index(&x1->high_low_container, pos1, c,+                                          result_type);+            }+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            roaring_inplace_merge_bulk(x1, x2, pos1, pos1, pos2, false);+            return;+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&x1->high_low_container, &x2->high_low_container,+                             pos2, length2, is_cow(x2));+    }+}++roaring_bitmap_t *roaring_bitmap_xor(const roaring_bitmap_t *x1,+                                     const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    if (0 == length1) {+        return roaring_bitmap_copy(x2);+    }+    if (0 == length2) {+        return roaring_bitmap_copy(x1);+    }+    roaring_bitmap_t *answer =+        roaring_bitmap_create_with_capacity(length1 + length2);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c = container_xor(c1, type1, c2, type2, &result_type);++            if (container_nonzero_cardinality(c, result_type)) {+                ra_append(&answer->high_low_container, s1, c, result_type);+            } else {+                container_free(c, result_type);+            }+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            c1 = get_copy_of_container(c1, &type1, is_cow(x1));+            if (is_cow(x1)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c1,+                                          type1);+            }+            ra_append(&answer->high_low_container, s1, c1, type1);+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_append(&answer->high_low_container, s2, c2, type2);+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&answer->high_low_container,+                             &x2->high_low_container, pos2, length2,+                             is_cow(x2));+    } else if (pos2 == length2) {+        ra_append_copy_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1,+                             is_cow(x1));+    }+    return answer;+}++// inplace xor (modifies its first argument).++void roaring_bitmap_xor_inplace(roaring_bitmap_t *x1,+                                const roaring_bitmap_t *x2) {+    assert(x1 != x2);+    uint8_t result_type = 0;+    int length1 = x1->high_low_container.size;+    const int length2 = x2->high_low_container.size;++    if (0 == length2) return;++    if (0 == length1) {+        roaring_bitmap_overwrite(x1, x2);+        return;+    }++    // XOR can have new containers inserted from x2, but can also+    // lose containers when x1 and x2 are nonempty and identical.++    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);++            // We do the computation "in place" only when c1 is not a shared+            // container. Rationale: using a shared container safely with in+            // place computation would require making a copy and then doing the+            // computation in place which is likely less efficient than avoiding+            // in place entirely and always generating a new container.++            container_t *c;+            if (type1 == SHARED_CONTAINER_TYPE) {+                c = container_xor(c1, type1, c2, type2, &result_type);+                shared_container_free(CAST_shared(c1));  // so release+            } else {+                c = container_ixor(c1, type1, c2, type2, &result_type);+            }++            if (container_nonzero_cardinality(c, result_type)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c,+                                          result_type);+                ++pos1;+            } else {+                container_free(c, result_type);+                roaring_inplace_merge_bulk(x1, x2, pos1, pos1 + 1, pos2 + 1,+                                           true);+                return;+            }++            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            roaring_inplace_merge_bulk(x1, x2, pos1, pos1, pos2, true);+            return;+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&x1->high_low_container, &x2->high_low_container,+                             pos2, length2, is_cow(x2));+    }+}++roaring_bitmap_t *roaring_bitmap_andnot(const roaring_bitmap_t *x1,+                                        const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    if (0 == length1) {+        roaring_bitmap_t *empty_bitmap = roaring_bitmap_create();+        roaring_bitmap_set_copy_on_write(empty_bitmap,+                                         is_cow(x1) || is_cow(x2));+        return empty_bitmap;+    }+    if (0 == length2) {+        return roaring_bitmap_copy(x1);+    }+    roaring_bitmap_t *answer = roaring_bitmap_create_with_capacity(length1);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));++    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = 0;+    uint16_t s2 = 0;+    while (true) {+        s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+        s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c =+                container_andnot(c1, type1, c2, type2, &result_type);++            if (container_nonzero_cardinality(c, result_type)) {+                ra_append(&answer->high_low_container, s1, c, result_type);+            } else {+                container_free(c, result_type);+            }+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+        } else if (s1 < s2) {  // s1 < s2+            const int next_pos1 =+                ra_advance_until(&x1->high_low_container, s2, pos1);+            ra_append_copy_range(&answer->high_low_container,+                                 &x1->high_low_container, pos1, next_pos1,+                                 is_cow(x1));+            // TODO : perhaps some of the copy_on_write should be based on+            // answer rather than x1 (more stringent?).  Many similar cases+            pos1 = next_pos1;+            if (pos1 == length1) break;+        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+            if (pos2 == length2) break;+        }+    }+    if (pos2 == length2) {+        ra_append_copy_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1,+                             is_cow(x1));+    }+    return answer;+}++// inplace andnot (modifies its first argument).++void roaring_bitmap_andnot_inplace(roaring_bitmap_t *x1,+                                   const roaring_bitmap_t *x2) {+    assert(x1 != x2);++    uint8_t result_type = 0;+    int length1 = x1->high_low_container.size;+    const int length2 = x2->high_low_container.size;+    int intersection_size = 0;++    if (0 == length2) return;++    if (0 == length1) {+        roaring_bitmap_clear(x1);+        return;+    }++    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);++            // We do the computation "in place" only when c1 is not a shared+            // container. Rationale: using a shared container safely with in+            // place computation would require making a copy and then doing the+            // computation in place which is likely less efficient than avoiding+            // in place entirely and always generating a new container.++            container_t *c;+            if (type1 == SHARED_CONTAINER_TYPE) {+                c = container_andnot(c1, type1, c2, type2, &result_type);+                shared_container_free(CAST_shared(c1));  // release+            } else {+                c = container_iandnot(c1, type1, c2, type2, &result_type);+            }++            if (container_nonzero_cardinality(c, result_type)) {+                ra_replace_key_and_container_at_index(&x1->high_low_container,+                                                      intersection_size++, s1,+                                                      c, result_type);+            } else {+                container_free(c, result_type);+            }++            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            if (pos1 != intersection_size) {+                container_t *c1 = ra_get_container_at_index(+                    &x1->high_low_container, (uint16_t)pos1, &type1);++                ra_replace_key_and_container_at_index(+                    &x1->high_low_container, intersection_size, s1, c1, type1);+            }+            intersection_size++;+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }++    if (pos1 < length1) {+        // all containers between intersection_size and+        // pos1 are junk.  However, they have either been moved+        // (thus still referenced) or involved in an iandnot+        // that will clean up all containers that could not be reused.+        // Thus we should not free the junk containers between+        // intersection_size and pos1.+        if (pos1 > intersection_size) {+            // left slide of remaining items+            ra_copy_range(&x1->high_low_container, pos1, length1,+                          intersection_size);+        }+        // else current placement is fine+        intersection_size += (length1 - pos1);+    }+    ra_downsize(&x1->high_low_container, intersection_size);+}++uint64_t roaring_bitmap_get_cardinality(const roaring_bitmap_t *r) {+    const roaring_array_t *ra = &r->high_low_container;++    uint64_t card = 0;+    for (int i = 0; i < ra->size; ++i)+        card += container_get_cardinality(ra->containers[i], ra->typecodes[i]);+    return card;+}++uint64_t roaring_bitmap_range_cardinality(const roaring_bitmap_t *r,+                                          uint64_t range_start,+                                          uint64_t range_end) {+    if (range_end > (uint64_t)UINT32_MAX + 1) {+        range_end = (uint64_t)UINT32_MAX + 1;+    }+    if (range_start >= range_end) {+        return 0;+    }+    return roaring_bitmap_range_cardinality_closed(r, (uint32_t)range_start,+                                                   (uint32_t)(range_end - 1));+}++uint64_t roaring_bitmap_range_cardinality_closed(const roaring_bitmap_t *r,+                                                 uint32_t range_start,+                                                 uint32_t range_end) {+    const roaring_array_t *ra = &r->high_low_container;++    if (range_start > range_end) {+        return 0;+    }++    // now we have: 0 <= range_start <= range_end <= UINT32_MAX++    uint16_t minhb = (uint16_t)(range_start >> 16);+    uint16_t maxhb = (uint16_t)(range_end >> 16);++    uint64_t card = 0;++    int i = ra_get_index(ra, minhb);+    if (i >= 0) {+        if (minhb == maxhb) {+            card += container_rank(ra->containers[i], ra->typecodes[i],+                                   range_end & 0xffff);+        } else {+            card +=+                container_get_cardinality(ra->containers[i], ra->typecodes[i]);+        }+        if ((range_start & 0xffff) != 0) {+            card -= container_rank(ra->containers[i], ra->typecodes[i],+                                   (range_start & 0xffff) - 1);+        }+        i++;+    } else {+        i = -i - 1;+    }++    for (; i < ra->size; i++) {+        uint16_t key = ra->keys[i];+        if (key < maxhb) {+            card +=+                container_get_cardinality(ra->containers[i], ra->typecodes[i]);+        } else if (key == maxhb) {+            card += container_rank(ra->containers[i], ra->typecodes[i],+                                   range_end & 0xffff);+            break;+        } else {+            break;+        }+    }++    return card;+}++bool roaring_bitmap_is_empty(const roaring_bitmap_t *r) {+    return r->high_low_container.size == 0;+}++void roaring_bitmap_to_uint32_array(const roaring_bitmap_t *r, uint32_t *ans) {+    ra_to_uint32_array(&r->high_low_container, ans);+}++bool roaring_bitmap_range_uint32_array(const roaring_bitmap_t *r, size_t offset,+                                       size_t limit, uint32_t *ans) {+    roaring_uint32_iterator_t it;+    roaring_iterator_init(r, &it);+    roaring_uint32_iterator_skip(&it, offset);+    roaring_uint32_iterator_read(&it, ans, limit);++    // This function always succeeds+    return true;+}++/** convert array and bitmap containers to run containers when it is more+ * efficient;+ * also convert from run containers when more space efficient.  Returns+ * true if the result has at least one run container.+ */+bool roaring_bitmap_run_optimize(roaring_bitmap_t *r) {+    bool answer = false;+    for (int i = 0; i < r->high_low_container.size; i++) {+        uint8_t type_original, type_after;+        ra_unshare_container_at_index(+            &r->high_low_container,+            (uint16_t)i);  // TODO: this introduces extra cloning!+        container_t *c = ra_get_container_at_index(&r->high_low_container,+                                                   (uint16_t)i, &type_original);+        container_t *c1 = convert_run_optimize(c, type_original, &type_after);+        if (type_after == RUN_CONTAINER_TYPE) {+            answer = true;+        }+        ra_set_container_at_index(&r->high_low_container, i, c1, type_after);+    }+    return answer;+}++size_t roaring_bitmap_shrink_to_fit(roaring_bitmap_t *r) {+    size_t answer = 0;+    for (int i = 0; i < r->high_low_container.size; i++) {+        uint8_t type_original;+        container_t *c = ra_get_container_at_index(&r->high_low_container,+                                                   (uint16_t)i, &type_original);+        answer += container_shrink_to_fit(c, type_original);+    }+    answer += ra_shrink_to_fit(&r->high_low_container);+    return answer;+}++/**+ *  Remove run-length encoding even when it is more space efficient+ *  return whether a change was applied+ */+bool roaring_bitmap_remove_run_compression(roaring_bitmap_t *r) {+    bool answer = false;+    for (int i = 0; i < r->high_low_container.size; i++) {+        uint8_t type_original, type_after;+        container_t *c = ra_get_container_at_index(&r->high_low_container,+                                                   (uint16_t)i, &type_original);+        if (get_container_type(c, type_original) == RUN_CONTAINER_TYPE) {+            answer = true;+            if (type_original == SHARED_CONTAINER_TYPE) {+                run_container_t *truec = CAST_run(CAST_shared(c)->container);+                int32_t card = run_container_cardinality(truec);+                container_t *c1 = convert_to_bitset_or_array_container(+                    truec, card, &type_after);+                shared_container_free(CAST_shared(c));  // frees run as needed+                ra_set_container_at_index(&r->high_low_container, i, c1,+                                          type_after);++            } else {+                int32_t card = run_container_cardinality(CAST_run(c));+                container_t *c1 = convert_to_bitset_or_array_container(+                    CAST_run(c), card, &type_after);+                run_container_free(CAST_run(c));+                ra_set_container_at_index(&r->high_low_container, i, c1,+                                          type_after);+            }+        }+    }+    return answer;+}++size_t roaring_bitmap_serialize(const roaring_bitmap_t *r, char *buf) {+    size_t portablesize = roaring_bitmap_portable_size_in_bytes(r);+    uint64_t cardinality = roaring_bitmap_get_cardinality(r);+    uint64_t sizeasarray = cardinality * sizeof(uint32_t) + sizeof(uint32_t);+    if (portablesize < sizeasarray) {+        buf[0] = CROARING_SERIALIZATION_CONTAINER;+        return roaring_bitmap_portable_serialize(r, buf + 1) + 1;+    } else {+        buf[0] = CROARING_SERIALIZATION_ARRAY_UINT32;+        uint32_t card_le = croaring_htole32((uint32_t)cardinality);+        memcpy(buf + 1, &card_le, sizeof(uint32_t));+        uint32_t *out = (uint32_t *)(buf + 1 + sizeof(uint32_t));+        roaring_bitmap_to_uint32_array(r, out);+#if CROARING_IS_BIG_ENDIAN+        for (uint64_t i = 0; i < cardinality; ++i) {+            uint32_t v;+            memcpy(&v, out + i, sizeof(uint32_t));+            v = croaring_htole32(v);+            memcpy(out + i, &v, sizeof(uint32_t));+        }+#endif+        return 1 + (size_t)sizeasarray;+    }+}++size_t roaring_bitmap_size_in_bytes(const roaring_bitmap_t *r) {+    size_t portablesize = roaring_bitmap_portable_size_in_bytes(r);+    uint64_t sizeasarray =+        roaring_bitmap_get_cardinality(r) * sizeof(uint32_t) + sizeof(uint32_t);+    return portablesize < sizeasarray ? portablesize + 1+                                      : (size_t)sizeasarray + 1;+}++size_t roaring_bitmap_portable_size_in_bytes(const roaring_bitmap_t *r) {+    return ra_portable_size_in_bytes(&r->high_low_container);+}++roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf,+                                                           size_t maxbytes) {+    roaring_bitmap_t *ans =+        (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t));+    if (ans == NULL) {+        return NULL;+    }+    size_t bytesread;+    bool is_ok = ra_portable_deserialize(&ans->high_low_container, buf,+                                         maxbytes, &bytesread);+    if (!is_ok) {+        roaring_free(ans);+        return NULL;+    }+    roaring_bitmap_set_copy_on_write(ans, false);+    if (!is_ok) {+        roaring_free(ans);+        return NULL;+    }+    return ans;+}++roaring_bitmap_t *roaring_bitmap_portable_deserialize(const char *buf) {+    return roaring_bitmap_portable_deserialize_safe(buf, SIZE_MAX);+}++size_t roaring_bitmap_portable_deserialize_size(const char *buf,+                                                size_t maxbytes) {+    return ra_portable_deserialize_size(buf, maxbytes);+}++size_t roaring_bitmap_portable_serialize(const roaring_bitmap_t *r, char *buf) {+    return ra_portable_serialize(&r->high_low_container, buf);+}++roaring_bitmap_t *roaring_bitmap_deserialize(const void *buf) {+    const char *bufaschar = (const char *)buf;+    if (bufaschar[0] == CROARING_SERIALIZATION_ARRAY_UINT32) {+        /* This looks like a compressed set of uint32_t elements */+        uint32_t card;++        memcpy(&card, bufaschar + 1, sizeof(uint32_t));+        card = croaring_letoh32(card);++        // The elements may not be aligned: keep a byte pointer and read+        // with memcpy rather than forming a misaligned uint32_t pointer.+        const char *elems = bufaschar + 1 + sizeof(uint32_t);++        roaring_bitmap_t *bitmap = roaring_bitmap_create();+        if (bitmap == NULL) {+            return NULL;+        }+        roaring_bulk_context_t context = CROARING_ZERO_INITIALIZER;+        for (uint32_t i = 0; i < card; i++) {+            uint32_t elem;+            memcpy(&elem, elems + i * sizeof(uint32_t), sizeof(elem));+            elem = croaring_letoh32(elem);+            roaring_bitmap_add_bulk(bitmap, &context, elem);+        }+        return bitmap;++    } else if (bufaschar[0] == CROARING_SERIALIZATION_CONTAINER) {+        return roaring_bitmap_portable_deserialize(bufaschar + 1);+    } else+        return (NULL);+}++roaring_bitmap_t *roaring_bitmap_deserialize_safe(const void *buf,+                                                  size_t maxbytes) {+    if (maxbytes < 1) {+        return NULL;+    }++    const char *bufaschar = (const char *)buf;+    if (bufaschar[0] == CROARING_SERIALIZATION_ARRAY_UINT32) {+        if (maxbytes < 1 + sizeof(uint32_t)) {+            return NULL;+        }++        /* This looks like a compressed set of uint32_t elements */+        uint32_t card;+        memcpy(&card, bufaschar + 1, sizeof(uint32_t));+        card = croaring_letoh32(card);++        // Check the buffer is big enough to contain card uint32_t elements+        if (maxbytes < 1 + sizeof(uint32_t) + card * sizeof(uint32_t)) {+            return NULL;+        }++        // The elements may not be aligned: keep a byte pointer and read+        // with memcpy rather than forming a misaligned uint32_t pointer.+        const char *elems = bufaschar + 1 + sizeof(uint32_t);++        roaring_bitmap_t *bitmap = roaring_bitmap_create();+        if (bitmap == NULL) {+            return NULL;+        }+        roaring_bulk_context_t context = CROARING_ZERO_INITIALIZER;+        for (uint32_t i = 0; i < card; i++) {+            uint32_t elem;+            memcpy(&elem, elems + i * sizeof(uint32_t), sizeof(elem));+            elem = croaring_letoh32(elem);+            roaring_bitmap_add_bulk(bitmap, &context, elem);+        }+        return bitmap;++    } else if (bufaschar[0] == CROARING_SERIALIZATION_CONTAINER) {+        return roaring_bitmap_portable_deserialize_safe(bufaschar + 1,+                                                        maxbytes - 1);+    } else+        return (NULL);+}++bool roaring_iterate(const roaring_bitmap_t *r, roaring_iterator iterator,+                     void *ptr) {+    const roaring_array_t *ra = &r->high_low_container;++    for (int i = 0; i < ra->size; ++i)+        if (!container_iterate(ra->containers[i], ra->typecodes[i],+                               ((uint32_t)ra->keys[i]) << 16, iterator, ptr)) {+            return false;+        }+    return true;+}++bool roaring_iterate64(const roaring_bitmap_t *r, roaring_iterator64 iterator,+                       uint64_t high_bits, void *ptr) {+    const roaring_array_t *ra = &r->high_low_container;++    for (int i = 0; i < ra->size; ++i)+        if (!container_iterate64(ra->containers[i], ra->typecodes[i],+                                 ((uint32_t)ra->keys[i]) << 16, iterator,+                                 high_bits, ptr)) {+            return false;+        }+    return true;+}++/****+ * begin roaring_uint32_iterator_t+ *****/++/**+ * Partially initializes the iterator. Leaves it in either state:+ * 1. Invalid due to `has_value = false`, or+ * 2. At a container, with the high bits set, `has_value = true`.+ */+CROARING_WARN_UNUSED static bool iter_new_container_partial_init(+    roaring_uint32_iterator_t *newit) {+    newit->current_value = 0;+    if (newit->container_index >= newit->parent->high_low_container.size ||+        newit->container_index < 0) {+        newit->current_value = UINT32_MAX;+        return (newit->has_value = false);+    }+    newit->has_value = true;+    // we precompute container, typecode and highbits so that successive+    // iterators do not have to grab them from odd memory locations+    // and have to worry about the (easily predicted) container_unwrap_shared+    // call.+    newit->container =+        newit->parent->high_low_container.containers[newit->container_index];+    newit->typecode =+        newit->parent->high_low_container.typecodes[newit->container_index];+    newit->highbits =+        ((uint32_t)+             newit->parent->high_low_container.keys[newit->container_index])+        << 16;+    newit->container =+        container_unwrap_shared(newit->container, &(newit->typecode));+    return true;+}++/**+ * Positions the iterator at the first value of the current container that the+ * iterator points at, if available.+ */+CROARING_WARN_UNUSED static bool loadfirstvalue(+    roaring_uint32_iterator_t *newit) {+    if (iter_new_container_partial_init(newit)) {+        uint16_t value = 0;+        newit->container_it =+            container_init_iterator(newit->container, newit->typecode, &value);+        newit->current_value = newit->highbits | value;+    }+    return newit->has_value;+}++/**+ * Positions the iterator at the last value of the current container that the+ * iterator points at, if available.+ */+CROARING_WARN_UNUSED static bool loadlastvalue(+    roaring_uint32_iterator_t *newit) {+    if (iter_new_container_partial_init(newit)) {+        uint16_t value = 0;+        newit->container_it = container_init_iterator_last(+            newit->container, newit->typecode, &value);+        newit->current_value = newit->highbits | value;+    }+    return newit->has_value;+}++/**+ * Positions the iterator at the smallest value that is larger than or equal to+ * `val` within the current container that the iterator points at. Assumes such+ * a value exists within the current container.+ */+CROARING_WARN_UNUSED static bool loadfirstvalue_largeorequal(+    roaring_uint32_iterator_t *newit, uint32_t val) {+    bool partial_init = iter_new_container_partial_init(newit);+    assert(partial_init);+    if (!partial_init) {+        return false;+    }+    uint16_t value = 0;+    newit->container_it =+        container_init_iterator(newit->container, newit->typecode, &value);+    bool found = container_iterator_lower_bound(+        newit->container, newit->typecode, &newit->container_it, &value,+        val & 0xFFFF);+    assert(found);+    if (!found) {+        return false;+    }+    newit->current_value = newit->highbits | value;+    return true;+}++void roaring_iterator_init(const roaring_bitmap_t *r,+                           roaring_uint32_iterator_t *newit) {+    newit->parent = r;+    newit->container_index = 0;+    newit->has_value = loadfirstvalue(newit);+}++void roaring_iterator_init_last(const roaring_bitmap_t *r,+                                roaring_uint32_iterator_t *newit) {+    newit->parent = r;+    newit->container_index = newit->parent->high_low_container.size - 1;+    newit->has_value = loadlastvalue(newit);+}++roaring_uint32_iterator_t *roaring_iterator_create(const roaring_bitmap_t *r) {+    roaring_uint32_iterator_t *newit =+        (roaring_uint32_iterator_t *)roaring_malloc(+            sizeof(roaring_uint32_iterator_t));+    if (newit == NULL) return NULL;+    roaring_iterator_init(r, newit);+    return newit;+}++roaring_uint32_iterator_t *roaring_uint32_iterator_copy(+    const roaring_uint32_iterator_t *it) {+    roaring_uint32_iterator_t *newit =+        (roaring_uint32_iterator_t *)roaring_malloc(+            sizeof(roaring_uint32_iterator_t));+    memcpy(newit, it, sizeof(roaring_uint32_iterator_t));+    return newit;+}++bool roaring_uint32_iterator_move_equalorlarger(roaring_uint32_iterator_t *it,+                                                uint32_t val) {+    uint16_t hb = val >> 16;+    const int i = ra_get_index(&it->parent->high_low_container, hb);+    if (i >= 0) {+        uint32_t lowvalue =+            container_maximum(it->parent->high_low_container.containers[i],+                              it->parent->high_low_container.typecodes[i]);+        uint16_t lb = val & 0xFFFF;+        if (lowvalue < lb) {+            // will have to load first value of next container+            it->container_index = i + 1;+        } else {+            // the value is necessarily within the range of the container+            it->container_index = i;+            it->has_value = loadfirstvalue_largeorequal(it, val);+            return it->has_value;+        }+    } else {+        // there is no matching, so we are going for the next container+        it->container_index = -i - 1;+    }+    it->has_value = loadfirstvalue(it);+    return it->has_value;+}++bool roaring_uint32_iterator_advance(roaring_uint32_iterator_t *it) {+    if (it->container_index >= it->parent->high_low_container.size) {+        return (it->has_value = false);+    }+    if (it->container_index < 0) {+        it->container_index = 0;+        return (it->has_value = loadfirstvalue(it));+    }+    uint16_t low16 = (uint16_t)it->current_value;+    if (container_iterator_next(it->container, it->typecode, &it->container_it,+                                &low16)) {+        it->current_value = it->highbits | low16;+        return (it->has_value = true);+    }+    it->container_index++;+    return (it->has_value = loadfirstvalue(it));+}++bool roaring_uint32_iterator_previous(roaring_uint32_iterator_t *it) {+    if (it->container_index < 0) {+        return (it->has_value = false);+    }+    if (it->container_index >= it->parent->high_low_container.size) {+        it->container_index = it->parent->high_low_container.size - 1;+        return (it->has_value = loadlastvalue(it));+    }+    uint16_t low16 = (uint16_t)it->current_value;+    if (container_iterator_prev(it->container, it->typecode, &it->container_it,+                                &low16)) {+        it->current_value = it->highbits | low16;+        return (it->has_value = true);+    }+    it->container_index--;+    return (it->has_value = loadlastvalue(it));+}++uint32_t roaring_uint32_iterator_read(roaring_uint32_iterator_t *it,+                                      uint32_t *buf, uint32_t count) {+    uint32_t ret = 0;+    while (it->has_value && ret < count) {+        uint32_t consumed;+        uint16_t low16 = (uint16_t)it->current_value;+        bool has_value = container_iterator_read_into_uint32(+            it->container, it->typecode, &it->container_it, it->highbits, buf,+            count - ret, &consumed, &low16);+        ret += consumed;+        buf += consumed;+        if (has_value) {+            it->has_value = true;+            it->current_value = it->highbits | low16;+            // If the container still has values, we must have stopped because+            // we read enough values.+            assert(ret == count);+            return ret;+        }+        it->container_index++;+        it->has_value = loadfirstvalue(it);+    }+    return ret;+}++uint32_t roaring_uint32_iterator_read_backward(roaring_uint32_iterator_t *it,+                                               uint32_t *buf, uint32_t count) {+    uint32_t ret = 0;+    while (it->has_value && ret < count) {+        uint32_t consumed;+        uint16_t low16 = (uint16_t)it->current_value;+        bool has_value = container_iterator_read_backward_into_uint32(+            it->container, it->typecode, &it->container_it, it->highbits, buf,+            count - ret, &consumed, &low16);+        ret += consumed;+        buf += consumed;+        if (has_value) {+            it->has_value = true;+            it->current_value = it->highbits | low16;+            // If the container still has values, we must have stopped because+            // we read enough values.+            assert(ret == count);+            return ret;+        }+        it->container_index--;+        it->has_value = loadlastvalue(it);+    }+    return ret;+}++uint32_t roaring_uint32_iterator_skip(roaring_uint32_iterator_t *it,+                                      uint32_t count) {+    uint32_t ret = 0;+    while (it->has_value && ret < count) {+        uint32_t consumed;+        uint16_t low16 = (uint16_t)it->current_value;+        bool has_value = container_iterator_skip(it->container, it->typecode,+                                                 &it->container_it, count - ret,+                                                 &consumed, &low16);+        ret += consumed;+        if (has_value) {+            it->has_value = true;+            it->current_value = it->highbits | low16;+            // If the container still has values, we must have stopped because+            // we skipped enough values.+            assert(ret == count);+            return ret;+        }+        // We have skipped over all items in the current container, so set+        // ourselves at the first item of the next container.+        // We do NOT need to count another item skipped here.+        it->container_index++;+        it->has_value = loadfirstvalue(it);+    }+    return ret;+}++uint32_t roaring_uint32_iterator_skip_backward(roaring_uint32_iterator_t *it,+                                               uint32_t count) {+    uint32_t ret = 0;+    while (it->has_value && ret < count) {+        uint32_t consumed;+        uint16_t low16 = (uint16_t)it->current_value;+        bool has_value = container_iterator_skip_backward(+            it->container, it->typecode, &it->container_it, count - ret,+            &consumed, &low16);+        ret += consumed;+        if (has_value) {+            it->has_value = true;+            it->current_value = it->highbits | low16;+            return ret;+        }+        // We have skipped over all items in the current container backwards.+        // Moving to the previous container counts as consuming one more skip.+        it->container_index--;+        it->has_value = loadlastvalue(it);+    }+    return ret;+}++size_t roaring_uint32_iterator_read_ranges(roaring_uint32_iterator_t *it,+                                           roaring_uint32_range_closed_t *buf,+                                           size_t count) {+    size_t ret = 0;+    while (it->has_value && ret < count) {+        buf[ret].min = it->current_value;+        for (;;) {+            uint16_t low16 = (uint16_t)it->current_value;+            bool container_has_more;+            uint16_t run_end_low16 = container_iterator_find_run_end(+                it->container, it->typecode, &it->container_it, &low16,+                &container_has_more);+            buf[ret].max = it->highbits | run_end_low16;++            if (container_has_more) {+                it->current_value = it->highbits | low16;+                break;+            }+            // Move to next container+            it->container_index++;+            it->has_value = loadfirstvalue(it);+            // Continue merging only if the run reached the container+            // boundary and the next container starts exactly at max+1.+            if (run_end_low16 != UINT16_MAX || !it->has_value ||+                it->current_value != buf[ret].max + 1) {+                break;+            }+        }+        ret++;+    }+    return ret;+}++size_t roaring_uint32_iterator_read_prev_ranges(+    roaring_uint32_iterator_t *it, roaring_uint32_range_closed_t *buf,+    size_t count) {+    size_t ret = 0;+    while (it->has_value && ret < count) {+        buf[ret].max = it->current_value;+        for (;;) {+            uint16_t low16 = (uint16_t)it->current_value;+            bool container_has_more;+            uint16_t run_start_low16 = container_iterator_find_run_start(+                it->container, it->typecode, &it->container_it, &low16,+                &container_has_more);+            buf[ret].min = it->highbits | run_start_low16;++            if (container_has_more) {+                it->current_value = it->highbits | low16;+                break;+            }+            // Move to previous container+            it->container_index--;+            it->has_value = loadlastvalue(it);+            // Continue merging only if the run reached the container+            // boundary and the previous container ends exactly at min-1.+            if (run_start_low16 != 0 || !it->has_value ||+                it->current_value != buf[ret].min - 1) {+                break;+            }+        }+        ret++;+    }+    return ret;+}++void roaring_uint32_iterator_free(roaring_uint32_iterator_t *it) {+    roaring_free(it);+}++/****+ * end of roaring_uint32_iterator_t+ *****/++bool roaring_bitmap_equals(const roaring_bitmap_t *r1,+                           const roaring_bitmap_t *r2) {+    const roaring_array_t *ra1 = &r1->high_low_container;+    const roaring_array_t *ra2 = &r2->high_low_container;++    if (ra1->size != ra2->size) {+        return false;+    }+    for (int i = 0; i < ra1->size; ++i) {+        if (ra1->keys[i] != ra2->keys[i]) {+            return false;+        }+    }+    for (int i = 0; i < ra1->size; ++i) {+        bool areequal = container_equals(ra1->containers[i], ra1->typecodes[i],+                                         ra2->containers[i], ra2->typecodes[i]);+        if (!areequal) {+            return false;+        }+    }+    return true;+}++bool roaring_bitmap_is_subset(const roaring_bitmap_t *r1,+                              const roaring_bitmap_t *r2) {+    const roaring_array_t *ra1 = &r1->high_low_container;+    const roaring_array_t *ra2 = &r2->high_low_container;++    const int length1 = ra1->size, length2 = ra2->size;++    int pos1 = 0, pos2 = 0;++    while (pos1 < length1 && pos2 < length2) {+        const uint16_t s1 = ra_get_key_at_index(ra1, (uint16_t)pos1);+        const uint16_t s2 = ra_get_key_at_index(ra2, (uint16_t)pos2);++        if (s1 == s2) {+            uint8_t type1, type2;+            container_t *c1 =+                ra_get_container_at_index(ra1, (uint16_t)pos1, &type1);+            container_t *c2 =+                ra_get_container_at_index(ra2, (uint16_t)pos2, &type2);+            if (!container_is_subset(c1, type1, c2, type2)) return false;+            ++pos1;+            ++pos2;+        } else if (s1 < s2) {  // s1 < s2+            return false;+        } else {  // s1 > s2+            pos2 = ra_advance_until(ra2, s1, pos2);+        }+    }+    if (pos1 == length1)+        return true;+    else+        return false;+}++static void insert_flipped_container(roaring_array_t *ans_arr,+                                     const roaring_array_t *x1_arr, uint16_t hb,+                                     uint16_t lb_start, uint16_t lb_end) {+    const int i = ra_get_index(x1_arr, hb);+    const int j = ra_get_index(ans_arr, hb);+    uint8_t ctype_in, ctype_out;+    container_t *flipped_container = NULL;+    if (i >= 0) {+        container_t *container_to_flip =+            ra_get_container_at_index(x1_arr, (uint16_t)i, &ctype_in);+        flipped_container =+            container_not_range(container_to_flip, ctype_in, (uint32_t)lb_start,+                                (uint32_t)(lb_end + 1), &ctype_out);++        if (container_nonzero_cardinality(flipped_container, ctype_out))+            ra_insert_new_key_value_at(ans_arr, -j - 1, hb, flipped_container,+                                       ctype_out);+        else {+            container_free(flipped_container, ctype_out);+        }+    } else {+        flipped_container = container_range_of_ones(+            (uint32_t)lb_start, (uint32_t)(lb_end + 1), &ctype_out);+        ra_insert_new_key_value_at(ans_arr, -j - 1, hb, flipped_container,+                                   ctype_out);+    }+}++static void inplace_flip_container(roaring_array_t *x1_arr, uint16_t hb,+                                   uint16_t lb_start, uint16_t lb_end) {+    const int i = ra_get_index(x1_arr, hb);+    uint8_t ctype_in, ctype_out;+    container_t *flipped_container = NULL;+    if (i >= 0) {+        container_t *container_to_flip =+            ra_get_container_at_index(x1_arr, (uint16_t)i, &ctype_in);+        flipped_container = container_inot_range(+            container_to_flip, ctype_in, (uint32_t)lb_start,+            (uint32_t)(lb_end + 1), &ctype_out);+        // if a new container was created, the old one was already freed+        if (container_nonzero_cardinality(flipped_container, ctype_out)) {+            ra_set_container_at_index(x1_arr, i, flipped_container, ctype_out);+        } else {+            container_free(flipped_container, ctype_out);+            ra_remove_at_index(x1_arr, i);+        }++    } else {+        flipped_container = container_range_of_ones(+            (uint32_t)lb_start, (uint32_t)(lb_end + 1), &ctype_out);+        ra_insert_new_key_value_at(x1_arr, -i - 1, hb, flipped_container,+                                   ctype_out);+    }+}++static void insert_fully_flipped_container(roaring_array_t *ans_arr,+                                           const roaring_array_t *x1_arr,+                                           uint16_t hb) {+    const int i = ra_get_index(x1_arr, hb);+    const int j = ra_get_index(ans_arr, hb);+    uint8_t ctype_in, ctype_out;+    container_t *flipped_container = NULL;+    if (i >= 0) {+        container_t *container_to_flip =+            ra_get_container_at_index(x1_arr, (uint16_t)i, &ctype_in);+        flipped_container =+            container_not(container_to_flip, ctype_in, &ctype_out);+        if (container_nonzero_cardinality(flipped_container, ctype_out))+            ra_insert_new_key_value_at(ans_arr, -j - 1, hb, flipped_container,+                                       ctype_out);+        else {+            container_free(flipped_container, ctype_out);+        }+    } else {+        flipped_container = container_range_of_ones(0U, 0x10000U, &ctype_out);+        ra_insert_new_key_value_at(ans_arr, -j - 1, hb, flipped_container,+                                   ctype_out);+    }+}++static void inplace_fully_flip_container(roaring_array_t *x1_arr, uint16_t hb) {+    const int i = ra_get_index(x1_arr, hb);+    uint8_t ctype_in, ctype_out;+    container_t *flipped_container = NULL;+    if (i >= 0) {+        container_t *container_to_flip =+            ra_get_container_at_index(x1_arr, (uint16_t)i, &ctype_in);+        flipped_container =+            container_inot(container_to_flip, ctype_in, &ctype_out);++        if (container_nonzero_cardinality(flipped_container, ctype_out)) {+            ra_set_container_at_index(x1_arr, i, flipped_container, ctype_out);+        } else {+            container_free(flipped_container, ctype_out);+            ra_remove_at_index(x1_arr, i);+        }++    } else {+        flipped_container = container_range_of_ones(0U, 0x10000U, &ctype_out);+        ra_insert_new_key_value_at(x1_arr, -i - 1, hb, flipped_container,+                                   ctype_out);+    }+}++roaring_bitmap_t *roaring_bitmap_flip(const roaring_bitmap_t *x1,+                                      uint64_t range_start,+                                      uint64_t range_end) {+    if (range_end > (uint64_t)UINT32_MAX + 1) {+        range_end = (uint64_t)UINT32_MAX + 1;+    }+    if (range_start >= range_end) {+        return roaring_bitmap_copy(x1);+    }+    return roaring_bitmap_flip_closed(x1, (uint32_t)range_start,+                                      (uint32_t)(range_end - 1));+}++roaring_bitmap_t *roaring_bitmap_flip_closed(const roaring_bitmap_t *x1,+                                             uint32_t range_start,+                                             uint32_t range_end) {+    if (range_start > range_end) {+        return roaring_bitmap_copy(x1);+    }++    roaring_bitmap_t *ans = roaring_bitmap_create();+    roaring_bitmap_set_copy_on_write(ans, is_cow(x1));++    uint16_t hb_start = (uint16_t)(range_start >> 16);+    const uint16_t lb_start = (uint16_t)range_start;  // & 0xFFFF;+    uint16_t hb_end = (uint16_t)(range_end >> 16);+    const uint16_t lb_end = (uint16_t)range_end;  // & 0xFFFF;++    ra_append_copies_until(&ans->high_low_container, &x1->high_low_container,+                           hb_start, is_cow(x1));+    if (hb_start == hb_end) {+        insert_flipped_container(&ans->high_low_container,+                                 &x1->high_low_container, hb_start, lb_start,+                                 lb_end);+    } else {+        // start and end containers are distinct+        if (lb_start > 0) {+            // handle first (partial) container+            insert_flipped_container(&ans->high_low_container,+                                     &x1->high_low_container, hb_start,+                                     lb_start, 0xFFFF);+            ++hb_start;  // for the full containers.  Can't wrap.+        }++        if (lb_end != 0xFFFF) --hb_end;  // later we'll handle the partial block++        for (uint32_t hb = hb_start; hb <= hb_end; ++hb) {+            insert_fully_flipped_container(&ans->high_low_container,+                                           &x1->high_low_container,+                                           (uint16_t)hb);+        }++        // handle a partial final container+        if (lb_end != 0xFFFF) {+            insert_flipped_container(&ans->high_low_container,+                                     &x1->high_low_container, hb_end + 1, 0,+                                     lb_end);+            ++hb_end;+        }+    }+    ra_append_copies_after(&ans->high_low_container, &x1->high_low_container,+                           hb_end, is_cow(x1));+    return ans;+}++void roaring_bitmap_flip_inplace(roaring_bitmap_t *x1, uint64_t range_start,+                                 uint64_t range_end) {+    if (range_end > (uint64_t)UINT32_MAX + 1) {+        range_end = (uint64_t)UINT32_MAX + 1;+    }+    if (range_start >= range_end) {+        return;+    }+    roaring_bitmap_flip_inplace_closed(x1, (uint32_t)range_start,+                                       (uint32_t)(range_end - 1));+}++void roaring_bitmap_flip_inplace_closed(roaring_bitmap_t *x1,+                                        uint32_t range_start,+                                        uint32_t range_end) {+    if (range_start > range_end) {+        return;  // empty range+    }++    uint16_t hb_start = (uint16_t)(range_start >> 16);+    const uint16_t lb_start = (uint16_t)range_start;+    uint16_t hb_end = (uint16_t)(range_end >> 16);+    const uint16_t lb_end = (uint16_t)range_end;++    if (hb_start == hb_end) {+        inplace_flip_container(&x1->high_low_container, hb_start, lb_start,+                               lb_end);+    } else {+        // start and end containers are distinct+        if (lb_start > 0) {+            // handle first (partial) container+            inplace_flip_container(&x1->high_low_container, hb_start, lb_start,+                                   0xFFFF);+            ++hb_start;  // for the full containers.  Can't wrap.+        }++        if (lb_end != 0xFFFF) --hb_end;++        for (uint32_t hb = hb_start; hb <= hb_end; ++hb) {+            inplace_fully_flip_container(&x1->high_low_container, (uint16_t)hb);+        }+        // handle a partial final container+        if (lb_end != 0xFFFF) {+            inplace_flip_container(&x1->high_low_container, hb_end + 1, 0,+                                   lb_end);+            ++hb_end;+        }+    }+}++static void offset_append_with_merge(roaring_array_t *ra, int k, container_t *c,+                                     uint8_t t) {+    int size = ra_get_size(ra);+    if (size == 0 || ra_get_key_at_index(ra, (uint16_t)(size - 1)) != k) {+        // No merge.+        ra_append(ra, (uint16_t)k, c, t);+        return;+    }++    uint8_t last_t, new_t;+    container_t *last_c, *new_c;++    // NOTE: we don't need to unwrap here, since we added last_c ourselves+    // we have the certainty it's not a shared container.+    // The same applies to c, as it's the result of calling container_offset.+    last_c = ra_get_container_at_index(ra, (uint16_t)(size - 1), &last_t);+    new_c = container_ior(last_c, last_t, c, t, &new_t);++    ra_set_container_at_index(ra, size - 1, new_c, new_t);++    // Comparison of pointers of different origin is UB (or so claim some+    // compiler makers), so we compare their bit representation only.+    if ((uintptr_t)last_c != (uintptr_t)new_c) {+        container_free(last_c, last_t);+    }+    container_free(c, t);+}++// roaring_bitmap_add_offset adds the value 'offset' to each and every value in+// a bitmap, generating a new bitmap in the process. If offset + element is+// outside of the range [0,2^32), that the element will be dropped.+// We need "offset" to be 64 bits because we want to support values+// between -0xFFFFFFFF up to +0xFFFFFFFF.+roaring_bitmap_t *roaring_bitmap_add_offset(const roaring_bitmap_t *bm,+                                            int64_t offset) {+    roaring_bitmap_t *answer;+    roaring_array_t *ans_ra;+    int64_t container_offset;+    uint16_t in_offset;++    const roaring_array_t *bm_ra = &bm->high_low_container;+    int length = bm_ra->size;++    if (offset == 0) {+        return roaring_bitmap_copy(bm);+    }++    container_offset = offset >> 16;+    in_offset = (uint16_t)(offset - container_offset * (1 << 16));++    answer = roaring_bitmap_create();+    bool cow = is_cow(bm);+    roaring_bitmap_set_copy_on_write(answer, cow);++    ans_ra = &answer->high_low_container;++    if (in_offset == 0) {+        ans_ra = &answer->high_low_container;++        for (int i = 0, j = 0; i < length; ++i) {+            int64_t key = ra_get_key_at_index(bm_ra, (uint16_t)i);+            key += container_offset;++            if (key < 0 || key >= (1 << 16)) {+                continue;+            }+            ra_append_copy(ans_ra, bm_ra, (uint16_t)i, cow);+            ans_ra->keys[j++] = (uint16_t)key;+        }+        return answer;+    }++    uint8_t t;+    const container_t *c;+    container_t *lo, *hi, **lo_ptr, **hi_ptr;+    int64_t k;++    for (int i = 0; i < length; ++i) {+        lo = hi = NULL;+        lo_ptr = hi_ptr = NULL;++        k = ra_get_key_at_index(bm_ra, (uint16_t)i) + container_offset;+        if (k >= 0 && k < (1 << 16)) {+            lo_ptr = &lo;+        }+        if (k + 1 >= 0 && k + 1 < (1 << 16)) {+            hi_ptr = &hi;+        }+        if (lo_ptr == NULL && hi_ptr == NULL) {+            continue;+        }+        c = ra_get_container_at_index(bm_ra, (uint16_t)i, &t);+        c = container_unwrap_shared(c, &t);++        container_add_offset(c, t, lo_ptr, hi_ptr, in_offset);+        if (lo != NULL) {+            offset_append_with_merge(ans_ra, (int)k, lo, t);+        }+        if (hi != NULL) {+            ra_append(ans_ra, (uint16_t)(k + 1), hi, t);+        }+        // the `lo` and `hi` container type always keep same as container `c`.+        // in the case of `container_add_offset` on bitset container, `lo` and+        // `hi` may has small cardinality, they must be repaired to array+        // container.+    }++    roaring_bitmap_repair_after_lazy(answer);  // do required type conversions.+    return answer;+}++roaring_bitmap_t *roaring_bitmap_lazy_or(const roaring_bitmap_t *x1,+                                         const roaring_bitmap_t *x2,+                                         const bool bitsetconversion) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    if (0 == length1) {+        return roaring_bitmap_copy(x2);+    }+    if (0 == length2) {+        return roaring_bitmap_copy(x1);+    }+    roaring_bitmap_t *answer =+        roaring_bitmap_create_with_capacity(length1 + length2);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c;+            if (bitsetconversion &&+                (get_container_type(c1, type1) != BITSET_CONTAINER_TYPE) &&+                (get_container_type(c2, type2) != BITSET_CONTAINER_TYPE)) {+                container_t *newc1 =+                    container_mutable_unwrap_shared(c1, &type1);+                newc1 = container_to_bitset(newc1, type1);+                type1 = BITSET_CONTAINER_TYPE;+                c = container_lazy_ior(newc1, type1, c2, type2, &result_type);+                if (c != newc1) {  // should not happen+                    container_free(newc1, type1);+                }+            } else {+                c = container_lazy_or(c1, type1, c2, type2, &result_type);+            }+            // since we assume that the initial containers are non-empty,+            // the+            // result here+            // can only be non-empty+            ra_append(&answer->high_low_container, s1, c, result_type);+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            c1 = get_copy_of_container(c1, &type1, is_cow(x1));+            if (is_cow(x1)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c1,+                                          type1);+            }+            ra_append(&answer->high_low_container, s1, c1, type1);+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_append(&answer->high_low_container, s2, c2, type2);+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&answer->high_low_container,+                             &x2->high_low_container, pos2, length2,+                             is_cow(x2));+    } else if (pos2 == length2) {+        ra_append_copy_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1,+                             is_cow(x1));+    }+    return answer;+}++void roaring_bitmap_lazy_or_inplace(roaring_bitmap_t *x1,+                                    const roaring_bitmap_t *x2,+                                    const bool bitsetconversion) {+    uint8_t result_type = 0;+    int length1 = x1->high_low_container.size;+    const int length2 = x2->high_low_container.size;++    if (0 == length2) return;++    if (0 == length1) {+        roaring_bitmap_overwrite(x1, x2);+        return;+    }+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            if (!container_is_full(c1, type1)) {+                if ((bitsetconversion == false) ||+                    (get_container_type(c1, type1) == BITSET_CONTAINER_TYPE)) {+                    c1 = get_writable_copy_if_shared(c1, &type1);+                } else {+                    // convert to bitset+                    container_t *old_c1 = c1;+                    uint8_t old_type1 = type1;+                    c1 = container_mutable_unwrap_shared(c1, &type1);+                    c1 = container_to_bitset(c1, type1);+                    container_free(old_c1, old_type1);+                    type1 = BITSET_CONTAINER_TYPE;+                }++                container_t *c2 = ra_get_container_at_index(+                    &x2->high_low_container, (uint16_t)pos2, &type2);+                container_t *c =+                    container_lazy_ior(c1, type1, c2, type2, &result_type);++                if (c != c1) {  // in this instance a new container was created,+                                // and we need to free the old one+                    container_free(c1, type1);+                }++                ra_set_container_at_index(&x1->high_low_container, pos1, c,+                                          result_type);+            }+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            // container_t *c2_clone = container_clone(c2, type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_insert_new_key_value_at(&x1->high_low_container, pos1, s2, c2,+                                       type2);+            pos1++;+            length1++;+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&x1->high_low_container, &x2->high_low_container,+                             pos2, length2, is_cow(x2));+    }+}++roaring_bitmap_t *roaring_bitmap_lazy_xor(const roaring_bitmap_t *x1,+                                          const roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    if (0 == length1) {+        return roaring_bitmap_copy(x2);+    }+    if (0 == length2) {+        return roaring_bitmap_copy(x1);+    }+    roaring_bitmap_t *answer =+        roaring_bitmap_create_with_capacity(length1 + length2);+    roaring_bitmap_set_copy_on_write(answer, is_cow(x1) || is_cow(x2));+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            container_t *c =+                container_lazy_xor(c1, type1, c2, type2, &result_type);++            if (container_nonzero_cardinality(c, result_type)) {+                ra_append(&answer->high_low_container, s1, c, result_type);+            } else {+                container_free(c, result_type);+            }++            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            c1 = get_copy_of_container(c1, &type1, is_cow(x1));+            if (is_cow(x1)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c1,+                                          type1);+            }+            ra_append(&answer->high_low_container, s1, c1, type1);+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_append(&answer->high_low_container, s2, c2, type2);+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&answer->high_low_container,+                             &x2->high_low_container, pos2, length2,+                             is_cow(x2));+    } else if (pos2 == length2) {+        ra_append_copy_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1,+                             is_cow(x1));+    }+    return answer;+}++void roaring_bitmap_lazy_xor_inplace(roaring_bitmap_t *x1,+                                     const roaring_bitmap_t *x2) {+    assert(x1 != x2);+    uint8_t result_type = 0;+    int length1 = x1->high_low_container.size;+    const int length2 = x2->high_low_container.size;++    if (0 == length2) return;++    if (0 == length1) {+        roaring_bitmap_overwrite(x1, x2);+        return;+    }+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);++            // We do the computation "in place" only when c1 is not a shared+            // container. Rationale: using a shared container safely with in+            // place computation would require making a copy and then doing the+            // computation in place which is likely less efficient than avoiding+            // in place entirely and always generating a new container.++            container_t *c;+            if (type1 == SHARED_CONTAINER_TYPE) {+                c = container_lazy_xor(c1, type1, c2, type2, &result_type);+                shared_container_free(CAST_shared(c1));  // release+            } else {+                c = container_lazy_ixor(c1, type1, c2, type2, &result_type);+            }++            if (container_nonzero_cardinality(c, result_type)) {+                ra_set_container_at_index(&x1->high_low_container, pos1, c,+                                          result_type);+                ++pos1;+            } else {+                container_free(c, result_type);+                ra_remove_at_index(&x1->high_low_container, pos1);+                --length1;+            }+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            // container_t *c2_clone = container_clone(c2, type2);+            c2 = get_copy_of_container(c2, &type2, is_cow(x2));+            if (is_cow(x2)) {+                ra_set_container_at_index(&x2->high_low_container, pos2, c2,+                                          type2);+            }+            ra_insert_new_key_value_at(&x1->high_low_container, pos1, s2, c2,+                                       type2);+            pos1++;+            length1++;+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_copy_range(&x1->high_low_container, &x2->high_low_container,+                             pos2, length2, is_cow(x2));+    }+}++void roaring_bitmap_repair_after_lazy(roaring_bitmap_t *r) {+    roaring_array_t *ra = &r->high_low_container;++    for (int i = 0; i < ra->size; ++i) {+        const uint8_t old_type = ra->typecodes[i];+        container_t *old_c = ra->containers[i];+        uint8_t new_type = old_type;+        container_t *new_c = container_repair_after_lazy(old_c, &new_type);+        ra->containers[i] = new_c;+        ra->typecodes[i] = new_type;+    }+}++/**+ * roaring_bitmap_rank returns the number of integers that are smaller or equal+ * to x.+ */+uint64_t roaring_bitmap_rank(const roaring_bitmap_t *bm, uint32_t x) {+    uint64_t size = 0;+    uint32_t xhigh = x >> 16;+    for (int i = 0; i < bm->high_low_container.size; i++) {+        uint32_t key = bm->high_low_container.keys[i];+        if (xhigh > key) {+            size +=+                container_get_cardinality(bm->high_low_container.containers[i],+                                          bm->high_low_container.typecodes[i]);+        } else if (xhigh == key) {+            return size + container_rank(bm->high_low_container.containers[i],+                                         bm->high_low_container.typecodes[i],+                                         x & 0xFFFF);+        } else {+            return size;+        }+    }+    return size;+}+void roaring_bitmap_rank_many(const roaring_bitmap_t *bm, const uint32_t *begin,+                              const uint32_t *end, uint64_t *ans) {+    uint64_t size = 0;++    int i = 0;+    const uint32_t *iter = begin;+    while (i < bm->high_low_container.size && iter != end) {+        uint32_t x = *iter;+        uint32_t xhigh = x >> 16;+        uint32_t key = bm->high_low_container.keys[i];+        if (xhigh > key) {+            size +=+                container_get_cardinality(bm->high_low_container.containers[i],+                                          bm->high_low_container.typecodes[i]);+            i++;+        } else if (xhigh == key) {+            uint32_t consumed = container_rank_many(+                bm->high_low_container.containers[i],+                bm->high_low_container.typecodes[i], size, iter, end, ans);+            iter += consumed;+            ans += consumed;+        } else {+            *(ans++) = size;+            iter++;+        }+    }+    while (iter != end) {  // must have N outputs for N inputs...+        *(ans++) = size;   // ...everything left is beyond all containers+        iter++;+    }+}++/**+ * roaring_bitmap_get_index returns the index of x, if not exsist return -1.+ */+int64_t roaring_bitmap_get_index(const roaring_bitmap_t *bm, uint32_t x) {+    int64_t index = 0;+    const uint16_t xhigh = x >> 16;+    int32_t high_idx = ra_get_index(&bm->high_low_container, xhigh);+    if (high_idx < 0) return -1;++    for (int i = 0; i < bm->high_low_container.size; i++) {+        uint32_t key = bm->high_low_container.keys[i];+        if (xhigh > key) {+            index +=+                container_get_cardinality(bm->high_low_container.containers[i],+                                          bm->high_low_container.typecodes[i]);+        } else if (xhigh == key) {+            int32_t low_idx = container_get_index(+                bm->high_low_container.containers[high_idx],+                bm->high_low_container.typecodes[high_idx], x & 0xFFFF);+            if (low_idx < 0) return -1;+            return index + low_idx;+        } else {+            return -1;+        }+    }+    return index;+}++/**+ * roaring_bitmap_smallest returns the smallest value in the set.+ * Returns UINT32_MAX if the set is empty.+ */+uint32_t roaring_bitmap_minimum(const roaring_bitmap_t *bm) {+    if (bm->high_low_container.size > 0) {+        container_t *c = bm->high_low_container.containers[0];+        uint8_t type = bm->high_low_container.typecodes[0];+        uint32_t key = bm->high_low_container.keys[0];+        uint32_t lowvalue = container_minimum(c, type);+        return lowvalue | (key << 16);+    }+    return UINT32_MAX;+}++/**+ * roaring_bitmap_smallest returns the greatest value in the set.+ * Returns 0 if the set is empty.+ */+uint32_t roaring_bitmap_maximum(const roaring_bitmap_t *bm) {+    if (bm->high_low_container.size > 0) {+        container_t *container =+            bm->high_low_container.containers[bm->high_low_container.size - 1];+        uint8_t typecode =+            bm->high_low_container.typecodes[bm->high_low_container.size - 1];+        uint32_t key =+            bm->high_low_container.keys[bm->high_low_container.size - 1];+        uint32_t lowvalue = container_maximum(container, typecode);+        return lowvalue | (key << 16);+    }+    return 0;+}++bool roaring_bitmap_select(const roaring_bitmap_t *bm, uint32_t rank,+                           uint32_t *element) {+    const roaring_array_t *ra = &bm->high_low_container;+    uint32_t start_rank = 0;+    for (int32_t i = 0; i < ra->size; i++) {+        uint8_t typecode = ra->typecodes[i];+        const container_t *c =+            container_unwrap_shared(ra->containers[i], &typecode);+        uint32_t card = (uint32_t)container_get_cardinality(c, typecode);+        if (rank - start_rank < card) {  // rank >= start_rank always holds+            if (!container_select(c, typecode, &start_rank, rank, element)) {+                return false;+            }+            // w/o cast, key promotes signed+            *element |= (((uint32_t)ra->keys[i]) << 16);+            return true;+        }+        start_rank += card;+    }+    return false;+}++bool roaring_bitmap_intersect(const roaring_bitmap_t *x1,+                              const roaring_bitmap_t *x2) {+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    uint64_t answer = 0;+    int pos1 = 0, pos2 = 0;++    while (pos1 < length1 && pos2 < length2) {+        const uint16_t s1 =+            ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+        const uint16_t s2 =+            ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        if (s1 == s2) {+            uint8_t type1, type2;+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            if (container_intersect(c1, type1, c2, type2)) return true;+            ++pos1;+            ++pos2;+        } else if (s1 < s2) {  // s1 < s2+            pos1 = ra_advance_until(&x1->high_low_container, s2, pos1);+        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+        }+    }+    return answer != 0;+}++bool roaring_bitmap_intersect_with_range(const roaring_bitmap_t *bm, uint64_t x,+                                         uint64_t y) {+    if (y > (uint64_t)UINT32_MAX + 1) {+        y = (uint64_t)UINT32_MAX + 1;+    }+    if (x >= y) {+        // Empty range (or range entirely beyond UINT32_MAX).+        return false;+    }+    roaring_uint32_iterator_t it;+    roaring_iterator_init(bm, &it);+    if (!roaring_uint32_iterator_move_equalorlarger(&it, (uint32_t)x)) {+        // No values above x.+        return false;+    }+    if (it.current_value >= y) {+        // No values below y.+        return false;+    }+    return true;+}++uint64_t roaring_bitmap_and_cardinality(const roaring_bitmap_t *x1,+                                        const roaring_bitmap_t *x2) {+    const int length1 = x1->high_low_container.size,+              length2 = x2->high_low_container.size;+    uint64_t answer = 0;+    int pos1 = 0, pos2 = 0;+    while (pos1 < length1 && pos2 < length2) {+        const uint16_t s1 =+            ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+        const uint16_t s2 =+            ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        if (s1 == s2) {+            uint8_t type1, type2;+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            answer += container_and_cardinality(c1, type1, c2, type2);+            ++pos1;+            ++pos2;+        } else if (s1 < s2) {  // s1 < s2+            pos1 = ra_advance_until(&x1->high_low_container, s2, pos1);+        } else {  // s1 > s2+            pos2 = ra_advance_until(&x2->high_low_container, s1, pos2);+        }+    }+    return answer;+}++double roaring_bitmap_jaccard_index(const roaring_bitmap_t *x1,+                                    const roaring_bitmap_t *x2) {+    const uint64_t c1 = roaring_bitmap_get_cardinality(x1);+    const uint64_t c2 = roaring_bitmap_get_cardinality(x2);+    const uint64_t inter = roaring_bitmap_and_cardinality(x1, x2);+    return (double)inter / (double)(c1 + c2 - inter);+}++uint64_t roaring_bitmap_or_cardinality(const roaring_bitmap_t *x1,+                                       const roaring_bitmap_t *x2) {+    const uint64_t c1 = roaring_bitmap_get_cardinality(x1);+    const uint64_t c2 = roaring_bitmap_get_cardinality(x2);+    const uint64_t inter = roaring_bitmap_and_cardinality(x1, x2);+    return c1 + c2 - inter;+}++uint64_t roaring_bitmap_andnot_cardinality(const roaring_bitmap_t *x1,+                                           const roaring_bitmap_t *x2) {+    const uint64_t c1 = roaring_bitmap_get_cardinality(x1);+    const uint64_t inter = roaring_bitmap_and_cardinality(x1, x2);+    return c1 - inter;+}++uint64_t roaring_bitmap_xor_cardinality(const roaring_bitmap_t *x1,+                                        const roaring_bitmap_t *x2) {+    const uint64_t c1 = roaring_bitmap_get_cardinality(x1);+    const uint64_t c2 = roaring_bitmap_get_cardinality(x2);+    const uint64_t inter = roaring_bitmap_and_cardinality(x1, x2);+    return c1 + c2 - 2 * inter;+}++/**+ * Check whether a range of values from range_start (included) to range_end+ * (excluded) is present+ */+bool roaring_bitmap_contains_range(const roaring_bitmap_t *r,+                                   uint64_t range_start, uint64_t range_end) {+    if (range_end > (uint64_t)UINT32_MAX + 1) {+        range_end = (uint64_t)UINT32_MAX + 1;+    }+    if (range_start >= range_end) {+        return true;+    }+    return roaring_bitmap_contains_range_closed(r, (uint32_t)range_start,+                                                (uint32_t)(range_end - 1));+}++/**+ * Check whether a range of values from range_start (included) to range_end+ * (included) is present+ */+bool roaring_bitmap_contains_range_closed(const roaring_bitmap_t *r,+                                          uint32_t range_start,+                                          uint32_t range_end) {+    if (range_start > range_end) {+        return true;+    }  // empty range are always contained!+    if (range_end == range_start) {+        return roaring_bitmap_contains(r, (uint32_t)range_start);+    }+    uint16_t hb_rs = (uint16_t)(range_start >> 16);+    uint16_t hb_re = (uint16_t)(range_end >> 16);+    const int32_t span = hb_re - hb_rs;+    const int32_t hlc_sz = ra_get_size(&r->high_low_container);+    if (hlc_sz < span + 1) {+        return false;+    }+    int32_t is = ra_get_index(&r->high_low_container, hb_rs);+    int32_t ie = ra_get_index(&r->high_low_container, hb_re);+    if ((ie < 0) || (is < 0) || ((ie - is) != span) || ie >= hlc_sz) {+        return false;+    }+    const uint32_t lb_rs = range_start & 0xFFFF;+    const uint32_t lb_re = (range_end & 0xFFFF) + 1;+    uint8_t type;+    container_t *c =+        ra_get_container_at_index(&r->high_low_container, (uint16_t)is, &type);+    if (hb_rs == hb_re) {+        return container_contains_range(c, lb_rs, lb_re, type);+    }+    if (!container_contains_range(c, lb_rs, 1 << 16, type)) {+        return false;+    }+    c = ra_get_container_at_index(&r->high_low_container, (uint16_t)ie, &type);+    if (!container_contains_range(c, 0, lb_re, type)) {+        return false;+    }+    for (int32_t i = is + 1; i < ie; ++i) {+        c = ra_get_container_at_index(&r->high_low_container, (uint16_t)i,+                                      &type);+        if (!container_is_full(c, type)) {+            return false;+        }+    }+    return true;+}++bool roaring_bitmap_is_strict_subset(const roaring_bitmap_t *r1,+                                     const roaring_bitmap_t *r2) {+    return (roaring_bitmap_get_cardinality(r2) >+                roaring_bitmap_get_cardinality(r1) &&+            roaring_bitmap_is_subset(r1, r2));+}++/*+ * FROZEN SERIALIZATION FORMAT DESCRIPTION+ *+ * -- (beginning must be aligned by 32 bytes) --+ * <bitset_data> uint64_t[BITSET_CONTAINER_SIZE_IN_WORDS *+ * num_bitset_containers] <run_data>    rle16_t[total number of rle elements in+ * all run containers] <array_data>  uint16_t[total number of array elements in+ * all array containers] <keys>        uint16_t[num_containers] <counts>+ * uint16_t[num_containers] <typecodes>   uint8_t[num_containers] <header>+ * uint32_t+ *+ * <header> is a 4-byte value which is a bit union of FROZEN_COOKIE (15 bits)+ * and the number of containers (17 bits).+ *+ * <counts> stores number of elements for every container.+ * Its meaning depends on container type.+ * For array and bitset containers, this value is the container cardinality+ * minus one. For run container, it is the number of rle_t elements (n_runs).+ *+ * <bitset_data>,<array_data>,<run_data> are flat arrays of elements of+ * all containers of respective type.+ *+ * <*_data> and <keys> are kept close together because they are not accessed+ * during deserilization. This may reduce IO in case of large mmaped bitmaps.+ * All members have their native alignments during deserilization except+ * <header>, which is not guaranteed to be aligned by 4 bytes.+ */++size_t roaring_bitmap_frozen_size_in_bytes(const roaring_bitmap_t *rb) {+    const roaring_array_t *ra = &rb->high_low_container;+    size_t num_bytes = 0;+    for (int32_t i = 0; i < ra->size; i++) {+        uint8_t typecode = ra->typecodes[i];+        const container_t *c =+            container_unwrap_shared(ra->containers[i], &typecode);+        switch (typecode) {+            case BITSET_CONTAINER_TYPE: {+                num_bytes += BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+                break;+            }+            case RUN_CONTAINER_TYPE: {+                const run_container_t *rc = const_CAST_run(c);+                num_bytes += rc->n_runs * sizeof(rle16_t);+                break;+            }+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *ac = const_CAST_array(c);+                num_bytes += ac->cardinality * sizeof(uint16_t);+                break;+            }+            default:+                roaring_unreachable;+        }+    }+    num_bytes += (2 + 2 + 1) * ra->size;  // keys, counts, typecodes+    num_bytes += 4;                       // header+    return num_bytes;+}++inline static void *arena_alloc(char **arena, size_t num_bytes) {+    char *res = *arena;+    *arena += num_bytes;+    return res;+}++void roaring_bitmap_frozen_serialize(const roaring_bitmap_t *rb, char *buf) {+    /*+     * Note: we do not require user to supply a specifically aligned buffer.+     * Thus we have to use memcpy() everywhere.+     */++    const roaring_array_t *ra = &rb->high_low_container;++    size_t bitset_zone_size = 0;+    size_t run_zone_size = 0;+    size_t array_zone_size = 0;+    for (int32_t i = 0; i < ra->size; i++) {+        uint8_t typecode = ra->typecodes[i];+        const container_t *c =+            container_unwrap_shared(ra->containers[i], &typecode);+        switch (typecode) {+            case BITSET_CONTAINER_TYPE: {+                bitset_zone_size +=+                    BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+                break;+            }+            case RUN_CONTAINER_TYPE: {+                const run_container_t *rc = const_CAST_run(c);+                run_zone_size += rc->n_runs * sizeof(rle16_t);+                break;+            }+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *ac = const_CAST_array(c);+                array_zone_size += ac->cardinality * sizeof(uint16_t);+                break;+            }+            default:+                roaring_unreachable;+        }+    }++    uint64_t *bitset_zone = (uint64_t *)arena_alloc(&buf, bitset_zone_size);+    rle16_t *run_zone = (rle16_t *)arena_alloc(&buf, run_zone_size);+    uint16_t *array_zone = (uint16_t *)arena_alloc(&buf, array_zone_size);+    uint16_t *key_zone = (uint16_t *)arena_alloc(&buf, 2 * ra->size);+    uint16_t *count_zone = (uint16_t *)arena_alloc(&buf, 2 * ra->size);+    uint8_t *typecode_zone = (uint8_t *)arena_alloc(&buf, ra->size);+    char *header_zone = (char *)arena_alloc(&buf, 4);++    for (int32_t i = 0; i < ra->size; i++) {+        uint16_t count;+        uint8_t typecode = ra->typecodes[i];+        const container_t *c =+            container_unwrap_shared(ra->containers[i], &typecode);+        switch (typecode) {+            case BITSET_CONTAINER_TYPE: {+                const bitset_container_t *bc = const_CAST_bitset(c);+                memcpy(bitset_zone, bc->words,+                       BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t));+                bitset_zone += BITSET_CONTAINER_SIZE_IN_WORDS;+                if (bc->cardinality != BITSET_UNKNOWN_CARDINALITY) {+                    count = (uint16_t)(bc->cardinality - 1);+                } else {+                    count =+                        (uint16_t)(bitset_container_compute_cardinality(bc) -+                                   1);+                }+                break;+            }+            case RUN_CONTAINER_TYPE: {+                const run_container_t *rc = const_CAST_run(c);+                size_t num_bytes = rc->n_runs * sizeof(rle16_t);+                memcpy(run_zone, rc->runs, num_bytes);+                run_zone += rc->n_runs;+                count = (uint16_t)rc->n_runs;+                break;+            }+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *ac = const_CAST_array(c);+                size_t num_bytes = ac->cardinality * sizeof(uint16_t);+                memcpy(array_zone, ac->array, num_bytes);+                array_zone += ac->cardinality;+                count = (uint16_t)(ac->cardinality - 1);+                break;+            }+            default:+                roaring_unreachable;+        }+        memcpy(&count_zone[i], &count, 2);+        typecode_zone[i] = typecode;+    }+    if (ra->size > 0) {  // ra->keys is NULL when the bitmap is empty+        memcpy(key_zone, ra->keys, ra->size * sizeof(uint16_t));+    }+    uint32_t header = ((uint32_t)ra->size << 15) | FROZEN_COOKIE;+    memcpy(header_zone, &header, 4);+}++const roaring_bitmap_t *roaring_bitmap_frozen_view(const char *buf,+                                                   size_t length) {+    if ((uintptr_t)buf % 32 != 0) {+        return NULL;+    }++    // cookie and num_containers+    if (length < 4) {+        return NULL;+    }+    uint32_t header;+    memcpy(&header, buf + length - 4, 4);  // header may be misaligned+    if ((header & 0x7FFF) != FROZEN_COOKIE) {+        return NULL;+    }+    int32_t num_containers = (header >> 15);+    if (num_containers > (1 << 16)) {+        // there are at most 65536 distinct 16-bit keys+        return NULL;+    }++    // typecodes, counts and keys+    if (length < 4 + (size_t)num_containers * (1 + 2 + 2)) {+        return NULL;+    }+    uint16_t *keys = (uint16_t *)(buf + length - 4 - num_containers * 5);+    uint16_t *counts = (uint16_t *)(buf + length - 4 - num_containers * 3);+    uint8_t *typecodes = (uint8_t *)(buf + length - 4 - num_containers * 1);++    // {bitset,array,run}_zone+    int32_t num_bitset_containers = 0;+    int32_t num_run_containers = 0;+    int32_t num_array_containers = 0;+    size_t bitset_zone_size = 0;+    size_t run_zone_size = 0;+    size_t array_zone_size = 0;+    for (int32_t i = 0; i < num_containers; i++) {+        switch (typecodes[i]) {+            case BITSET_CONTAINER_TYPE:+                num_bitset_containers++;+                bitset_zone_size +=+                    BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+                break;+            case RUN_CONTAINER_TYPE:+                num_run_containers++;+                run_zone_size += counts[i] * sizeof(rle16_t);+                break;+            case ARRAY_CONTAINER_TYPE:+                num_array_containers++;+                array_zone_size += (counts[i] + UINT32_C(1)) * sizeof(uint16_t);+                break;+            default:+                return NULL;+        }+    }+    if (length != bitset_zone_size + run_zone_size + array_zone_size ++                      5 * num_containers + 4) {+        return NULL;+    }+    uint64_t *bitset_zone = (uint64_t *)(buf);+    rle16_t *run_zone = (rle16_t *)(buf + bitset_zone_size);+    uint16_t *array_zone = (uint16_t *)(buf + bitset_zone_size + run_zone_size);++    size_t alloc_size = 0;+    alloc_size += sizeof(roaring_bitmap_t);+    alloc_size += num_containers * sizeof(container_t *);+    alloc_size += num_bitset_containers * sizeof(bitset_container_t);+    alloc_size += num_run_containers * sizeof(run_container_t);+    alloc_size += num_array_containers * sizeof(array_container_t);++    char *arena = (char *)roaring_malloc(alloc_size);+    if (arena == NULL) {+        return NULL;+    }++    roaring_bitmap_t *rb =+        (roaring_bitmap_t *)arena_alloc(&arena, sizeof(roaring_bitmap_t));+    rb->high_low_container.flags = ROARING_FLAG_FROZEN;+    rb->high_low_container.allocation_size = num_containers;+    rb->high_low_container.size = num_containers;+    rb->high_low_container.keys = (uint16_t *)keys;+    rb->high_low_container.typecodes = (uint8_t *)typecodes;+    rb->high_low_container.containers = (container_t **)arena_alloc(+        &arena, sizeof(container_t *) * num_containers);+    // Ensure offset of high_low_container.containers is known distance used in+    // C++ wrapper. sizeof(roaring_bitmap_t) is used as it is the size of the+    // only allocation that precedes high_low_container.containers. If this is+    // changed (new allocation or changed order), this offset will also need to+    // be changed in the C++ wrapper.+    assert(rb ==+           (roaring_bitmap_t *)((char *)rb->high_low_container.containers -+                                sizeof(roaring_bitmap_t)));+    for (int32_t i = 0; i < num_containers; i++) {+        switch (typecodes[i]) {+            case BITSET_CONTAINER_TYPE: {+                bitset_container_t *bitset = (bitset_container_t *)arena_alloc(+                    &arena, sizeof(bitset_container_t));+                bitset->words = bitset_zone;+                bitset->cardinality = counts[i] + UINT32_C(1);+                rb->high_low_container.containers[i] = bitset;+                bitset_zone += BITSET_CONTAINER_SIZE_IN_WORDS;+                break;+            }+            case RUN_CONTAINER_TYPE: {+                run_container_t *run = (run_container_t *)arena_alloc(+                    &arena, sizeof(run_container_t));+                run->capacity = counts[i];+                run->n_runs = counts[i];+                run->runs = run_zone;+                rb->high_low_container.containers[i] = run;+                run_zone += run->n_runs;+                break;+            }+            case ARRAY_CONTAINER_TYPE: {+                array_container_t *array = (array_container_t *)arena_alloc(+                    &arena, sizeof(array_container_t));+                array->capacity = counts[i] + UINT32_C(1);+                array->cardinality = counts[i] + UINT32_C(1);+                array->array = array_zone;+                rb->high_low_container.containers[i] = array;+                array_zone += counts[i] + UINT32_C(1);+                break;+            }+            default:+                roaring_free(rb);+                return NULL;+        }+    }++    return rb;+}++CROARING_ALLOW_UNALIGNED+roaring_bitmap_t *roaring_bitmap_portable_deserialize_frozen(const char *buf) {+#if CROARING_IS_BIG_ENDIAN+    // The portable format is little-endian on every host, and this function+    // uses the container payloads where they sit rather than converting them.+    // There is therefore no correct in-place view of them here: refuse rather+    // than hand back a bitmap that silently reads byte-swapped values. Use+    // roaring_bitmap_portable_deserialize_safe(), which converts as it copies.+    (void)buf;+    return NULL;+#else+    char *start_of_buf = (char *)buf;+    uint32_t cookie;+    int32_t num_containers;+    uint16_t *descriptive_headers;+    uint32_t *offset_headers = NULL;+    const char *run_flag_bitset = NULL;+    bool hasrun = false;++    // deserialize cookie+    memcpy(&cookie, buf, sizeof(uint32_t));+    buf += sizeof(uint32_t);+    if (cookie == SERIAL_COOKIE_NO_RUNCONTAINER) {+        memcpy(&num_containers, buf, sizeof(int32_t));+        buf += sizeof(int32_t);+        descriptive_headers = (uint16_t *)buf;+        buf += num_containers * 2 * sizeof(uint16_t);+        offset_headers = (uint32_t *)buf;+        buf += num_containers * sizeof(uint32_t);+    } else if ((cookie & 0xFFFF) == SERIAL_COOKIE) {+        num_containers = (cookie >> 16) + 1;+        hasrun = true;+        int32_t run_flag_bitset_size = (num_containers + 7) / 8;+        run_flag_bitset = buf;+        buf += run_flag_bitset_size;+        descriptive_headers = (uint16_t *)buf;+        buf += num_containers * 2 * sizeof(uint16_t);+        if (num_containers >= NO_OFFSET_THRESHOLD) {+            offset_headers = (uint32_t *)buf;+            buf += num_containers * sizeof(uint32_t);+        }+    } else {+        return NULL;+    }++    // calculate total size for allocation+    int32_t num_bitset_containers = 0;+    int32_t num_run_containers = 0;+    int32_t num_array_containers = 0;++    for (int32_t i = 0; i < num_containers; i++) {+        uint16_t tmp;+        memcpy(&tmp, descriptive_headers + 2 * i + 1, sizeof(tmp));+        uint32_t cardinality = tmp + 1;+        bool isbitmap = (cardinality > DEFAULT_MAX_SIZE);+        bool isrun = false;+        if (hasrun) {+            if ((run_flag_bitset[i / 8] & (1 << (i % 8))) != 0) {+                isbitmap = false;+                isrun = true;+            }+        }++        if (isbitmap) {+            num_bitset_containers++;+        } else if (isrun) {+            num_run_containers++;+        } else {+            num_array_containers++;+        }+    }++    size_t container_size = 0;+    container_size += num_bitset_containers * sizeof(bitset_container_t);+    container_size += num_run_containers * sizeof(run_container_t);+    container_size += num_array_containers * sizeof(array_container_t);++    size_t alloc_size = 0;+    alloc_size += sizeof(roaring_bitmap_t);+    alloc_size += num_containers * sizeof(container_t *);+    alloc_size += container_size;+    alloc_size += num_containers * sizeof(uint16_t);  // keys+    alloc_size += num_containers * sizeof(uint8_t);   // typecodes++    // allocate bitmap and construct containers+    char *arena = (char *)roaring_malloc(alloc_size);+    if (arena == NULL) {+        return NULL;+    }++    roaring_bitmap_t *rb =+        (roaring_bitmap_t *)arena_alloc(&arena, sizeof(roaring_bitmap_t));+    rb->high_low_container.flags = ROARING_FLAG_FROZEN;+    rb->high_low_container.allocation_size = num_containers;+    rb->high_low_container.size = num_containers;+    rb->high_low_container.containers = (container_t **)arena_alloc(+        &arena, sizeof(container_t *) * num_containers);++    // The keys and typecodes go after the containers, which are allocated in+    // the loop below: placed before them, they would misalign the containers.+    uint16_t *keys = (uint16_t *)(arena + container_size);+    uint8_t *typecodes = (uint8_t *)(keys + num_containers);++    rb->high_low_container.keys = keys;+    rb->high_low_container.typecodes = typecodes;++    for (int32_t i = 0; i < num_containers; i++) {+        uint16_t tmp;+        memcpy(&tmp, descriptive_headers + 2 * i + 1, sizeof(tmp));+        int32_t cardinality = tmp + 1;+        bool isbitmap = (cardinality > DEFAULT_MAX_SIZE);+        bool isrun = false;+        if (hasrun) {+            if ((run_flag_bitset[i / 8] & (1 << (i % 8))) != 0) {+                isbitmap = false;+                isrun = true;+            }+        }++        memcpy(&keys[i], descriptive_headers + 2 * i, sizeof(uint16_t));++        if (isbitmap) {+            typecodes[i] = BITSET_CONTAINER_TYPE;+            bitset_container_t *c = (bitset_container_t *)arena_alloc(+                &arena, sizeof(bitset_container_t));+            c->cardinality = cardinality;+            if (offset_headers != NULL) {+                c->words = (uint64_t *)(start_of_buf + offset_headers[i]);+            } else {+                c->words = (uint64_t *)buf;+                buf += BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+            }+            rb->high_low_container.containers[i] = c;+        } else if (isrun) {+            typecodes[i] = RUN_CONTAINER_TYPE;+            run_container_t *c =+                (run_container_t *)arena_alloc(&arena, sizeof(run_container_t));+            c->capacity = cardinality;+            uint16_t n_runs;+            if (offset_headers != NULL) {+                memcpy(&n_runs, start_of_buf + offset_headers[i],+                       sizeof(uint16_t));+                c->n_runs = n_runs;+                c->runs = (rle16_t *)(start_of_buf + offset_headers[i] ++                                      sizeof(uint16_t));+            } else {+                memcpy(&n_runs, buf, sizeof(uint16_t));+                c->n_runs = n_runs;+                buf += sizeof(uint16_t);+                c->runs = (rle16_t *)buf;+                buf += c->n_runs * sizeof(rle16_t);+            }+            rb->high_low_container.containers[i] = c;+        } else {+            typecodes[i] = ARRAY_CONTAINER_TYPE;+            array_container_t *c = (array_container_t *)arena_alloc(+                &arena, sizeof(array_container_t));+            c->cardinality = cardinality;+            c->capacity = cardinality;+            if (offset_headers != NULL) {+                c->array = (uint16_t *)(start_of_buf + offset_headers[i]);+            } else {+                c->array = (uint16_t *)buf;+                buf += cardinality * sizeof(uint16_t);+            }+            rb->high_low_container.containers[i] = c;+        }+    }++    return rb;+#endif+}++void roaring_bitmap_word_range_to_bitset(const roaring_bitmap_t *r,+                                         uint32_t first_word, uint32_t n_words,+                                         uint64_t *words) {+    memset(words, 0, (size_t)n_words * sizeof(uint64_t));+    if (n_words == 0) return;+    assert((uint64_t)first_word + n_words <= (UINT64_C(1) << 26));+    const uint64_t begin = (uint64_t)first_word * 64;+    const uint64_t end = begin + (uint64_t)n_words * 64;  // excluded+    const roaring_array_t *ra = &r->high_low_container;+    int32_t i = ra_get_index(ra, (uint16_t)(begin >> 16));+    if (i < 0) i = -i - 1;+    for (; i < ra->size; ++i) {+        const uint64_t chunk = (uint64_t)ra->keys[i] << 16;+        if (chunk >= end) break;+        // The window covers the words [lo, hi) of the container, which go+        // to out[0, hi - lo).+        const uint32_t lo =+            chunk < begin ? (uint32_t)((begin - chunk) / 64) : 0;+        const uint32_t hi =+            chunk + 65536 > end ? (uint32_t)((end - chunk) / 64) : 1024;+        uint64_t *out = words + (chunk / 64 + lo - first_word);+        const uint32_t vlo = lo * 64, vhi = hi * 64;  // values [vlo, vhi)+        uint8_t type = ra->typecodes[i];+        const container_t *c =+            container_unwrap_shared(ra->containers[i], &type);+        switch (type) {+            case BITSET_CONTAINER_TYPE:+                memcpy(out, const_CAST_bitset(c)->words + lo,+                       (size_t)(hi - lo) * sizeof(uint64_t));+                break;+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *src = const_CAST_array(c);+                int32_t k = 0;+                if (vlo > 0) {+                    k = binarySearch(src->array, src->cardinality,+                                     (uint16_t)vlo);+                    if (k < 0) k = -k - 1;+                }+                for (; k < src->cardinality && src->array[k] < vhi; ++k) {+                    const uint32_t v = src->array[k] - vlo;+                    out[v / 64] |= UINT64_C(1) << (v % 64);+                }+            } break;+            case RUN_CONTAINER_TYPE: {+                const run_container_t *src = const_CAST_run(c);+                int32_t k = 0;+                if (vlo > 0) {+                    // The last run starting at or before vlo may reach it.+                    k = interleavedBinarySearch(src->runs, src->n_runs,+                                                (uint16_t)vlo);+                    if (k < 0) k = -k - 2;+                    if (k < 0) k = 0;+                }+                for (; k < src->n_runs && src->runs[k].value < vhi; ++k) {+                    uint32_t s = src->runs[k].value;+                    uint32_t e = s + src->runs[k].length + 1;  // excluded+                    if (e <= vlo) continue;+                    if (s < vlo) s = vlo;+                    if (e > vhi) e = vhi;+                    bitset_set_range(out, s - vlo, e - vlo);+                }+            } break;+            default:+                roaring_unreachable;+        }+    }+}++bool roaring_bitmap_to_bitset(const roaring_bitmap_t *r, bitset_t *bitset) {+    uint32_t max_value = roaring_bitmap_maximum(r);+    size_t new_array_size = (size_t)(max_value / 64 + 1);+    bool resize_ok = bitset_resize(bitset, new_array_size, true);+    if (!resize_ok) {+        return false;+    }+    const roaring_array_t *ra = &r->high_low_container;+    for (int i = 0; i < ra->size; ++i) {+        uint64_t *words = bitset->array + (ra->keys[i] << 10);+        uint8_t type = ra->typecodes[i];+        const container_t *c = ra->containers[i];+        if (type == SHARED_CONTAINER_TYPE) {+            c = container_unwrap_shared(c, &type);+        }+        switch (type) {+            case BITSET_CONTAINER_TYPE: {+                size_t max_word_index = new_array_size - (ra->keys[i] << 10);+                if (max_word_index > 1024) {+                    max_word_index = 1024;+                }+                const bitset_container_t *src = const_CAST_bitset(c);+                memcpy(words, src->words, max_word_index * sizeof(uint64_t));+            } break;+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *src = const_CAST_array(c);+                bitset_set_list(words, src->array, src->cardinality);+            } break;+            case RUN_CONTAINER_TYPE: {+                const run_container_t *src = const_CAST_run(c);+                for (int32_t rlepos = 0; rlepos < src->n_runs; ++rlepos) {+                    rle16_t rle = src->runs[rlepos];+                    bitset_set_lenrange(words, rle.value, rle.length);+                }+            } break;+            default:+                roaring_unreachable;+        }+    }+    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring {+#endif
+ cbits/CRoaring/src/roaring64.c view
@@ -0,0 +1,3773 @@+#include <assert.h>+#include <stdarg.h>+#include <stddef.h>+#include <stdint.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/art/art.h>+#include <roaring/isadetection.h>+#include <roaring/portability.h>+#include <roaring/roaring64.h>++// For serialization / deserialization+#include <roaring/containers/array.h>+#include <roaring/containers/bitset.h>+#include <roaring/containers/run.h>+#include <roaring/roaring.h>+#include <roaring/roaring_array.h>+// containers.h last to avoid conflict with ROARING_CONTAINER_T.+#include <roaring/containers/containers.h>++#define CROARING_ALIGN_BUF(buf, alignment)          \+    (char *)(((uintptr_t)(buf) + ((alignment)-1)) & \+             (ptrdiff_t)(~((alignment)-1)))++#define CROARING_BITSET_ALIGNMENT 64++#ifdef __cplusplus+using namespace ::roaring::internal;++extern "C" {+namespace roaring {+namespace api {+#endif++// TODO: Copy on write.+// TODO: Error on failed allocation.++typedef struct roaring64_bitmap_s {+    art_t art;+    uint8_t flags;+    // The container slots. Every slot at or above end is free; the free slots+    // below end (the holes) are kept in a min-heap, so that the lowest free+    // slot, first_free, is found without scanning the slots. first_free is+    // the top of the heap, or end if there are no holes (capacity if the+    // slots are all taken).+    uint64_t first_free;+    uint64_t end;+    uint64_t *holes;  // min-heap of num_holes indices+    uint64_t num_holes;+    uint64_t holes_capacity;+    uint64_t capacity;+    container_t **containers;+    // Parallel to containers[]. Live slots (non-NULL pointers) have the+    // matching typecode; NULL slots are skipped and their typecodes ignored.+    uint8_t *typecodes;+} roaring64_bitmap_t;++// Leaf type of the ART used to keep the high 48 bits of each entry.+// Low 8 bits: typecode+// High 56 bits: container index+typedef roaring64_leaf_t leaf_t;++// Iterator struct to hold iteration state.+typedef struct roaring64_iterator_s {+    // The order here is deliberate: everything `roaring64_iterator_advance`+    // touches per value is packed into the first 64 bytes, and `art_it` --+    // 136 bytes, of which only `art_it.value` is read per value -- is last.+    // Putting `art_it` earlier pushes the rest past the first cache line and+    // costs ~18% on a scalar iteration loop.++    // Must stay first, and must stay a `roaring64_iterator_public_t`: the+    // inline `roaring64_iterator_value` / `roaring64_iterator_has_value` in+    // the public header reach these two members by converting a+    // `roaring64_iterator_t *` to a pointer to its initial member.+    roaring64_iterator_public_t pub;++    uint64_t high48;  // Key that art_it points to.+    roaring_container_iterator_t container_it;++    // Forward-iteration cache for bitset containers. `container_iterator_next`+    // recomputes the word index from `container_it.index`, reloads the word+    // and re-masks it for every value; keeping the remaining bits of the+    // current word here turns that into a `tzcnt` and a `blsr`. Array and run+    // containers already advance by a single increment and gain nothing from+    // a cache, so they stay on the ordinary path.+    //+    // Every field is a pure function of the ART position and+    // `container_it.index`, so the cache describes where the iterator is+    // exactly when both still match what it was built from. `fast_type` is+    // BITSET_CONTAINER_TYPE, or 0 when there is no usable cache.+    uint32_t fast_wordindex;+    const art_val_t *fast_art_value;+    const uint64_t *fast_words;+    uint64_t fast_word;+    int32_t fast_index;+    uint8_t fast_type;++    // If has_value is false, then the iterator is saturated. This field+    // indicates the direction of saturation. If true, there are no more values+    // in the forward direction. If false, there are no more values in the+    // backward direction.+    bool saturated_forward;++    const roaring64_bitmap_t *r;+    art_iterator_t art_it;+} roaring64_iterator_t;++static inline bool is_frozen64(const roaring64_bitmap_t *r) {+    return r->flags & ROARING_FLAG_FROZEN;+}++static inline bool is_frozen_art64(const roaring64_bitmap_t *r) {+    return r->flags & ROARING_FLAG_FROZEN_ART;+}++typedef union {+    bitset_container_t bitset;+    array_container_t array;+    run_container_t run;+} frozen_container_header_t;++static void *roaring64_arena_alloc(char **arena, size_t num_bytes) {+    char *res = *arena;+    *arena += num_bytes;+    return res;+}++static char *roaring64_arena_pad(char *cursor, const char *base,+                                 size_t alignment) {+    uint64_t off = (uint64_t)(cursor - base);+    uint64_t aligned = (off + alignment - 1) & ~(uint64_t)(alignment - 1);+    return (char *)base + aligned;+}++// Splits the given uint64 key into high 48 bit and low 16 bit components.+// Expects high48_out to be of length ART_KEY_BYTES.+static inline uint16_t split_key(uint64_t key, uint8_t high48_out[]) {+    uint64_t tmp = croaring_htobe64(key);+    memcpy(high48_out, (uint8_t *)(&tmp), ART_KEY_BYTES);+    return (uint16_t)key;+}++// Recombines the high 48 bit and low 16 bit components into a uint64 key.+// Expects high48_out to be of length ART_KEY_BYTES.+static inline uint64_t combine_key(const uint8_t high48[], uint16_t low16) {+    uint64_t result = 0;+    memcpy((uint8_t *)(&result), high48, ART_KEY_BYTES);+    return croaring_be64toh(result) | low16;+}++static inline uint64_t minimum(uint64_t a, uint64_t b) {+    return (a < b) ? a : b;+}++static inline leaf_t create_leaf(uint64_t container_index, uint8_t typecode) {+    return (container_index << 8) | typecode;+}++static inline uint8_t get_typecode(leaf_t leaf) { return (uint8_t)leaf; }++static inline uint64_t get_index(leaf_t leaf) { return leaf >> 8; }++static inline container_t *get_container(const roaring64_bitmap_t *r,+                                         leaf_t leaf) {+    return r->containers[get_index(leaf)];+}++// Writes the pointer and its typecode together so they cannot drift.+static inline void set_container_at(roaring64_bitmap_t *r, uint64_t index,+                                    container_t *container, uint8_t typecode) {+    r->containers[index] = container;+    r->typecodes[index] = typecode;+}++// Replaces the container of `leaf` with the given container. Returns the+// modified leaf for convenience.+static inline leaf_t replace_container(roaring64_bitmap_t *r, leaf_t *leaf,+                                       container_t *container,+                                       uint8_t typecode) {+    uint64_t index = get_index(*leaf);+    set_container_at(r, index, container, typecode);+    *leaf = create_leaf(index, typecode);+    return *leaf;+}++/**+ * Grows the array of container pointers (and the parallel typecode array) to+ * `new_capacity` entries. Does nothing if `new_capacity` is not larger than+ * the current capacity.+ */+static void grow_containers_to(roaring64_bitmap_t *r, uint64_t new_capacity) {+    if (new_capacity <= r->capacity) {+        return;+    }+    uint64_t increase = new_capacity - r->capacity;+    r->containers = (container_t **)roaring_realloc(+        r->containers, new_capacity * sizeof(container_t *));+    memset(r->containers + r->capacity, 0, increase * sizeof(container_t *));+    r->typecodes = (uint8_t *)roaring_realloc(r->typecodes,+                                              new_capacity * sizeof(uint8_t));+    r->capacity = new_capacity;+}++/**+ * Extends the array of container pointers (and the parallel typecode array).+ */+static void extend_containers(roaring64_bitmap_t *r) {+    uint64_t size = r->first_free;+    if (size < r->capacity) {+        return;+    }+    uint64_t new_capacity;+    if (r->capacity == 0) {+        new_capacity = 2;+    } else if (r->capacity < 1024) {+        new_capacity = 2 * r->capacity;+    } else {+        new_capacity = 5 * r->capacity / 4;+    }+    grow_containers_to(r, new_capacity);+}++/**+ * Reserves room for `num_containers` containers in the ART and in the+ * container array, so that a result of known maximum size can be built+ * without repeatedly reallocating both. Intended for freshly created bitmaps.+ */+static void reserve_containers(roaring64_bitmap_t *r, uint64_t num_containers) {+    art_reserve(&r->art, num_containers);+    grow_containers_to(r, num_containers);+}++// The heap of holes: holes[0] is the smallest index, and every index is+// smaller than its children, holes[2i+1] and holes[2i+2].+static void holes_push(roaring64_bitmap_t *r, uint64_t index) {+    if (r->num_holes == r->holes_capacity) {+        uint64_t new_capacity =+            r->holes_capacity < 16 ? 16 : 2 * r->holes_capacity;+        r->holes = (uint64_t *)roaring_realloc(r->holes,+                                               new_capacity * sizeof(uint64_t));+        r->holes_capacity = new_capacity;+    }+    uint64_t i = r->num_holes++;+    while (i > 0) {+        uint64_t parent = (i - 1) / 2;+        if (r->holes[parent] <= index) {+            break;+        }+        r->holes[i] = r->holes[parent];+        i = parent;+    }+    r->holes[i] = index;+}++static void holes_pop(roaring64_bitmap_t *r) {+    uint64_t last = r->holes[--r->num_holes];+    uint64_t n = r->num_holes;+    uint64_t i = 0;+    while (2 * i + 1 < n) {+        uint64_t child = 2 * i + 1;+        if (child + 1 < n && r->holes[child + 1] < r->holes[child]) {+            child++;+        }+        if (last <= r->holes[child]) {+            break;+        }+        r->holes[i] = r->holes[child];+        i = child;+    }+    if (n > 0) {+        r->holes[i] = last;+    }+}++// Forgets every hole (the slots below end are then all taken).+static void holes_clear(roaring64_bitmap_t *r) { r->num_holes = 0; }++static inline void update_first_free(roaring64_bitmap_t *r) {+    r->first_free = r->num_holes > 0 ? r->holes[0] : r->end;+}++// Takes the lowest free slot and returns its index. The caller fills it.+static uint64_t allocate_index(roaring64_bitmap_t *r) {+    uint64_t first_free = r->first_free;+    if (first_free == r->capacity) {+        extend_containers(r);+    }+    if (r->num_holes > 0) {+        holes_pop(r);  // first_free is the top of the heap+    } else {+        r->end = first_free + 1;  // first_free == end+    }+    update_first_free(r);+    return first_free;+}++static leaf_t add_container(roaring64_bitmap_t *r, container_t *container,+                            uint8_t typecode) {+    uint64_t index = allocate_index(r);+    set_container_at(r, index, container, typecode);+    return create_leaf(index, typecode);+}++static void ensure_container_capacity(roaring64_bitmap_t *r, uint64_t extra) {+    uint64_t needed = r->first_free + extra;+    if (needed <= r->capacity) {+        return;+    }+    uint64_t new_capacity = r->capacity;+    if (new_capacity == 0) {+        new_capacity = 2;+    }+    while (new_capacity < needed) {+        uint64_t grown;+        if (new_capacity < 1024) {+            grown = 2 * new_capacity;+        } else {+            grown = new_capacity + new_capacity / 4;+        }+        if (grown <= new_capacity) {+            new_capacity = needed;+            break;+        }+        new_capacity = grown;+    }+    uint64_t increase = new_capacity - r->capacity;+    r->containers = (container_t **)roaring_realloc(+        r->containers, new_capacity * sizeof(container_t *));+    memset(r->containers + r->capacity, 0, increase * sizeof(container_t *));+    r->typecodes = (uint8_t *)roaring_realloc(r->typecodes,+                                              new_capacity * sizeof(uint8_t));+    r->capacity = new_capacity;+}++// Frees a slot (it does not free the container in it).+static void free_index(roaring64_bitmap_t *r, uint64_t index) {+    r->containers[index] = NULL;+    if (index + 1 == r->end) {+        // The holes, all below index, stay below end.+        r->end = index;+    } else {+        holes_push(r, index);+    }+    update_first_free(r);+}++static void remove_container(roaring64_bitmap_t *r, leaf_t leaf) {+    free_index(r, get_index(leaf));+}++// Copies the container referenced by `leaf` from `r1` to `r2`.+static inline leaf_t copy_leaf_container(const roaring64_bitmap_t *r1,+                                         roaring64_bitmap_t *r2, leaf_t leaf) {+    uint8_t typecode = get_typecode(leaf);+    container_t *container = container_clone(get_container(r1, leaf), typecode);+    return add_container(r2, container, typecode);+}++static inline int compare_high48(art_key_chunk_t key1[],+                                 art_key_chunk_t key2[]) {+    return art_compare_keys(key1, key2);+}++// Cache the current container so advance() need not re-read the ART leaf,+// reload the container pointer, or (for bitsets) re-mask the current word.+// Called after any positioning that sets container_it.index.+// fast_type is 0 when there is no usable cache (run containers, exhausted).+CROARING_ALLOW_UNALIGNED+static inline void roaring64_iterator_prime(roaring64_iterator_t *it) {+    it->fast_type = 0;+    if (it->art_it.value == NULL) {+        return;+    }+    leaf_t leaf = (leaf_t)*it->art_it.value;+    if (get_typecode(leaf) != BITSET_CONTAINER_TYPE) {+        return;+    }+    const bitset_container_t *bc =+        const_CAST_bitset(get_container(it->r, leaf));+    int32_t index = it->container_it.index;+    uint32_t wordindex = (uint32_t)index >> 6;+    uint32_t bit = (uint32_t)index & 63u;+    uint64_t word = bc->words[wordindex];+    // Bits strictly after the current value in this word. A shift of 64 is+    // undefined, so the last bit of a word is a special case.+    it->fast_word =+        (bit == 63u) ? UINT64_C(0) : (word & (UINT64_MAX << (bit + 1)));+    it->fast_words = bc->words;+    it->fast_wordindex = wordindex;+    it->fast_index = index;+    it->fast_art_value = it->art_it.value;+    it->fast_type = BITSET_CONTAINER_TYPE;+}++static inline bool roaring64_iterator_init_at_leaf_first(+    roaring64_iterator_t *it) {+    it->high48 = combine_key(it->art_it.key, 0);+    leaf_t leaf = (leaf_t)*it->art_it.value;+    uint16_t low16 = 0;+    it->container_it = container_init_iterator(get_container(it->r, leaf),+                                               get_typecode(leaf), &low16);+    it->pub.value = it->high48 | low16;+    it->pub.has_value = true;+    roaring64_iterator_prime(it);+    return true;+}++static inline bool roaring64_iterator_init_at_leaf_last(+    roaring64_iterator_t *it) {+    it->high48 = combine_key(it->art_it.key, 0);+    leaf_t leaf = (leaf_t)*it->art_it.value;+    uint16_t low16 = 0;+    it->container_it = container_init_iterator_last(get_container(it->r, leaf),+                                                    get_typecode(leaf), &low16);+    it->pub.value = it->high48 | low16;+    it->pub.has_value = true;+    roaring64_iterator_prime(it);+    return true;+}++static inline roaring64_iterator_t *roaring64_iterator_init_at(+    const roaring64_bitmap_t *r, roaring64_iterator_t *it, bool first) {+    it->r = r;+    it->art_it = art_init_iterator((art_t *)&r->art, first);+    it->pub.has_value = it->art_it.value != NULL;+    if (it->pub.has_value) {+        if (first) {+            roaring64_iterator_init_at_leaf_first(it);+        } else {+            roaring64_iterator_init_at_leaf_last(it);+        }+    } else {+        it->saturated_forward = first;+        it->fast_type = 0;+    }+    return it;+}++roaring64_bitmap_t *roaring64_bitmap_create(void) {+    roaring64_bitmap_t *r =+        (roaring64_bitmap_t *)roaring_malloc(sizeof(roaring64_bitmap_t));+    art_init_cleared(&r->art);+    r->flags = 0;+    r->capacity = 0;+    r->first_free = 0;+    r->end = 0;+    r->holes = NULL;+    r->num_holes = 0;+    r->holes_capacity = 0;+    r->containers = NULL;+    r->typecodes = NULL;+    return r;+}++void roaring64_bitmap_free(roaring64_bitmap_t *r) {+    if (!r) {+        return;+    }+    if (is_frozen64(r)) {+        // Headers, containers[], and typecodes[] live in the same allocation+        // as `r`. Payloads alias a caller buffer.+        if (!is_frozen_art64(r)) {+            art_free(&r->art);+        }+        roaring_free(r);+        return;+    }+    art_iterator_t it = art_init_iterator(&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        container_free(get_container(r, leaf), get_typecode(leaf));+        art_iterator_next(&it);+    }+    art_free(&r->art);+    roaring_free(r->containers);+    roaring_free(r->typecodes);+    roaring_free(r->holes);+    roaring_free(r);+}++roaring64_bitmap_t *roaring64_bitmap_copy(const roaring64_bitmap_t *r) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();+    reserve_containers(result, art_num_leaves(&r->art));++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t result_typecode = get_typecode(leaf);+        container_t *result_container =+            container_clone(get_container(r, leaf), result_typecode);+        leaf_t result_leaf =+            add_container(result, result_container, result_typecode);+        art_insert(&result->art, it.key, (art_val_t)result_leaf);+        art_iterator_next(&it);+    }+    return result;+}++void roaring64_bitmap_overwrite(roaring64_bitmap_t *dest,+                                const roaring64_bitmap_t *src) {+    if (dest == src) {+        return;+    }++    // Free dest's containers.+    art_iterator_t it = art_init_iterator(&dest->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        container_free(get_container(dest, leaf), get_typecode(leaf));+        art_iterator_next(&it);+    }+    art_free(&dest->art);++    // Reinitialize dest.+    art_init_cleared(&dest->art);+    dest->flags = 0;+    dest->first_free = 0;+    dest->end = 0;+    holes_clear(dest);+    if (dest->capacity > 0) {+        memset(dest->containers, 0,+               sizeof(dest->containers[0]) * dest->capacity);+    }++    // Copy src's containers into dest.+    it = art_init_iterator((art_t *)&src->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);+        container_t *container =+            container_clone(get_container(src, leaf), typecode);+        leaf_t dest_leaf = add_container(dest, container, typecode);+        art_insert(&dest->art, it.key, (art_val_t)dest_leaf);+        art_iterator_next(&it);+    }+}++/**+ * Steal src's containers into dst at high_bits, copying those still shared.+ *+ * After calling this function, the original bitmap will be empty, and the+ * returned bitmap will contain all the values from the original bitmap.+ */+static void move_from_roaring32_offset(roaring64_bitmap_t *dst,+                                       roaring_bitmap_t *src,+                                       uint32_t high_bits) {+    uint64_t key_base = ((uint64_t)high_bits) << 32;+    uint32_t r32_size = ra_get_size(&src->high_low_container);+    ensure_container_capacity(dst, r32_size);+    for (uint32_t i = 0; i < r32_size; ++i) {+        uint16_t key = ra_get_key_at_index(&src->high_low_container, i);+        uint8_t typecode;+        container_t *container = ra_get_container_at_index(+            &src->high_low_container, (uint16_t)i, &typecode);+        container = get_writable_copy_if_shared(container, &typecode);++        uint8_t high48[ART_KEY_BYTES];+        uint64_t high48_bits = key_base | ((uint64_t)key << 16);+        split_key(high48_bits, high48);+        leaf_t leaf = add_container(dst, container, typecode);+        art_insert(&dst->art, high48, (art_val_t)leaf);+    }+    // We stole all the containers, so leave behind a size of zero+    src->high_low_container.size = 0;+}++roaring64_bitmap_t *roaring64_bitmap_move_from_roaring32(+    roaring_bitmap_t *bitmap32) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();++    move_from_roaring32_offset(result, bitmap32, 0);++    return result;+}++roaring64_bitmap_t *roaring64_bitmap_from_range(uint64_t min, uint64_t max,+                                                uint64_t step) {+    if (step == 0 || max <= min) {+        return NULL;+    }+    roaring64_bitmap_t *r = roaring64_bitmap_create();+    if (step >= (1 << 16)) {+        // Only one value per container.+        for (uint64_t value = min; value < max; value += step) {+            roaring64_bitmap_add(r, value);+            if (value > UINT64_MAX - step) {+                break;+            }+        }+        return r;+    }+    do {+        uint64_t high_bits = min & 0xFFFFFFFFFFFF0000;+        uint16_t container_min = min & 0xFFFF;+        uint32_t container_max = (uint32_t)minimum(max - high_bits, 1 << 16);++        uint8_t typecode;+        container_t *container = container_from_range(+            &typecode, container_min, container_max, (uint16_t)step);++        uint8_t high48[ART_KEY_BYTES];+        split_key(min, high48);+        leaf_t leaf = add_container(r, container, typecode);+        art_insert(&r->art, high48, (art_val_t)leaf);++        uint64_t gap = container_max - container_min + step - 1;+        uint64_t increment = gap - (gap % step);+        if (min > UINT64_MAX - increment) {+            break;+        }+        min += increment;+    } while (min < max);+    return r;+}++roaring64_bitmap_t *roaring64_bitmap_of_ptr(size_t n_args,+                                            const uint64_t *vals) {+    roaring64_bitmap_t *r = roaring64_bitmap_create();+    roaring64_bitmap_add_many(r, n_args, vals);+    return r;+}++static inline leaf_t *containerptr_roaring64_bitmap_add(roaring64_bitmap_t *r,+                                                        uint8_t *high48,+                                                        uint16_t low16,+                                                        leaf_t *leaf) {+    if (leaf != NULL) {+        uint8_t typecode = get_typecode(*leaf);+        container_t *container = get_container(r, *leaf);+        uint8_t typecode2;+        container_t *container2 =+            container_add(container, low16, typecode, &typecode2);+        if (container2 != container) {+            container_free(container, typecode);+            replace_container(r, leaf, container2, typecode2);+        }+        return leaf;+    } else {+        array_container_t *ac = array_container_create();+        uint8_t typecode;+        container_t *container =+            container_add(ac, low16, ARRAY_CONTAINER_TYPE, &typecode);+        assert(ac == container);+        leaf_t new_leaf = add_container(r, container, typecode);+        return (leaf_t *)art_insert(&r->art, high48, (art_val_t)new_leaf);+    }+}++void roaring64_bitmap_add(roaring64_bitmap_t *r, uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    leaf_t *leaf = (leaf_t *)art_find(&r->art, high48);+    containerptr_roaring64_bitmap_add(r, high48, low16, leaf);+}++bool roaring64_bitmap_add_checked(roaring64_bitmap_t *r, uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    leaf_t *leaf = (leaf_t *)art_find(&r->art, high48);++    int old_cardinality = 0;+    if (leaf != NULL) {+        old_cardinality = container_get_cardinality(get_container(r, *leaf),+                                                    get_typecode(*leaf));+    }+    leaf = containerptr_roaring64_bitmap_add(r, high48, low16, leaf);+    int new_cardinality =+        container_get_cardinality(get_container(r, *leaf), get_typecode(*leaf));+    return old_cardinality != new_cardinality;+}++void roaring64_bitmap_add_bulk(roaring64_bitmap_t *r,+                               roaring64_bulk_context_t *context,+                               uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    leaf_t *leaf = context->leaf;+    if (leaf != NULL && compare_high48(context->high_bytes, high48) == 0) {+        // We're at a container with the correct high bits.+        uint8_t typecode1 = get_typecode(*leaf);+        container_t *container1 = get_container(r, *leaf);+        uint8_t typecode2;+        container_t *container2 =+            container_add(container1, low16, typecode1, &typecode2);+        if (container2 != container1) {+            container_free(container1, typecode1);+            replace_container(r, leaf, container2, typecode2);+        }+    } else {+        // We're not positioned anywhere yet or the high bits of the key+        // differ.+        leaf = (leaf_t *)art_find(&r->art, high48);+        context->leaf =+            containerptr_roaring64_bitmap_add(r, high48, low16, leaf);+        memcpy(context->high_bytes, high48, ART_KEY_BYTES);+    }+}++void roaring64_bitmap_add_many(roaring64_bitmap_t *r, size_t n_args,+                               const uint64_t *vals) {+    if (n_args == 0) {+        return;+    }+    const uint64_t *end = vals + n_args;+    roaring64_bulk_context_t context = CROARING_ZERO_INITIALIZER;+    for (const uint64_t *current_val = vals; current_val != end;+         current_val++) {+        roaring64_bitmap_add_bulk(r, &context, *current_val);+    }+}++static inline void add_range_closed_at(roaring64_bitmap_t *r, art_t *art,+                                       uint8_t *high48, uint16_t min,+                                       uint16_t max) {+    leaf_t *leaf = (leaf_t *)art_find(art, high48);+    if (leaf != NULL) {+        uint8_t typecode1 = get_typecode(*leaf);+        container_t *container1 = get_container(r, *leaf);+        uint8_t typecode2;+        container_t *container2 =+            container_add_range(container1, typecode1, min, max, &typecode2);+        if (container2 != container1) {+            container_free(container1, typecode1);+            replace_container(r, leaf, container2, typecode2);+        }+        return;+    }+    uint8_t typecode;+    // container_add_range is inclusive, but `container_range_of_ones` is+    // exclusive.+    container_t *container = container_range_of_ones(min, max + 1, &typecode);+    leaf_t new_leaf = add_container(r, container, typecode);+    art_insert(art, high48, (art_val_t)new_leaf);+}++void roaring64_bitmap_add_range(roaring64_bitmap_t *r, uint64_t min,+                                uint64_t max) {+    if (min >= max) {+        return;+    }+    roaring64_bitmap_add_range_closed(r, min, max - 1);+}++void roaring64_bitmap_add_range_closed(roaring64_bitmap_t *r, uint64_t min,+                                       uint64_t max) {+    if (min > max) {+        return;+    }++    art_t *art = &r->art;+    uint8_t min_high48[ART_KEY_BYTES];+    uint16_t min_low16 = split_key(min, min_high48);+    uint8_t max_high48[ART_KEY_BYTES];+    uint16_t max_low16 = split_key(max, max_high48);+    if (compare_high48(min_high48, max_high48) == 0) {+        // Only populate range within one container.+        add_range_closed_at(r, art, min_high48, min_low16, max_low16);+        return;+    }++    // Populate a range across containers. Fill intermediate containers+    // entirely.+    add_range_closed_at(r, art, min_high48, min_low16, 0xffff);+    uint64_t min_high_bits = min >> 16;+    uint64_t max_high_bits = max >> 16;+    for (uint64_t current = min_high_bits + 1; current < max_high_bits;+         ++current) {+        uint8_t current_high48[ART_KEY_BYTES];+        split_key(current << 16, current_high48);+        add_range_closed_at(r, art, current_high48, 0, 0xffff);+    }+    add_range_closed_at(r, art, max_high48, 0, max_low16);+}++bool roaring64_bitmap_contains(const roaring64_bitmap_t *r, uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    leaf_t *leaf = (leaf_t *)art_find(&r->art, high48);+    if (leaf != NULL) {+        return container_contains(get_container(r, *leaf), low16,+                                  get_typecode(*leaf));+    }+    return false;+}++bool roaring64_bitmap_contains_range(const roaring64_bitmap_t *r, uint64_t min,+                                     uint64_t max) {+    if (min >= max) {+        return true;+    }+    return roaring64_bitmap_contains_range_closed(r, min, max - 1);+}++bool roaring64_bitmap_contains_range_closed(const roaring64_bitmap_t *r,+                                            uint64_t min, uint64_t max) {+    if (min > max) {+        return true;+    }++    uint8_t min_high48[ART_KEY_BYTES];+    uint16_t min_low16 = split_key(min, min_high48);+    uint8_t max_high48[ART_KEY_BYTES];+    uint16_t max_low16 = split_key(max, max_high48);+    uint64_t max_high48_bits = max & 0xFFFFFFFFFFFF0000;++    art_iterator_t it = art_lower_bound((art_t *)&r->art, min_high48);+    if (it.value == NULL || combine_key(it.key, 0) > min) {+        return false;+    }+    uint64_t prev_high48_bits = min & 0xFFFFFFFFFFFF0000;+    while (it.value != NULL) {+        uint64_t current_high48_bits = combine_key(it.key, 0);+        if (current_high48_bits > max_high48_bits) {+            // We've passed the end of the range. The range is contained only+            // if the container holding max was checked.+            break;+        }+        if (current_high48_bits - prev_high48_bits > 0x10000) {+            // There is a gap in the iterator that falls in the range.+            return false;+        }++        leaf_t leaf = (leaf_t)*it.value;+        uint32_t container_min = 0;+        if (compare_high48(it.key, min_high48) == 0) {+            container_min = min_low16;+        }+        uint32_t container_max = 0xFFFF + 1;  // Exclusive+        if (compare_high48(it.key, max_high48) == 0) {+            container_max = (uint32_t)max_low16 + 1;+        }++        // For the first and last containers we use container_contains_range,+        // for the intermediate containers we can use container_is_full.+        if (container_min == 0 && container_max == 0xFFFF + 1) {+            if (!container_is_full(get_container(r, leaf),+                                   get_typecode(leaf))) {+                return false;+            }+        } else if (!container_contains_range(get_container(r, leaf),+                                             container_min, container_max,+                                             get_typecode(leaf))) {+            return false;+        }+        prev_high48_bits = current_high48_bits;+        art_iterator_next(&it);+    }+    return prev_high48_bits == max_high48_bits;+}++bool roaring64_bitmap_contains_bulk(const roaring64_bitmap_t *r,+                                    roaring64_bulk_context_t *context,+                                    uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);++    if (context->leaf == NULL ||+        art_compare_keys(context->high_bytes, high48) != 0) {+        // We're not positioned anywhere yet or the high bits of the key+        // differ.+        leaf_t *leaf = (leaf_t *)art_find(&r->art, high48);+        if (leaf == NULL) {+            return false;+        }+        context->leaf = leaf;+        memcpy(context->high_bytes, high48, ART_KEY_BYTES);+    }+    return container_contains(get_container(r, *context->leaf), low16,+                              get_typecode(*context->leaf));+}++bool roaring64_bitmap_select(const roaring64_bitmap_t *r, uint64_t rank,+                             uint64_t *element) {+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    uint64_t start_rank = 0;+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint64_t cardinality = container_get_cardinality(get_container(r, leaf),+                                                         get_typecode(leaf));+        if (start_rank + cardinality > rank) {+            uint32_t uint32_start = 0;+            uint32_t uint32_rank = rank - start_rank;+            uint32_t uint32_element = 0;+            if (container_select(get_container(r, leaf), get_typecode(leaf),+                                 &uint32_start, uint32_rank, &uint32_element)) {+                *element = combine_key(it.key, (uint16_t)uint32_element);+                return true;+            }+            return false;+        }+        start_rank += cardinality;+        art_iterator_next(&it);+    }+    return false;+}++uint64_t roaring64_bitmap_rank(const roaring64_bitmap_t *r, uint64_t val) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    uint64_t rank = 0;+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        int compare_result = compare_high48(it.key, high48);+        if (compare_result < 0) {+            rank += container_get_cardinality(get_container(r, leaf),+                                              get_typecode(leaf));+        } else if (compare_result == 0) {+            return rank + container_rank(get_container(r, leaf),+                                         get_typecode(leaf), low16);+        } else {+            return rank;+        }+        art_iterator_next(&it);+    }+    return rank;+}++bool roaring64_bitmap_get_index(const roaring64_bitmap_t *r, uint64_t val,+                                uint64_t *out_index) {+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    uint64_t index = 0;+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        int compare_result = compare_high48(it.key, high48);+        if (compare_result < 0) {+            index += container_get_cardinality(get_container(r, leaf),+                                               get_typecode(leaf));+        } else if (compare_result == 0) {+            int index16 = container_get_index(get_container(r, leaf),+                                              get_typecode(leaf), low16);+            if (index16 < 0) {+                return false;+            }+            *out_index = index + index16;+            return true;+        } else {+            return false;+        }+        art_iterator_next(&it);+    }+    return false;+}++// Returns true if a container was removed.+static inline bool containerptr_roaring64_bitmap_remove(roaring64_bitmap_t *r,+                                                        uint8_t *high48,+                                                        uint16_t low16,+                                                        leaf_t *leaf) {+    if (leaf == NULL) {+        return false;+    }++    uint8_t typecode = get_typecode(*leaf);+    container_t *container = get_container(r, *leaf);+    uint8_t typecode2;+    container_t *container2 =+        container_remove(container, low16, typecode, &typecode2);+    if (container2 != container) {+        container_free(container, typecode);+        replace_container(r, leaf, container2, typecode2);+    }+    if (!container_nonzero_cardinality(container2, typecode2)) {+        container_free(container2, typecode2);+        bool erased = art_erase(&r->art, high48, (art_val_t *)leaf);+        assert(erased);+        (void)erased;+        remove_container(r, *leaf);+        return true;+    }+    return false;+}++void roaring64_bitmap_remove(roaring64_bitmap_t *r, uint64_t val) {+    art_t *art = &r->art;+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);++    leaf_t *leaf = (leaf_t *)art_find(art, high48);+    containerptr_roaring64_bitmap_remove(r, high48, low16, leaf);+}++bool roaring64_bitmap_remove_checked(roaring64_bitmap_t *r, uint64_t val) {+    art_t *art = &r->art;+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    leaf_t *leaf = (leaf_t *)art_find(art, high48);++    if (leaf == NULL) {+        return false;+    }+    int old_cardinality =+        container_get_cardinality(get_container(r, *leaf), get_typecode(*leaf));+    if (containerptr_roaring64_bitmap_remove(r, high48, low16, leaf)) {+        return true;+    }+    int new_cardinality =+        container_get_cardinality(get_container(r, *leaf), get_typecode(*leaf));+    return new_cardinality != old_cardinality;+}++void roaring64_bitmap_remove_bulk(roaring64_bitmap_t *r,+                                  roaring64_bulk_context_t *context,+                                  uint64_t val) {+    art_t *art = &r->art;+    uint8_t high48[ART_KEY_BYTES];+    uint16_t low16 = split_key(val, high48);+    if (context->leaf != NULL &&+        compare_high48(context->high_bytes, high48) == 0) {+        // We're at a container with the correct high bits.+        uint8_t typecode = get_typecode(*context->leaf);+        container_t *container = get_container(r, *context->leaf);+        uint8_t typecode2;+        container_t *container2 =+            container_remove(container, low16, typecode, &typecode2);+        if (container2 != container) {+            container_free(container, typecode);+            replace_container(r, context->leaf, container2, typecode2);+        }+        if (!container_nonzero_cardinality(container2, typecode2)) {+            container_free(container2, typecode2);+            leaf_t leaf = 0;+            bool erased = art_erase(art, high48, (art_val_t *)&leaf);+            assert(erased);+            (void)erased;+            remove_container(r, leaf);+            context->leaf = NULL;+        }+    } else {+        // We're not positioned anywhere yet or the high bits of the key+        // differ.+        leaf_t *leaf = (leaf_t *)art_find(art, high48);+        if (containerptr_roaring64_bitmap_remove(r, high48, low16, leaf)) {+            context->leaf = NULL;+        } else {+            context->leaf = leaf;+            memcpy(context->high_bytes, high48, ART_KEY_BYTES);+        }+    }+}++void roaring64_bitmap_remove_many(roaring64_bitmap_t *r, size_t n_args,+                                  const uint64_t *vals) {+    if (n_args == 0) {+        return;+    }+    const uint64_t *end = vals + n_args;+    roaring64_bulk_context_t context = CROARING_ZERO_INITIALIZER;+    for (const uint64_t *current_val = vals; current_val != end;+         current_val++) {+        roaring64_bitmap_remove_bulk(r, &context, *current_val);+    }+}++static inline void remove_range_closed_at(roaring64_bitmap_t *r, art_t *art,+                                          uint8_t *high48, uint16_t min,+                                          uint16_t max) {+    leaf_t *leaf = (leaf_t *)art_find(art, high48);+    if (leaf == NULL) {+        return;+    }+    uint8_t typecode = get_typecode(*leaf);+    container_t *container = get_container(r, *leaf);+    uint8_t typecode2;+    container_t *container2 =+        container_remove_range(container, typecode, min, max, &typecode2);+    if (container2 != container) {+        container_free(container, typecode);+        if (container2 != NULL) {+            replace_container(r, leaf, container2, typecode2);+        } else {+            bool erased = art_erase(art, high48, NULL);+            assert(erased);+            (void)erased;+            remove_container(r, *leaf);+        }+    }+}++void roaring64_bitmap_remove_range(roaring64_bitmap_t *r, uint64_t min,+                                   uint64_t max) {+    if (min >= max) {+        return;+    }+    roaring64_bitmap_remove_range_closed(r, min, max - 1);+}++void roaring64_bitmap_remove_range_closed(roaring64_bitmap_t *r, uint64_t min,+                                          uint64_t max) {+    if (min > max) {+        return;+    }++    art_t *art = &r->art;+    uint8_t min_high48[ART_KEY_BYTES];+    uint16_t min_low16 = split_key(min, min_high48);+    uint8_t max_high48[ART_KEY_BYTES];+    uint16_t max_low16 = split_key(max, max_high48);+    if (compare_high48(min_high48, max_high48) == 0) {+        // Only remove a range within one container.+        remove_range_closed_at(r, art, min_high48, min_low16, max_low16);+        return;+    }++    // Remove a range across containers. Remove intermediate containers+    // entirely.+    remove_range_closed_at(r, art, min_high48, min_low16, 0xffff);++    art_iterator_t it = art_upper_bound(art, min_high48);+    while (it.value != NULL && art_compare_keys(it.key, max_high48) < 0) {+        leaf_t leaf = 0;+        bool erased = art_iterator_erase(&it, (art_val_t *)&leaf);+        assert(erased);+        (void)erased;+        container_free(get_container(r, leaf), get_typecode(leaf));+        remove_container(r, leaf);+    }+    remove_range_closed_at(r, art, max_high48, 0, max_low16);+}++void roaring64_bitmap_clear(roaring64_bitmap_t *r) {+    roaring64_bitmap_remove_range_closed(r, 0, UINT64_MAX);+}++uint64_t roaring64_bitmap_get_cardinality(const roaring64_bitmap_t *r) {+    // Scan the pointer array rather than the ART: the arrays are sequential+    // and the ART is not. first_free is a free-list head, not a size, so+    // after deletes live containers can sit above it; skip NULL slots.+    uint64_t cardinality = 0;+    for (uint64_t i = 0; i < r->capacity; ++i) {+        if (r->containers[i] != NULL) {+            cardinality +=+                container_get_cardinality(r->containers[i], r->typecodes[i]);+        }+    }+    return cardinality;+}++uint64_t roaring64_bitmap_range_cardinality(const roaring64_bitmap_t *r,+                                            uint64_t min, uint64_t max) {+    if (min >= max) {+        return 0;+    }+    // Convert to a closed range+    // No underflow here: passing the above condition implies min < max, so+    // there is a number less than max+    return roaring64_bitmap_range_closed_cardinality(r, min, max - 1);+}++uint64_t roaring64_bitmap_range_closed_cardinality(const roaring64_bitmap_t *r,+                                                   uint64_t min, uint64_t max) {+    if (min > max) {+        return 0;+    }++    uint64_t cardinality = 0;+    uint8_t min_high48[ART_KEY_BYTES];+    uint16_t min_low16 = split_key(min, min_high48);+    uint8_t max_high48[ART_KEY_BYTES];+    uint16_t max_low16 = split_key(max, max_high48);++    art_iterator_t it = art_lower_bound((art_t *)&r->art, min_high48);+    while (it.value != NULL) {+        int max_compare_result = compare_high48(it.key, max_high48);+        if (max_compare_result > 0) {+            // We're outside the range.+            break;+        }++        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);+        container_t *container = get_container(r, leaf);+        if (max_compare_result == 0) {+            // We're at the max high key, add only the range up to the low+            // 16 bits of max.+            cardinality += container_rank(container, typecode, max_low16);+        } else {+            // We're not yet at the max high key, add the full container+            // range.+            cardinality += container_get_cardinality(container, typecode);+        }+        if (compare_high48(it.key, min_high48) == 0 && min_low16 > 0) {+            // We're at the min high key, remove the range up to the low 16+            // bits of min.+            cardinality -= container_rank(container, typecode, min_low16 - 1);+        }+        art_iterator_next(&it);+    }+    return cardinality;+}++bool roaring64_bitmap_is_empty(const roaring64_bitmap_t *r) {+    return art_is_empty(&r->art);+}++uint64_t roaring64_bitmap_minimum(const roaring64_bitmap_t *r) {+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    if (it.value == NULL) {+        return UINT64_MAX;+    }+    leaf_t leaf = (leaf_t)*it.value;+    return combine_key(+        it.key, container_minimum(get_container(r, leaf), get_typecode(leaf)));+}++uint64_t roaring64_bitmap_maximum(const roaring64_bitmap_t *r) {+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/false);+    if (it.value == NULL) {+        return 0;+    }+    leaf_t leaf = (leaf_t)*it.value;+    return combine_key(+        it.key, container_maximum(get_container(r, leaf), get_typecode(leaf)));+}++bool roaring64_bitmap_remove_run_compression(roaring64_bitmap_t *r) {+    art_iterator_t it = art_init_iterator(&r->art, /*first=*/true);+    bool removed = false;+    while (it.value != NULL) {+        leaf_t *leaf = (leaf_t *)it.value;+        if (get_typecode(*leaf) == RUN_CONTAINER_TYPE) {+            run_container_t *run = CAST_run(get_container(r, *leaf));+            int32_t card = run_container_cardinality(run);+            uint8_t new_typecode;+            container_t *new_container =+                convert_to_bitset_or_array_container(run, card, &new_typecode);+            run_container_free(run);+            replace_container(r, leaf, new_container, new_typecode);+            removed = true;+        }+        art_iterator_next(&it);+    }+    return removed;+}++bool roaring64_bitmap_run_optimize(roaring64_bitmap_t *r) {+    art_iterator_t it = art_init_iterator(&r->art, /*first=*/true);+    bool has_run_container = false;+    while (it.value != NULL) {+        leaf_t *leaf = (leaf_t *)it.value;+        uint8_t new_typecode;+        // We don't need to free the existing container if a new one was+        // created, convert_run_optimize does that internally.+        container_t *new_container = convert_run_optimize(+            get_container(r, *leaf), get_typecode(*leaf), &new_typecode);+        replace_container(r, leaf, new_container, new_typecode);+        has_run_container |= new_typecode == RUN_CONTAINER_TYPE;+        art_iterator_next(&it);+    }+    return has_run_container;+}++static void move_to_shrink(roaring64_bitmap_t *r, leaf_t *leaf) {+    uint64_t idx = get_index(*leaf);+    if (idx < r->first_free) {+        return;+    }+    uint8_t typecode = get_typecode(*leaf);+    container_t *c = get_container(r, *leaf);+    uint64_t new_idx = allocate_index(r);+    set_container_at(r, new_idx, c, typecode);+    free_index(r, idx);+    *leaf = create_leaf(new_idx, typecode);+}++static inline bool is_shrunken(const roaring64_bitmap_t *r) {+    return art_is_shrunken(&r->art) && r->first_free == r->capacity;+}++size_t roaring64_bitmap_shrink_to_fit(roaring64_bitmap_t *r) {+    size_t freed = art_shrink_to_fit(&r->art);+    art_iterator_t it = art_init_iterator(&r->art, true);+    while (it.value != NULL) {+        leaf_t *leaf = (leaf_t *)it.value;+        freed += container_shrink_to_fit(get_container(r, *leaf),+                                         get_typecode(*leaf));+        move_to_shrink(r, leaf);+        art_iterator_next(&it);+    }+    // Every container now sits below first_free, and every slot below it is+    // taken.+    r->end = r->first_free;+    holes_clear(r);+    if (r->holes != NULL) {+        freed += r->holes_capacity * sizeof(uint64_t);+        roaring_free(r->holes);+        r->holes = NULL;+        r->holes_capacity = 0;+    }+    uint64_t new_capacity = r->first_free;+    uint64_t capacity = r->capacity;+    if (new_capacity == capacity) {+        return freed;+    }+    size_t slack = (size_t)(capacity - new_capacity);+    // A failed realloc keeps the larger block, valid under the smaller capacity+    // since growth reallocs from the block; a later shrink does not retry it.+    bool shrunk;+    r->containers = (container_t **)croaring_realloc_shrink(+        r->containers, new_capacity * sizeof(container_t *), &shrunk);+    if (shrunk) freed += slack * sizeof(container_t *);+    r->typecodes = (uint8_t *)croaring_realloc_shrink(+        r->typecodes, new_capacity * sizeof(uint8_t), &shrunk);+    if (shrunk) freed += slack * sizeof(uint8_t);+    r->capacity = new_capacity;+    return freed;+}++/**+ *  (For advanced users.)+ * Collect statistics about the bitmap+ */+void roaring64_bitmap_statistics(const roaring64_bitmap_t *r,+                                 roaring64_statistics_t *stat) {+    memset(stat, 0, sizeof(*stat));+    stat->min_value = roaring64_bitmap_minimum(r);+    stat->max_value = roaring64_bitmap_maximum(r);++    art_iterator_t it = art_init_iterator((art_t *)&r->art, true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        stat->n_containers++;+        uint8_t truetype = get_typecode(leaf);+        uint32_t card = container_get_cardinality(get_container(r, leaf),+                                                  get_typecode(leaf));+        uint32_t sbytes =+            container_size_in_bytes(get_container(r, leaf), get_typecode(leaf));+        stat->cardinality += card;+        switch (truetype) {+            case BITSET_CONTAINER_TYPE:+                stat->n_bitset_containers++;+                stat->n_values_bitset_containers += card;+                stat->n_bytes_bitset_containers += sbytes;+                break;+            case ARRAY_CONTAINER_TYPE:+                stat->n_array_containers++;+                stat->n_values_array_containers += card;+                stat->n_bytes_array_containers += sbytes;+                break;+            case RUN_CONTAINER_TYPE:+                stat->n_run_containers++;+                stat->n_values_run_containers += card;+                stat->n_bytes_run_containers += sbytes;+                break;+            default:+                assert(false);+                roaring_unreachable;+        }+        art_iterator_next(&it);+    }+}++static bool roaring64_leaf_internal_validate(const art_val_t val,+                                             const char **reason,+                                             void *context) {+    leaf_t leaf = (leaf_t)val;+    roaring64_bitmap_t *r = (roaring64_bitmap_t *)context;+    uint64_t index = get_index(leaf);+    uint8_t typecode = get_typecode(leaf);+    if (index >= r->capacity || r->containers[index] == NULL) {+        *reason = "ART leaf points at an empty container slot";+        return false;+    }+    if (r->typecodes[index] != typecode) {+        *reason = "typecode array does not match ART leaf";+        return false;+    }+    if (typecode == SHARED_CONTAINER_TYPE) {+        *reason = "64-bit bitmaps cannot hold shared containers";+        return false;+    }+    return container_internal_validate(r->containers[index], typecode, reason);+}++static int compare_uint64(const void *a, const void *b) {+    uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b;+    return (x > y) - (x < y);+}++// Checks the bookkeeping of the container slots (see roaring64_bitmap_s).+static bool roaring64_slots_internal_validate(const roaring64_bitmap_t *r,+                                              const char **reason) {+    if (r->flags & ROARING_FLAG_FROZEN) {+        return true;  // read-only: every slot is marked as taken+    }+    if (r->end > r->capacity) {+        *reason = "container slots end beyond capacity";+        return false;+    }+    uint64_t free_below_end = 0;+    for (uint64_t i = 0; i < r->capacity; ++i) {+        if (r->containers[i] == NULL) {+            free_below_end += i < r->end;+        } else if (i >= r->end) {+            *reason = "container slot in use at or above end";+            return false;+        }+    }+    if (free_below_end != r->num_holes) {+        *reason = "wrong count of free container slots";+        return false;+    }+    // Each hole must be a distinct free slot below end: since there are as+    // many holes as free slots below end, they are then exactly those slots.+    uint64_t lowest = r->end;+    if (r->num_holes > 0) {+        uint64_t *sorted =+            (uint64_t *)roaring_malloc(r->num_holes * sizeof(uint64_t));+        if (sorted == NULL) {+            *reason = "out of memory";+            return false;+        }+        memcpy(sorted, r->holes, r->num_holes * sizeof(uint64_t));+        qsort(sorted, r->num_holes, sizeof(uint64_t), compare_uint64);+        bool ok = true;+        for (uint64_t i = 0; i < r->num_holes && ok; ++i) {+            ok = sorted[i] < r->end && r->containers[sorted[i]] == NULL &&+                 (i == 0 || sorted[i - 1] < sorted[i]);+        }+        roaring_free(sorted);+        if (!ok) {+            *reason = "container slot hole list is wrong";+            return false;+        }+        for (uint64_t i = 1; i < r->num_holes; ++i) {+            if (r->holes[(i - 1) / 2] > r->holes[i]) {+                *reason = "container slot holes are not a heap";+                return false;+            }+        }+        lowest = r->holes[0];+    }+    if (lowest != r->first_free) {+        *reason = "first_free is not the lowest free container slot";+        return false;+    }+    return true;+}++bool roaring64_bitmap_internal_validate(const roaring64_bitmap_t *r,+                                        const char **reason) {+    const char *reason_local;+    if (reason == NULL) {+        reason = &reason_local;+    }+    if (!roaring64_slots_internal_validate(r, reason)) {+        return false;+    }+    return art_internal_validate(&r->art, reason,+                                 roaring64_leaf_internal_validate, (void *)r);+}++bool roaring64_bitmap_equals(const roaring64_bitmap_t *r1,+                             const roaring64_bitmap_t *r2) {+    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL && it2.value != NULL) {+        if (compare_high48(it1.key, it2.key) != 0) {+            return false;+        }+        leaf_t leaf1 = (leaf_t)*it1.value;+        leaf_t leaf2 = (leaf_t)*it2.value;+        if (!container_equals(get_container(r1, leaf1), get_typecode(leaf1),+                              get_container(r2, leaf2), get_typecode(leaf2))) {+            return false;+        }+        art_iterator_next(&it1);+        art_iterator_next(&it2);+    }+    return it1.value == NULL && it2.value == NULL;+}++bool roaring64_bitmap_is_subset(const roaring64_bitmap_t *r1,+                                const roaring64_bitmap_t *r2) {+    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL) {+        bool it2_present = it2.value != NULL;++        int compare_result = 0;+        if (it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                leaf_t leaf1 = (leaf_t)*it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                if (!container_is_subset(+                        get_container(r1, leaf1), get_typecode(leaf1),+                        get_container(r2, leaf2), get_typecode(leaf2))) {+                    return false;+                }+                art_iterator_next(&it1);+                art_iterator_next(&it2);+            }+        }+        if (!it2_present || compare_result < 0) {+            return false;+        } else if (compare_result > 0) {+            art_iterator_lower_bound(&it2, it1.key);+        }+    }+    return true;+}++bool roaring64_bitmap_is_strict_subset(const roaring64_bitmap_t *r1,+                                       const roaring64_bitmap_t *r2) {+    return roaring64_bitmap_get_cardinality(r1) <+               roaring64_bitmap_get_cardinality(r2) &&+           roaring64_bitmap_is_subset(r1, r2);+}++roaring64_bitmap_t *roaring64_bitmap_and(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();++    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL && it2.value != NULL) {+        // Cases:+        // 1. it1 <  it2 -> it1+++        // 2. it1 == it1 -> output it1 & it2, it1++, it2+++        // 3. it1 >  it2 -> it2+++        int compare_result = compare_high48(it1.key, it2.key);+        if (compare_result == 0) {+            // Case 2: iterators at the same high key position.+            leaf_t leaf1 = (leaf_t)*it1.value;+            leaf_t leaf2 = (leaf_t)*it2.value;+            uint8_t result_typecode;+            container_t *result_container =+                container_and(get_container(r1, leaf1), get_typecode(leaf1),+                              get_container(r2, leaf2), get_typecode(leaf2),+                              &result_typecode);+            if (container_nonzero_cardinality(result_container,+                                              result_typecode)) {+                leaf_t result_leaf =+                    add_container(result, result_container, result_typecode);+                art_insert(&result->art, it1.key, (art_val_t)result_leaf);+            } else {+                container_free(result_container, result_typecode);+            }+            art_iterator_next(&it1);+            art_iterator_next(&it2);+        } else if (compare_result < 0) {+            // Case 1: it1 is before it2.+            art_iterator_lower_bound(&it1, it2.key);+        } else {+            // Case 3: it2 is before it1.+            art_iterator_lower_bound(&it2, it1.key);+        }+    }+    return result;+}++uint64_t roaring64_bitmap_and_cardinality(const roaring64_bitmap_t *r1,+                                          const roaring64_bitmap_t *r2) {+    uint64_t result = 0;++    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL && it2.value != NULL) {+        // Cases:+        // 1. it1 <  it2 -> it1+++        // 2. it1 == it1 -> output cardinaltiy it1 & it2, it1++, it2+++        // 3. it1 >  it2 -> it2+++        int compare_result = compare_high48(it1.key, it2.key);+        if (compare_result == 0) {+            // Case 2: iterators at the same high key position.+            leaf_t leaf1 = (leaf_t)*it1.value;+            leaf_t leaf2 = (leaf_t)*it2.value;+            result += container_and_cardinality(+                get_container(r1, leaf1), get_typecode(leaf1),+                get_container(r2, leaf2), get_typecode(leaf2));+            art_iterator_next(&it1);+            art_iterator_next(&it2);+        } else if (compare_result < 0) {+            // Case 1: it1 is before it2.+            art_iterator_lower_bound(&it1, it2.key);+        } else {+            // Case 3: it2 is before it1.+            art_iterator_lower_bound(&it2, it1.key);+        }+    }+    return result;+}++// Inplace and (modifies its first argument).+void roaring64_bitmap_and_inplace(roaring64_bitmap_t *r1,+                                  const roaring64_bitmap_t *r2) {+    if (r1 == r2) {+        return;+    }+    art_iterator_t it1 = art_init_iterator(&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL) {+        // Cases:+        // 1. !it2_present -> erase it1+        // 2. it2_present+        //    a. it1 <  it2 -> erase it1+        //    b. it1 == it2 -> output it1 & it2, it1++, it2+++        //    c. it1 >  it2 -> it2+++        bool it2_present = it2.value != NULL;+        int compare_result = 0;+        if (it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 2b: iterators at the same high key position.+                leaf_t *leaf1 = (leaf_t *)it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;++                uint8_t typecode = get_typecode(*leaf1);+                container_t *container = get_container(r1, *leaf1);+                uint8_t typecode2;+                container_t *container2 = container_iand(+                    container, typecode, get_container(r2, leaf2),+                    get_typecode(leaf2), &typecode2);++                if (container2 != container) {+                    container_free(container, typecode);+                }+                if (!container_nonzero_cardinality(container2, typecode2)) {+                    container_free(container2, typecode2);+                    art_iterator_erase(&it1, NULL);+                    remove_container(r1, *leaf1);+                } else {+                    if (container2 != container) {+                        replace_container(r1, leaf1, container2, typecode2);+                    }+                    // Only advance the iterator if we didn't delete the+                    // leaf, as erasing advances by itself.+                    art_iterator_next(&it1);+                }+                art_iterator_next(&it2);+            }+        }++        if (!it2_present || compare_result < 0) {+            // Cases 1 and 2a: it1 is the only iterator or is before it2.+            leaf_t leaf = 0;+            bool erased = art_iterator_erase(&it1, (art_val_t *)&leaf);+            assert(erased);+            (void)erased;+            container_free(get_container(r1, leaf), get_typecode(leaf));+            remove_container(r1, leaf);+        } else if (compare_result > 0) {+            // Case 2c: it1 is after it2.+            art_iterator_lower_bound(&it2, it1.key);+        }+    }+}++bool roaring64_bitmap_intersect(const roaring64_bitmap_t *r1,+                                const roaring64_bitmap_t *r2) {+    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL && it2.value != NULL) {+        // Cases:+        // 1. it1 <  it2 -> it1+++        // 2. it1 == it2 -> return true if the containers intersect,+        //    else it1++, it2+++        // 3. it1 >  it2 -> it2+++        int compare_result = compare_high48(it1.key, it2.key);+        if (compare_result == 0) {+            // Case 2: iterators at the same high key position.+            leaf_t leaf1 = (leaf_t)*it1.value;+            leaf_t leaf2 = (leaf_t)*it2.value;+            if (container_intersect(+                    get_container(r1, leaf1), get_typecode(leaf1),+                    get_container(r2, leaf2), get_typecode(leaf2))) {+                return true;+            }+            art_iterator_next(&it1);+            art_iterator_next(&it2);+        } else if (compare_result < 0) {+            // Case 1: it1 is before it2.+            art_iterator_lower_bound(&it1, it2.key);+        } else {+            // Case 3: it2 is before it1.+            art_iterator_lower_bound(&it2, it1.key);+        }+    }+    return false;+}++bool roaring64_bitmap_intersect_with_range(const roaring64_bitmap_t *r,+                                           uint64_t min, uint64_t max) {+    if (min >= max) {+        return false;+    }+    roaring64_iterator_t it;+    roaring64_iterator_init_at(r, &it, /*first=*/true);+    if (!roaring64_iterator_move_equalorlarger(&it, min)) {+        return false;+    }+    return roaring64_iterator_has_value(&it) &&+           roaring64_iterator_value(&it) < max;+}++double roaring64_bitmap_jaccard_index(const roaring64_bitmap_t *r1,+                                      const roaring64_bitmap_t *r2) {+    uint64_t c1 = roaring64_bitmap_get_cardinality(r1);+    uint64_t c2 = roaring64_bitmap_get_cardinality(r2);+    uint64_t inter = roaring64_bitmap_and_cardinality(r1, r2);+    return (double)inter / (double)(c1 + c2 - inter);+}++roaring64_bitmap_t *roaring64_bitmap_or(const roaring64_bitmap_t *r1,+                                        const roaring64_bitmap_t *r2) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();+    // The result has at least max(n1, n2) containers and at most n1 + n2, so+    // reserving the upper bound is at most a 2x overestimate.+    reserve_containers(result,+                       art_num_leaves(&r1->art) + art_num_leaves(&r2->art));++    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL || it2.value != NULL) {+        bool it1_present = it1.value != NULL;+        bool it2_present = it2.value != NULL;++        // Cases:+        // 1. it1_present  && !it2_present -> output it1, it1+++        // 2. !it1_present && it2_present  -> output it2, it2+++        // 3. it1_present  && it2_present+        //    a. it1 <  it2 -> output it1, it1+++        //    b. it1 == it2 -> output it1 | it2, it1++, it2+++        //    c. it1 >  it2 -> output it2, it2+++        int compare_result = 0;+        if (it1_present && it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 3b: iterators at the same high key position.+                leaf_t leaf1 = (leaf_t)*it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t result_typecode;+                container_t *result_container =+                    container_or(get_container(r1, leaf1), get_typecode(leaf1),+                                 get_container(r2, leaf2), get_typecode(leaf2),+                                 &result_typecode);+                leaf_t result_leaf =+                    add_container(result, result_container, result_typecode);+                art_insert(&result->art, it1.key, (art_val_t)result_leaf);+                art_iterator_next(&it1);+                art_iterator_next(&it2);+            }+        }+        if ((it1_present && !it2_present) || compare_result < 0) {+            // Cases 1 and 3a: it1 is the only iterator or is before it2.+            leaf_t result_leaf =+                copy_leaf_container(r1, result, (leaf_t)*it1.value);+            art_insert(&result->art, it1.key, (art_val_t)result_leaf);+            art_iterator_next(&it1);+        } else if ((!it1_present && it2_present) || compare_result > 0) {+            // Cases 2 and 3c: it2 is the only iterator or is before it1.+            leaf_t result_leaf =+                copy_leaf_container(r2, result, (leaf_t)*it2.value);+            art_insert(&result->art, it2.key, (art_val_t)result_leaf);+            art_iterator_next(&it2);+        }+    }+    return result;+}++uint64_t roaring64_bitmap_or_cardinality(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2) {+    uint64_t c1 = roaring64_bitmap_get_cardinality(r1);+    uint64_t c2 = roaring64_bitmap_get_cardinality(r2);+    uint64_t inter = roaring64_bitmap_and_cardinality(r1, r2);+    return c1 + c2 - inter;+}++void roaring64_bitmap_or_inplace(roaring64_bitmap_t *r1,+                                 const roaring64_bitmap_t *r2) {+    if (r1 == r2) {+        return;+    }+    art_iterator_t it1 = art_init_iterator(&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL || it2.value != NULL) {+        bool it1_present = it1.value != NULL;+        bool it2_present = it2.value != NULL;++        // Cases:+        // 1. it1_present  && !it2_present -> it1+++        // 2. !it1_present && it2_present  -> add it2, it2+++        // 3. it1_present  && it2_present+        //    a. it1 <  it2 -> it1+++        //    b. it1 == it2 -> it1 | it2, it1++, it2+++        //    c. it1 >  it2 -> add it2, it2+++        int compare_result = 0;+        if (it1_present && it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 3b: iterators at the same high key position.+                leaf_t *leaf1 = (leaf_t *)it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t typecode1 = get_typecode(*leaf1);+                container_t *container1 = get_container(r1, *leaf1);+                uint8_t typecode2;+                container_t *container2 = container_ior(+                    container1, typecode1, get_container(r2, leaf2),+                    get_typecode(leaf2), &typecode2);+                if (container2 != container1) {+                    container_free(container1, typecode1);+                    replace_container(r1, leaf1, container2, typecode2);+                }+                art_iterator_next(&it1);+                art_iterator_next(&it2);+            }+        }+        if ((it1_present && !it2_present) || compare_result < 0) {+            // Cases 1 and 3a: it1 is the only iterator or is before it2.+            art_iterator_next(&it1);+        } else if ((!it1_present && it2_present) || compare_result > 0) {+            // Cases 2 and 3c: it2 is the only iterator or is before it1.+            leaf_t result_leaf =+                copy_leaf_container(r2, r1, (leaf_t)*it2.value);+            art_iterator_insert(&it1, it2.key, (art_val_t)result_leaf);+            art_iterator_next(&it2);+        }+    }+}++roaring64_bitmap_t *roaring64_bitmap_xor(const roaring64_bitmap_t *r1,+                                         const roaring64_bitmap_t *r2) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();++    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL || it2.value != NULL) {+        bool it1_present = it1.value != NULL;+        bool it2_present = it2.value != NULL;++        // Cases:+        // 1. it1_present  && !it2_present -> output it1, it1+++        // 2. !it1_present && it2_present  -> output it2, it2+++        // 3. it1_present  && it2_present+        //    a. it1 <  it2 -> output it1, it1+++        //    b. it1 == it2 -> output it1 ^ it2, it1++, it2+++        //    c. it1 >  it2 -> output it2, it2+++        int compare_result = 0;+        if (it1_present && it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 3b: iterators at the same high key position.+                leaf_t leaf1 = (leaf_t)*it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t result_typecode;+                container_t *result_container =+                    container_xor(get_container(r1, leaf1), get_typecode(leaf1),+                                  get_container(r2, leaf2), get_typecode(leaf2),+                                  &result_typecode);+                if (container_nonzero_cardinality(result_container,+                                                  result_typecode)) {+                    leaf_t result_leaf = add_container(result, result_container,+                                                       result_typecode);+                    art_insert(&result->art, it1.key, (art_val_t)result_leaf);+                } else {+                    container_free(result_container, result_typecode);+                }+                art_iterator_next(&it1);+                art_iterator_next(&it2);+            }+        }+        if ((it1_present && !it2_present) || compare_result < 0) {+            // Cases 1 and 3a: it1 is the only iterator or is before it2.+            leaf_t result_leaf =+                copy_leaf_container(r1, result, (leaf_t)*it1.value);+            art_insert(&result->art, it1.key, (art_val_t)result_leaf);+            art_iterator_next(&it1);+        } else if ((!it1_present && it2_present) || compare_result > 0) {+            // Cases 2 and 3c: it2 is the only iterator or is before it1.+            leaf_t result_leaf =+                copy_leaf_container(r2, result, (leaf_t)*it2.value);+            art_insert(&result->art, it2.key, (art_val_t)result_leaf);+            art_iterator_next(&it2);+        }+    }+    return result;+}++uint64_t roaring64_bitmap_xor_cardinality(const roaring64_bitmap_t *r1,+                                          const roaring64_bitmap_t *r2) {+    uint64_t c1 = roaring64_bitmap_get_cardinality(r1);+    uint64_t c2 = roaring64_bitmap_get_cardinality(r2);+    uint64_t inter = roaring64_bitmap_and_cardinality(r1, r2);+    return c1 + c2 - 2 * inter;+}++void roaring64_bitmap_xor_inplace(roaring64_bitmap_t *r1,+                                  const roaring64_bitmap_t *r2) {+    assert(r1 != r2);+    art_iterator_t it1 = art_init_iterator(&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL || it2.value != NULL) {+        bool it1_present = it1.value != NULL;+        bool it2_present = it2.value != NULL;++        // Cases:+        // 1.  it1_present && !it2_present -> it1+++        // 2. !it1_present &&  it2_present -> add it2, it2+++        // 3.  it1_present &&  it2_present+        //    a. it1 <  it2 -> it1+++        //    b. it1 == it2 -> it1 ^ it2, it1++, it2+++        //    c. it1 >  it2 -> add it2, it2+++        int compare_result = 0;+        if (it1_present && it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 3b: iterators at the same high key position.+                leaf_t *leaf1 = (leaf_t *)it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t typecode1 = get_typecode(*leaf1);+                container_t *container1 = get_container(r1, *leaf1);+                uint8_t typecode2;+                container_t *container2 = container_ixor(+                    container1, typecode1, get_container(r2, leaf2),+                    get_typecode(leaf2), &typecode2);++                if (!container_nonzero_cardinality(container2, typecode2)) {+                    container_free(container2, typecode2);+                    bool erased = art_iterator_erase(&it1, NULL);+                    assert(erased);+                    (void)erased;+                    remove_container(r1, *leaf1);+                } else {+                    if (container2 != container1) {+                        replace_container(r1, leaf1, container2, typecode2);+                    }+                    // Only advance the iterator if we didn't delete the+                    // leaf, as erasing advances by itself.+                    art_iterator_next(&it1);+                }+                art_iterator_next(&it2);+            }+        }+        if ((it1_present && !it2_present) || compare_result < 0) {+            // Cases 1 and 3a: it1 is the only iterator or is before it2.+            art_iterator_next(&it1);+        } else if ((!it1_present && it2_present) || compare_result > 0) {+            // Cases 2 and 3c: it2 is the only iterator or is before it1.+            leaf_t result_leaf =+                copy_leaf_container(r2, r1, (leaf_t)*it2.value);+            if (it1_present) {+                art_iterator_insert(&it1, it2.key, (art_val_t)result_leaf);+                art_iterator_next(&it1);+            } else {+                art_insert(&r1->art, it2.key, (art_val_t)result_leaf);+            }+            art_iterator_next(&it2);+        }+    }+}++roaring64_bitmap_t *roaring64_bitmap_andnot(const roaring64_bitmap_t *r1,+                                            const roaring64_bitmap_t *r2) {+    roaring64_bitmap_t *result = roaring64_bitmap_create();++    art_iterator_t it1 = art_init_iterator((art_t *)&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL) {+        // Cases:+        // 1. it1_present && !it2_present -> output it1, it1+++        // 2. it1_present && it2_present+        //    a. it1 <  it2 -> output it1, it1+++        //    b. it1 == it2 -> output it1 - it2, it1++, it2+++        //    c. it1 >  it2 -> it2+++        bool it2_present = it2.value != NULL;+        int compare_result = 0;+        if (it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 2b: iterators at the same high key position.+                leaf_t *leaf1 = (leaf_t *)it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t result_typecode;+                container_t *result_container = container_andnot(+                    get_container(r1, *leaf1), get_typecode(*leaf1),+                    get_container(r2, leaf2), get_typecode(leaf2),+                    &result_typecode);++                if (container_nonzero_cardinality(result_container,+                                                  result_typecode)) {+                    leaf_t result_leaf = add_container(result, result_container,+                                                       result_typecode);+                    art_insert(&result->art, it1.key, (art_val_t)result_leaf);+                } else {+                    container_free(result_container, result_typecode);+                }+                art_iterator_next(&it1);+                art_iterator_next(&it2);+            }+        }+        if (!it2_present || compare_result < 0) {+            // Cases 1 and 2a: it1 is the only iterator or is before it2.+            leaf_t result_leaf =+                copy_leaf_container(r1, result, (leaf_t)*it1.value);+            art_insert(&result->art, it1.key, (art_val_t)result_leaf);+            art_iterator_next(&it1);+        } else if (compare_result > 0) {+            // Case 2c: it1 is after it2.+            art_iterator_next(&it2);+        }+    }+    return result;+}++uint64_t roaring64_bitmap_andnot_cardinality(const roaring64_bitmap_t *r1,+                                             const roaring64_bitmap_t *r2) {+    uint64_t c1 = roaring64_bitmap_get_cardinality(r1);+    uint64_t inter = roaring64_bitmap_and_cardinality(r1, r2);+    return c1 - inter;+}++void roaring64_bitmap_andnot_inplace(roaring64_bitmap_t *r1,+                                     const roaring64_bitmap_t *r2) {+    art_iterator_t it1 = art_init_iterator(&r1->art, /*first=*/true);+    art_iterator_t it2 = art_init_iterator((art_t *)&r2->art, /*first=*/true);++    while (it1.value != NULL) {+        // Cases:+        // 1. it1_present && !it2_present -> it1+++        // 2. it1_present &&  it2_present+        //    a. it1 <  it2 -> it1+++        //    b. it1 == it2 -> it1 - it2, it1++, it2+++        //    c. it1 >  it2 -> it2+++        bool it2_present = it2.value != NULL;+        int compare_result = 0;+        if (it2_present) {+            compare_result = compare_high48(it1.key, it2.key);+            if (compare_result == 0) {+                // Case 2b: iterators at the same high key position.+                leaf_t *leaf1 = (leaf_t *)it1.value;+                leaf_t leaf2 = (leaf_t)*it2.value;+                uint8_t typecode1 = get_typecode(*leaf1);+                container_t *container1 = get_container(r1, *leaf1);+                uint8_t typecode2;+                container_t *container2 = container_iandnot(+                    container1, typecode1, get_container(r2, leaf2),+                    get_typecode(leaf2), &typecode2);++                if (!container_nonzero_cardinality(container2, typecode2)) {+                    container_free(container2, typecode2);+                    bool erased = art_iterator_erase(&it1, NULL);+                    assert(erased);+                    (void)erased;+                    remove_container(r1, *leaf1);+                } else {+                    if (container2 != container1) {+                        replace_container(r1, leaf1, container2, typecode2);+                    }+                    // Only advance the iterator if we didn't delete the+                    // leaf, as erasing advances by itself.+                    art_iterator_next(&it1);+                }+                art_iterator_next(&it2);+            }+        }+        if (!it2_present || compare_result < 0) {+            // Cases 1 and 2a: it1 is the only iterator or is before it2.+            art_iterator_next(&it1);+        } else if (compare_result > 0) {+            // Case 2c: it1 is after it2.+            art_iterator_next(&it2);+        }+    }+}++/**+ * Flips the leaf at high48 in the range [min, max), adding the result to+ * `r2`. If the high48 key is not found in `r1`, a new container is created.+ */+static void roaring64_flip_leaf(const roaring64_bitmap_t *r1,+                                roaring64_bitmap_t *r2, uint8_t high48[],+                                uint32_t min, uint32_t max) {+    leaf_t *leaf1 = (leaf_t *)art_find(&r1->art, high48);+    uint8_t typecode2;+    container_t *container2;+    if (leaf1 == NULL) {+        // No container at this key, create a full container.+        container2 = container_range_of_ones(min, max, &typecode2);+    } else if (min == 0 && max > 0xFFFF) {+        // Flip whole container.+        container2 = container_not(get_container(r1, *leaf1),+                                   get_typecode(*leaf1), &typecode2);+    } else {+        // Partially flip a container.+        container2 =+            container_not_range(get_container(r1, *leaf1), get_typecode(*leaf1),+                                min, max, &typecode2);+    }+    if (container_nonzero_cardinality(container2, typecode2)) {+        leaf_t leaf2 = add_container(r2, container2, typecode2);+        art_insert(&r2->art, high48, (art_val_t)leaf2);+    } else {+        container_free(container2, typecode2);+    }+}++/**+ * Flips the leaf at high48 in the range [min, max). If the high48 key is+ * not found in the bitmap, a new container is created. Deletes the leaf and+ * associated container if the negation results in an empty range.+ */+static void roaring64_flip_leaf_inplace(roaring64_bitmap_t *r, uint8_t high48[],+                                        uint32_t min, uint32_t max) {+    leaf_t *leaf = (leaf_t *)art_find(&r->art, high48);+    container_t *container2;+    uint8_t typecode2;+    if (leaf == NULL) {+        // No container at this key, insert a full container.+        container2 = container_range_of_ones(min, max, &typecode2);+        leaf_t new_leaf = add_container(r, container2, typecode2);+        art_insert(&r->art, high48, (art_val_t)new_leaf);+        return;+    }++    if (min == 0 && max > 0xFFFF) {+        // Flip whole container.+        container2 = container_inot(get_container(r, *leaf),+                                    get_typecode(*leaf), &typecode2);+    } else {+        // Partially flip a container.+        container2 = container_inot_range(+            get_container(r, *leaf), get_typecode(*leaf), min, max, &typecode2);+    }++    if (container_nonzero_cardinality(container2, typecode2)) {+        replace_container(r, leaf, container2, typecode2);+    } else {+        bool erased = art_erase(&r->art, high48, NULL);+        assert(erased);+        (void)erased;+        container_free(container2, typecode2);+        remove_container(r, *leaf);+    }+}++roaring64_bitmap_t *roaring64_bitmap_flip(const roaring64_bitmap_t *r,+                                          uint64_t min, uint64_t max) {+    if (min >= max) {+        return roaring64_bitmap_copy(r);+    }+    return roaring64_bitmap_flip_closed(r, min, max - 1);+}++roaring64_bitmap_t *roaring64_bitmap_flip_closed(const roaring64_bitmap_t *r1,+                                                 uint64_t min, uint64_t max) {+    if (min > max) {+        return roaring64_bitmap_copy(r1);+    }+    uint8_t min_high48_key[ART_KEY_BYTES];+    uint16_t min_low16 = split_key(min, min_high48_key);+    uint8_t max_high48_key[ART_KEY_BYTES];+    uint16_t max_low16 = split_key(max, max_high48_key);+    uint64_t min_high48_bits = (min & 0xFFFFFFFFFFFF0000ULL) >> 16;+    uint64_t max_high48_bits = (max & 0xFFFFFFFFFFFF0000ULL) >> 16;++    roaring64_bitmap_t *r2 = roaring64_bitmap_create();+    art_iterator_t it = art_init_iterator((art_t *)&r1->art, /*first=*/true);++    // Copy the containers before min unchanged.+    while (it.value != NULL && compare_high48(it.key, min_high48_key) < 0) {+        leaf_t leaf1 = (leaf_t)*it.value;+        uint8_t typecode2 = get_typecode(leaf1);+        container_t *container2 =+            container_clone(get_container(r1, leaf1), typecode2);+        leaf_t leaf2 = add_container(r2, container2, typecode2);+        art_insert(&r2->art, it.key, (art_val_t)leaf2);+        art_iterator_next(&it);+    }++    // Flip the range (including non-existent containers!) between min and+    // max.+    for (uint64_t high48_bits = min_high48_bits; high48_bits <= max_high48_bits;+         high48_bits++) {+        uint8_t current_high48_key[ART_KEY_BYTES];+        split_key(high48_bits << 16, current_high48_key);++        uint32_t min_container = 0;+        if (high48_bits == min_high48_bits) {+            min_container = min_low16;+        }+        uint32_t max_container = 0xFFFF + 1;  // Exclusive range.+        if (high48_bits == max_high48_bits) {+            max_container = max_low16 + 1;  // Exclusive.+        }++        roaring64_flip_leaf(r1, r2, current_high48_key, min_container,+                            max_container);+    }++    // Copy the containers after max unchanged.+    it = art_upper_bound((art_t *)&r1->art, max_high48_key);+    while (it.value != NULL) {+        leaf_t leaf1 = (leaf_t)*it.value;+        uint8_t typecode2 = get_typecode(leaf1);+        container_t *container2 =+            container_clone(get_container(r1, leaf1), typecode2);+        leaf_t leaf2 = add_container(r2, container2, typecode2);+        art_insert(&r2->art, it.key, (art_val_t)leaf2);+        art_iterator_next(&it);+    }++    return r2;+}++void roaring64_bitmap_flip_inplace(roaring64_bitmap_t *r, uint64_t min,+                                   uint64_t max) {+    if (min >= max) {+        return;+    }+    roaring64_bitmap_flip_closed_inplace(r, min, max - 1);+}++void roaring64_bitmap_flip_closed_inplace(roaring64_bitmap_t *r, uint64_t min,+                                          uint64_t max) {+    if (min > max) {+        return;+    }+    uint16_t min_low16 = (uint16_t)min;+    uint16_t max_low16 = (uint16_t)max;+    uint64_t min_high48_bits = (min & 0xFFFFFFFFFFFF0000ULL) >> 16;+    uint64_t max_high48_bits = (max & 0xFFFFFFFFFFFF0000ULL) >> 16;++    // Flip the range (including non-existent containers!) between min and+    // max.+    for (uint64_t high48_bits = min_high48_bits; high48_bits <= max_high48_bits;+         high48_bits++) {+        uint8_t current_high48_key[ART_KEY_BYTES];+        split_key(high48_bits << 16, current_high48_key);++        uint32_t min_container = 0;+        if (high48_bits == min_high48_bits) {+            min_container = min_low16;+        }+        uint32_t max_container = 0xFFFF + 1;  // Exclusive range.+        if (high48_bits == max_high48_bits) {+            max_container = max_low16 + 1;  // Exclusive.+        }++        roaring64_flip_leaf_inplace(r, current_high48_key, min_container,+                                    max_container);+    }+}++roaring64_bitmap_t *roaring64_bitmap_add_offset_signed(+    const roaring64_bitmap_t *r, bool positive, uint64_t offset) {+    if (offset == 0) {+        return roaring64_bitmap_copy(r);+    }++    roaring64_bitmap_t *answer = roaring64_bitmap_create();++    // Decompose the offset into a signed container-level shift and an+    // intra-container shift. For negative offsets the low 16 bits wrap: e.g.+    // -1 = container_offset(-1) + in_offset(0xffff), because shifting by -1+    // container is a shift of -0x1_0000, so we need to shift up within+    // containers to get back to -1+    uint16_t low16 = (uint16_t)offset;+    int64_t container_offset;+    uint16_t in_offset;+    if (positive) {+        container_offset = (int64_t)(offset >> 16);+        in_offset = low16;+    } else if (low16 == 0) {+        container_offset = -(int64_t)(offset >> 16);+        in_offset = 0;+    } else {+        container_offset = -(int64_t)(offset >> 16) - 1;+        in_offset = (uint16_t)-low16;+    }++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);++    if (in_offset == 0) {+        while (it.value != NULL) {+            leaf_t leaf = (leaf_t)*it.value;+            int64_t k =+                (int64_t)(combine_key(it.key, 0) >> 16) + container_offset;+            if ((uint64_t)k < (uint64_t)1 << 48) {+                uint8_t new_high48[ART_KEY_BYTES];+                split_key((uint64_t)k << 16, new_high48);+                uint8_t typecode = get_typecode(leaf);+                container_t *container =+                    container_clone(get_container(r, leaf), typecode);+                leaf_t new_leaf = add_container(answer, container, typecode);+                art_insert(&answer->art, new_high48, (art_val_t)new_leaf);+            }+            art_iterator_next(&it);+        }+        return answer;+    }++    // Track the most recently inserted hi container so that the next+    // iteration's lo can merge with it without re-searching the ART.+    leaf_t *prev_hi_leaf = NULL;+    int64_t prev_hi_k = -1;++    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        int64_t k = (int64_t)(combine_key(it.key, 0) >> 16) + container_offset;++        container_t *lo = NULL, *hi = NULL;+        container_t **lo_ptr = NULL, **hi_ptr = NULL;++        if ((uint64_t)k < (uint64_t)1 << 48) {+            lo_ptr = &lo;+        }+        if ((uint64_t)(k + 1) < (uint64_t)1 << 48) {+            hi_ptr = &hi;+        }+        if (lo_ptr == NULL && hi_ptr == NULL) {+            art_iterator_next(&it);+            continue;+        }++        uint8_t typecode = get_typecode(leaf);+        const container_t *c = get_container(r, leaf);+        container_add_offset(c, typecode, lo_ptr, hi_ptr, in_offset);++        if (lo != NULL) {+            if (prev_hi_leaf != NULL && prev_hi_k == k) {+                uint8_t existing_type = get_typecode(*prev_hi_leaf);+                container_t *existing_c = get_container(answer, *prev_hi_leaf);+                uint8_t merged_type;+                container_t *merged_c = container_ior(+                    existing_c, existing_type, lo, typecode, &merged_type);+                if (merged_c != existing_c) {+                    container_free(existing_c, existing_type);+                }+                replace_container(answer, prev_hi_leaf, merged_c, merged_type);+                container_free(lo, typecode);+            } else {+                uint8_t lo_high48[ART_KEY_BYTES];+                split_key((uint64_t)k << 16, lo_high48);+                leaf_t new_leaf = add_container(answer, lo, typecode);+                art_insert(&answer->art, lo_high48, (art_val_t)new_leaf);+            }+        }++        prev_hi_leaf = NULL;+        if (hi != NULL) {+            uint8_t hi_high48[ART_KEY_BYTES];+            split_key((uint64_t)(k + 1) << 16, hi_high48);+            leaf_t new_leaf = add_container(answer, hi, typecode);+            prev_hi_leaf = (leaf_t *)art_insert(&answer->art, hi_high48,+                                                (art_val_t)new_leaf);+            prev_hi_k = k + 1;+        }++        art_iterator_next(&it);+    }++    // Repair containers (e.g., convert low-cardinality bitset containers to+    // array containers after lazy union operations).+    art_iterator_t repair_it = art_init_iterator(&answer->art, /*first=*/true);+    while (repair_it.value != NULL) {+        leaf_t *leaf_ptr = (leaf_t *)repair_it.value;+        uint8_t typecode = get_typecode(*leaf_ptr);+        container_t *repaired = container_repair_after_lazy(+            get_container(answer, *leaf_ptr), &typecode);+        replace_container(answer, leaf_ptr, repaired, typecode);+        art_iterator_next(&repair_it);+    }++    return answer;+}++size_t roaring64_bitmap_portable_size_in_bytes(const roaring64_bitmap_t *r) {+    // https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+    size_t size = 0;++    // Write as uint64 the distinct number of "buckets", where a bucket is+    // defined as the most significant 32 bits of an element.+    uint64_t high32_count;+    size += sizeof(high32_count);++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);++    // Iterate through buckets ordered by increasing keys.+    while (it.value != NULL) {+        uint32_t current_high32 = (uint32_t)(combine_key(it.key, 0) >> 32);++        // Write as uint32 the most significant 32 bits of the bucket.+        size += sizeof(current_high32);++        // Write the 32-bit Roaring bitmaps representing the least significant+        // bits of a set of elements: the header from+        // ra_portable_header_size_from_counts plus the size of each container.+        int32_t containers_with_high32 = 0;+        bool hasrun = false;+        bool bucket_done = false;+        while (!bucket_done) {+            // Read the containers in batches, so that their cache misses+            // overlap instead of each one waiting behind the tree walk.+            enum { LEAF_BATCH_SIZE = 64 };+            leaf_t leaves[LEAF_BATCH_SIZE];+            int32_t num_leaves = 0;+            do {+                leaves[num_leaves++] = (leaf_t)*it.value;+                art_iterator_next(&it);+                bucket_done =+                    it.value == NULL ||+                    (uint32_t)(combine_key(it.key, 0) >> 32) != current_high32;+            } while (!bucket_done && num_leaves < LEAF_BATCH_SIZE);+            for (int32_t i = 0; i < num_leaves; ++i) {+                const container_t *container = get_container(r, leaves[i]);+                uint8_t typecode = get_typecode(leaves[i]);+                hasrun |= typecode == RUN_CONTAINER_TYPE;+                size += container_size_in_bytes(container, typecode);+            }+            containers_with_high32 += num_leaves;+        }+        size +=+            ra_portable_header_size_from_counts(containers_with_high32, hasrun);+    }++    return size;+}++size_t roaring64_bitmap_portable_serialize(const roaring64_bitmap_t *r,+                                           char *buf) {+    // https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+    if (buf == NULL) {+        return 0;+    }+    const char *initial_buf = buf;++    // Write as uint64 the distinct number of "buckets", where a bucket is+    // defined as the most significant 32 bits of an element. The count is+    // only known after the loop, so we reserve its place for now.+    char *high32_count_buf = buf;+    buf += sizeof(uint64_t);+    uint64_t high32_count = 0;++    // A 32-bit roaring array that borrows the containers of the current+    // bucket. It is reused across buckets to avoid allocating per bucket.+    roaring_array_t bucket;+    ra_init(&bucket);++    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);++    // Iterate through buckets ordered by increasing keys.+    while (it.value != NULL) {+        uint32_t current_high32 = (uint32_t)(combine_key(it.key, 0) >> 32);+        bucket.size = 0;+        do {+            leaf_t leaf = (leaf_t)*it.value;+            ra_append(&bucket, (uint16_t)(combine_key(it.key, 0) >> 16),+                      get_container(r, leaf), get_typecode(leaf));+            art_iterator_next(&it);+        } while (it.value != NULL &&+                 (uint32_t)(combine_key(it.key, 0) >> 32) == current_high32);++        // Write as uint32 the most significant 32 bits of the bucket.+        uint32_t current_high32_le = croaring_htole32(current_high32);+        memcpy(buf, &current_high32_le, sizeof(current_high32_le));+        buf += sizeof(current_high32);++        // Write the 32-bit Roaring bitmaps representing the least+        // significant bits of a set of elements.+        buf += ra_portable_serialize(&bucket, buf);+        high32_count++;+    }+    ra_clear_without_containers(&bucket);++    uint64_t high32_count_le = croaring_htole64(high32_count);+    memcpy(high32_count_buf, &high32_count_le, sizeof(high32_count_le));++    return buf - initial_buf;+}++size_t roaring64_bitmap_portable_deserialize_size(const char *buf,+                                                  size_t maxbytes) {+    // https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+    if (buf == NULL) {+        return 0;+    }+    size_t read_bytes = 0;++    // Read as uint64 the distinct number of "buckets", where a bucket is+    // defined as the most significant 32 bits of an element.+    uint64_t buckets;+    if (read_bytes + sizeof(buckets) > maxbytes) {+        return 0;+    }+    memcpy(&buckets, buf, sizeof(buckets));+    buckets = croaring_letoh64(buckets);+    buf += sizeof(buckets);+    read_bytes += sizeof(buckets);++    // Buckets should be 32 bits with 4 bits of zero padding.+    if (buckets > UINT32_MAX) {+        return 0;+    }++    // Iterate through buckets ordered by increasing keys.+    for (uint64_t bucket = 0; bucket < buckets; ++bucket) {+        // Read as uint32 the most significant 32 bits of the bucket.+        uint32_t high32;+        if (read_bytes + sizeof(high32) > maxbytes) {+            return 0;+        }+        buf += sizeof(high32);+        read_bytes += sizeof(high32);++        // Read the 32-bit Roaring bitmaps representing the least+        // significant bits of a set of elements.+        size_t bitmap32_size = roaring_bitmap_portable_deserialize_size(+            buf, maxbytes - read_bytes);+        if (bitmap32_size == 0) {+            return 0;+        }+        buf += bitmap32_size;+        read_bytes += bitmap32_size;+    }+    return read_bytes;+}++roaring64_bitmap_t *roaring64_bitmap_portable_deserialize_safe(+    const char *buf, size_t maxbytes) {+    // https://github.com/RoaringBitmap/RoaringFormatSpec#extension-for-64-bit-implementations+    if (buf == NULL) {+        return NULL;+    }+    size_t read_bytes = 0;++    // Read as uint64 the distinct number of "buckets", where a bucket is+    // defined as the most significant 32 bits of an element.+    uint64_t buckets;+    if (read_bytes + sizeof(buckets) > maxbytes) {+        return NULL;+    }+    memcpy(&buckets, buf, sizeof(buckets));+    buckets = croaring_letoh64(buckets);+    buf += sizeof(buckets);+    read_bytes += sizeof(buckets);++    // Buckets should be 32 bits with 4 bits of zero padding.+    if (buckets > UINT32_MAX) {+        return NULL;+    }++    roaring64_bitmap_t *r = roaring64_bitmap_create();+    // Iterate through buckets ordered by increasing keys.+    int64_t previous_high32 = -1;+    for (uint64_t bucket = 0; bucket < buckets; ++bucket) {+        // Read as uint32 the most significant 32 bits of the bucket.+        uint32_t high32;+        if (read_bytes + sizeof(high32) > maxbytes) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        memcpy(&high32, buf, sizeof(high32));+        high32 = croaring_letoh32(high32);+        buf += sizeof(high32);+        read_bytes += sizeof(high32);+        // High 32 bits must be strictly increasing.+        if (high32 <= previous_high32) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        previous_high32 = high32;++        // Read the 32-bit Roaring bitmaps representing the least+        // significant bits of a set of elements. ra_portable_deserialize+        // already reports bytes consumed, so we do not walk the 32-bit+        // format a second time with deserialize_size.+        roaring_bitmap_t *bitmap32 =+            (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t));+        if (bitmap32 == NULL) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        size_t bytesread = 0;+        bool is_ok = ra_portable_deserialize(&bitmap32->high_low_container, buf,+                                             maxbytes - read_bytes, &bytesread);+        if (!is_ok) {+            roaring_free(bitmap32);+            roaring64_bitmap_free(r);+            return NULL;+        }+        buf += bytesread;+        read_bytes += bytesread;++        // While we don't attempt to validate much, we must ensure that there+        // is no duplication in the high 48 bits - inserting into the ART+        // assumes (or UB) no duplicate keys. The top 32 bits must be unique+        // because we check for strict increasing values of  high32, but we+        // must also ensure the top 16 bits within each 32-bit bitmap are also+        // at least unique (we ensure they're strictly increasing as well,+        // which they must be for a _valid_ bitmap, since it's cheaper to check)+        int32_t last_bitmap_key = -1;+        for (int i = 0; i < bitmap32->high_low_container.size; i++) {+            uint16_t key = bitmap32->high_low_container.keys[i];+            if (key <= last_bitmap_key) {+                roaring_bitmap_free(bitmap32);+                roaring64_bitmap_free(r);+                return NULL;+            }+            last_bitmap_key = key;+        }++        // Insert all containers of the 32-bit bitmap into the 64-bit bitmap.+        move_from_roaring32_offset(r, bitmap32, high32);+        roaring_bitmap_free(bitmap32);+    }+    return r;+}++// Returns an "element count" for the given container. This has a different+// meaning for each container type, but the purpose is the minimal information+// required to serialize the container metadata.+static inline uint32_t container_get_element_count(const container_t *c,+                                                   uint8_t typecode) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            return ((bitset_container_t *)c)->cardinality;+        }+        case ARRAY_CONTAINER_TYPE: {+            return ((array_container_t *)c)->cardinality;+        }+        case RUN_CONTAINER_TYPE: {+            return ((run_container_t *)c)->n_runs;+        }+        default: {+            assert(false);+            roaring_unreachable;+            return 0;+        }+    }+}++static inline size_t container_get_frozen_size(const container_t *c,+                                               uint8_t typecode) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            return BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+        }+        case ARRAY_CONTAINER_TYPE: {+            return container_get_element_count(c, typecode) * sizeof(uint16_t);+        }+        case RUN_CONTAINER_TYPE: {+            return container_get_element_count(c, typecode) * sizeof(rle16_t);+        }+        default: {+            assert(false);+            roaring_unreachable;+            return 0;+        }+    }+}++uint64_t align_size(uint64_t size, uint64_t alignment) {+    return (size + alignment - 1) & ~(alignment - 1);+}++size_t roaring64_bitmap_frozen_size_in_bytes(const roaring64_bitmap_t *r) {+    if (!is_shrunken(r)) {+        return 0;+    }+    // Flags.+    uint64_t size = sizeof(r->flags);+    // Container count.+    size += sizeof(r->capacity);+    // Container element counts.+    size += r->capacity * sizeof(uint16_t);+    // Total container sizes.+    size += 3 * sizeof(uint64_t);+    // ART (8 byte aligned).+    size = align_size(size, 8);+    size += art_size_in_bytes(&r->art);++    uint64_t total_sizes[4] =+        CROARING_ZERO_INITIALIZER;  // Indexed by typecode.+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);+        total_sizes[typecode] +=+            container_get_frozen_size(get_container(r, leaf), typecode);+        art_iterator_next(&it);+    }+    // Containers (aligned).+    size = align_size(size, CROARING_BITSET_ALIGNMENT);+    size += total_sizes[BITSET_CONTAINER_TYPE];+    size = align_size(size, alignof(rle16_t));+    size += total_sizes[ARRAY_CONTAINER_TYPE];+    size = align_size(size, alignof(uint16_t));+    size += total_sizes[RUN_CONTAINER_TYPE];+    // Padding to make overall size a multiple of required alignment.+    size = align_size(size, CROARING_BITSET_ALIGNMENT);+    return size;+}++// The output cursors are byte pointers rather than typed pointers: the output+// buffer itself need not be aligned (only `roaring64_bitmap_frozen_view`+// requires alignment), and forming a misaligned typed pointer is undefined+// behavior even if it is only ever passed to memcpy.+static inline void container_frozen_serialize(const container_t *container,+                                              uint8_t typecode, char **bitsets,+                                              char **arrays, char **runs) {+    size_t size = container_get_frozen_size(container, typecode);+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            bitset_container_t *bitset = (bitset_container_t *)container;+            memcpy(*bitsets, bitset->words, size);+            *bitsets += size;+            break;+        }+        case ARRAY_CONTAINER_TYPE: {+            array_container_t *array = (array_container_t *)container;+            memcpy(*arrays, array->array, size);+            *arrays += size;+            break;+        }+        case RUN_CONTAINER_TYPE: {+            run_container_t *run = (run_container_t *)container;+            memcpy(*runs, run->runs, size);+            *runs += size;+            break;+        }+        default: {+            assert(false);+            roaring_unreachable;+        }+    }+}++static inline char *pad_align(char *buf, const char *initial_buf,+                              size_t alignment) {+    uint64_t buf_size = buf - initial_buf;+    uint64_t pad = align_size(buf_size, alignment) - buf_size;+    memset(buf, 0, pad);+    return buf + pad;+}++size_t roaring64_bitmap_frozen_serialize(const roaring64_bitmap_t *r,+                                         char *buf) {+    if (buf == NULL) {+        return 0;+    }+    if (!is_shrunken(r)) {+        return 0;+    }+    const char *initial_buf = buf;++    // Flags.+    memcpy(buf, &r->flags, sizeof(r->flags));+    buf += sizeof(r->flags);++    // Container count.+    memcpy(buf, &r->capacity, sizeof(r->capacity));+    buf += sizeof(r->capacity);++    // Container element counts.+    uint64_t total_sizes[4] =+        CROARING_ZERO_INITIALIZER;  // Indexed by typecode.+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);+        container_t *container = get_container(r, leaf);++        uint32_t elem_count = container_get_element_count(container, typecode);+        uint16_t compressed_elem_count = (uint16_t)(elem_count - 1);+        memcpy(buf, &compressed_elem_count, sizeof(compressed_elem_count));+        buf += sizeof(compressed_elem_count);++        total_sizes[typecode] += container_get_frozen_size(container, typecode);+        art_iterator_next(&it);+    }++    // Total container sizes.+    memcpy(buf, &(total_sizes[BITSET_CONTAINER_TYPE]), sizeof(uint64_t));+    buf += sizeof(uint64_t);+    memcpy(buf, &(total_sizes[RUN_CONTAINER_TYPE]), sizeof(uint64_t));+    buf += sizeof(uint64_t);+    memcpy(buf, &(total_sizes[ARRAY_CONTAINER_TYPE]), sizeof(uint64_t));+    buf += sizeof(uint64_t);++    // ART.+    buf = pad_align(buf, initial_buf, 8);+    buf += art_serialize(&r->art, buf);++    // Containers (aligned).+    // Runs before arrays as run elements are larger than array elements and+    // smaller than bitset elements.+    buf = pad_align(buf, initial_buf, CROARING_BITSET_ALIGNMENT);+    char *bitsets = buf;+    buf += total_sizes[BITSET_CONTAINER_TYPE];+    buf = pad_align(buf, initial_buf, alignof(rle16_t));+    char *runs = buf;+    buf += total_sizes[RUN_CONTAINER_TYPE];+    buf = pad_align(buf, initial_buf, alignof(uint16_t));+    char *arrays = buf;+    buf += total_sizes[ARRAY_CONTAINER_TYPE];++    it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);+        container_t *container = get_container(r, leaf);+        container_frozen_serialize(container, typecode, &bitsets, &arrays,+                                   &runs);+        art_iterator_next(&it);+    }++    // Padding to make overall size a multiple of required alignment.+    buf = pad_align(buf, initial_buf, CROARING_BITSET_ALIGNMENT);++    return buf - initial_buf;+}++static roaring64_bitmap_t *alloc_frozen_bitmap(+    uint64_t capacity, frozen_container_header_t **headers_out) {+    if (capacity > SIZE_MAX / sizeof(frozen_container_header_t)) {+        return NULL;+    }+    size_t ptrs = (size_t)capacity * sizeof(container_t *);+    size_t codes = (size_t)capacity * sizeof(uint8_t);+    size_t headers = (size_t)capacity * sizeof(frozen_container_header_t);+    size_t pad = alignof(container_t *) + alignof(frozen_container_header_t);+    if (sizeof(roaring64_bitmap_t) > SIZE_MAX - ptrs ||+        sizeof(roaring64_bitmap_t) + ptrs > SIZE_MAX - codes ||+        sizeof(roaring64_bitmap_t) + ptrs + codes > SIZE_MAX - headers ||+        sizeof(roaring64_bitmap_t) + ptrs + codes + headers > SIZE_MAX - pad) {+        return NULL;+    }+    size_t sz = sizeof(roaring64_bitmap_t) + ptrs + codes + headers + pad;+    char *base = (char *)roaring_malloc(sz);+    if (base == NULL) {+        return NULL;+    }+    char *cursor = base;+    roaring64_bitmap_t *r = (roaring64_bitmap_t *)roaring64_arena_alloc(+        &cursor, sizeof(roaring64_bitmap_t));+    art_init_cleared(&r->art);+    r->flags = ROARING_FLAG_FROZEN;+    r->capacity = capacity;+    r->first_free = 0;+    r->end = 0;+    r->holes = NULL;+    r->num_holes = 0;+    r->holes_capacity = 0;+    cursor = roaring64_arena_pad(cursor, base, alignof(container_t *));+    if (capacity == 0) {+        r->containers = NULL;+        r->typecodes = NULL;+        if (headers_out != NULL) {+            *headers_out = NULL;+        }+        return r;+    }+    r->containers = (container_t **)roaring64_arena_alloc(&cursor, ptrs);+    memset(r->containers, 0, ptrs);+    r->typecodes = (uint8_t *)roaring64_arena_alloc(&cursor, codes);+    cursor =+        roaring64_arena_pad(cursor, base, alignof(frozen_container_header_t));+    frozen_container_header_t *hdrs =+        (frozen_container_header_t *)roaring64_arena_alloc(&cursor, headers);+    if (headers_out != NULL) {+        *headers_out = hdrs;+    }+    return r;+}++static container_t *container_frozen_view_at(+    frozen_container_header_t *header, uint8_t typecode, uint32_t elem_count,+    const uint64_t **bitsets, const uint16_t **arrays, const rle16_t **runs) {+    switch (typecode) {+        case BITSET_CONTAINER_TYPE: {+            bitset_container_t *c = &header->bitset;+            c->cardinality = elem_count;+            c->words = (uint64_t *)*bitsets;+            *bitsets += BITSET_CONTAINER_SIZE_IN_WORDS;+            return (container_t *)c;+        }+        case ARRAY_CONTAINER_TYPE: {+            array_container_t *c = &header->array;+            c->cardinality = elem_count;+            c->capacity = elem_count;+            c->array = (uint16_t *)*arrays;+            *arrays += elem_count;+            return (container_t *)c;+        }+        case RUN_CONTAINER_TYPE: {+            run_container_t *c = &header->run;+            c->n_runs = elem_count;+            c->capacity = elem_count;+            c->runs = (rle16_t *)*runs;+            *runs += elem_count;+            return (container_t *)c;+        }+        default:+            return NULL;+    }+}++roaring64_bitmap_t *roaring64_bitmap_frozen_view(const char *buf,+                                                 size_t maxbytes) {+    if (buf == NULL) {+        return NULL;+    }+    if ((uintptr_t)buf % CROARING_BITSET_ALIGNMENT != 0) {+        return NULL;+    }++    uint8_t flags;+    uint64_t capacity;+    if (maxbytes < sizeof(flags) + sizeof(capacity)) {+        return NULL;+    }+    memcpy(&flags, buf, sizeof(flags));+    buf += sizeof(flags);+    maxbytes -= sizeof(flags);+    memcpy(&capacity, buf, sizeof(capacity));+    buf += sizeof(capacity);+    maxbytes -= sizeof(capacity);++    // The element counts alone need two bytes per container, so a capacity+    // larger than that cannot be satisfied by this buffer. Checked before+    // allocating, so a short buffer claiming a huge count cannot make us+    // reserve (and clear) an arena sized from attacker-controlled bytes.+    if (capacity > maxbytes / sizeof(uint16_t)) {+        return NULL;+    }++    frozen_container_header_t *headers = NULL;+    roaring64_bitmap_t *r = alloc_frozen_bitmap(capacity, &headers);+    if (r == NULL) {+        return NULL;+    }+    // Only flags the format actually defines; the byte comes from the buffer.+    r->flags = (uint8_t)(flags & ROARING_FLAG_COW) | ROARING_FLAG_FROZEN |+               ROARING_FLAG_FROZEN_ART;++    // Container element counts.+    if (maxbytes < capacity * sizeof(uint16_t)) {+        roaring64_bitmap_free(r);+        return NULL;+    }+    const char *elem_counts = buf;+    buf += capacity * sizeof(uint16_t);+    maxbytes -= capacity * sizeof(uint16_t);++    // Total container sizes.+    uint64_t total_sizes[4];+    if (maxbytes < sizeof(uint64_t) * 3) {+        roaring64_bitmap_free(r);+        return NULL;+    }+    memcpy(&(total_sizes[BITSET_CONTAINER_TYPE]), buf, sizeof(uint64_t));+    buf += sizeof(uint64_t);+    maxbytes -= sizeof(uint64_t);+    memcpy(&(total_sizes[RUN_CONTAINER_TYPE]), buf, sizeof(uint64_t));+    buf += sizeof(uint64_t);+    maxbytes -= sizeof(uint64_t);+    memcpy(&(total_sizes[ARRAY_CONTAINER_TYPE]), buf, sizeof(uint64_t));+    buf += sizeof(uint64_t);+    maxbytes -= sizeof(uint64_t);++    // ART (8 byte aligned).+    buf = CROARING_ALIGN_BUF(buf, 8);+    size_t art_size = art_frozen_view(buf, maxbytes, &r->art);+    if (art_size == 0) {+        roaring64_bitmap_free(r);+        return NULL;+    }+    buf += art_size;+    maxbytes -= art_size;++    // Containers (aligned).+    const char *before_containers = buf;+    buf = CROARING_ALIGN_BUF(buf, CROARING_BITSET_ALIGNMENT);+    const uint64_t *bitsets = (const uint64_t *)buf;+    buf += total_sizes[BITSET_CONTAINER_TYPE];+    buf = CROARING_ALIGN_BUF(buf, alignof(rle16_t));+    const rle16_t *runs = (const rle16_t *)buf;+    buf += total_sizes[RUN_CONTAINER_TYPE];+    buf = CROARING_ALIGN_BUF(buf, alignof(uint16_t));+    const uint16_t *arrays = (const uint16_t *)buf;+    buf += total_sizes[ARRAY_CONTAINER_TYPE];+    if (maxbytes < (uint64_t)(buf - before_containers)) {+        roaring64_bitmap_free(r);+        return NULL;+    }+    maxbytes -= buf - before_containers;++    // Deserialize in ART iteration order.+    art_iterator_t it = art_init_iterator(&r->art, /*first=*/true);+    for (size_t i = 0; it.value != NULL; ++i) {+        if (i >= capacity) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        leaf_t leaf = (leaf_t)*it.value;+        uint8_t typecode = get_typecode(leaf);++        uint16_t compressed_elem_count;+        memcpy(&compressed_elem_count, elem_counts + (i * sizeof(uint16_t)),+               sizeof(compressed_elem_count));+        uint32_t elem_count = (uint32_t)(compressed_elem_count) + 1;++        // The container index is unrelated to the iteration order.+        uint64_t index = get_index(leaf);+        if (index >= capacity) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        container_t *c = container_frozen_view_at(+            headers + index, typecode, elem_count, &bitsets, &arrays, &runs);+        if (c == NULL) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        set_container_at(r, index, c, typecode);++        art_iterator_next(&it);+    }++    // Padding to make overall size a multiple of required alignment.+    buf = CROARING_ALIGN_BUF(buf, CROARING_BITSET_ALIGNMENT);++    // A frozen view is read-only: mark every slot as taken.+    r->first_free = r->capacity;+    r->end = r->capacity;+    return r;+}++#if !CROARING_IS_BIG_ENDIAN+// Only used by roaring64_bitmap_portable_deserialize_frozen, which refuses to+// run on big-endian systems.+static bool view_one_portable32(roaring64_bitmap_t *r,+                                frozen_container_header_t *headers,+                                uint32_t high32, const char *buf,+                                size_t maxbytes, size_t *consumed) {+    *consumed = roaring_bitmap_portable_deserialize_size(buf, maxbytes);+    if (*consumed == 0 || *consumed > maxbytes) {+        return false;+    }+    const char *start = buf;+    size_t remaining = *consumed;++    uint32_t cookie;+    memcpy(&cookie, buf, sizeof(cookie));+    cookie = croaring_letoh32(cookie);+    buf += sizeof(cookie);+    remaining -= sizeof(cookie);++    int32_t num_containers;+    const char *run_flag_bitset = NULL;+    bool hasrun = false;+    bool has_offsets;++    if (cookie == SERIAL_COOKIE_NO_RUNCONTAINER) {+        if (remaining < sizeof(uint32_t)) {+            return false;+        }+        uint32_t n_le;+        memcpy(&n_le, buf, sizeof(n_le));+        num_containers = (int32_t)croaring_letoh32(n_le);+        buf += sizeof(uint32_t);+        remaining -= sizeof(uint32_t);+        has_offsets = true;+    } else if ((cookie & 0xFFFF) == SERIAL_COOKIE) {+        num_containers = (int32_t)(cookie >> 16) + 1;+        hasrun = true;+        int32_t run_flag_bitset_size = (num_containers + 7) / 8;+        if (num_containers < 0 || remaining < (size_t)run_flag_bitset_size) {+            return false;+        }+        run_flag_bitset = buf;+        buf += run_flag_bitset_size;+        remaining -= (size_t)run_flag_bitset_size;+        has_offsets = num_containers >= NO_OFFSET_THRESHOLD;+    } else {+        return false;+    }+    if (num_containers < 0 || num_containers > (1 << 16)) {+        return false;+    }++    size_t desc_bytes = (size_t)num_containers * 2 * sizeof(uint16_t);+    if (remaining < desc_bytes) {+        return false;+    }+    const char *keyscards = buf;+    buf += desc_bytes;+    remaining -= desc_bytes;++    const char *offset_bytes = NULL;+    if (has_offsets) {+        size_t off_bytes = (size_t)num_containers * sizeof(uint32_t);+        if (remaining < off_bytes) {+            return false;+        }+        offset_bytes = buf;+        buf += off_bytes;+        remaining -= off_bytes;+    }++    int32_t last_key = -1;+    uint64_t key_base = ((uint64_t)high32) << 32;++    for (int32_t i = 0; i < num_containers; ++i) {+        uint16_t key, card_m1;+        memcpy(&key, keyscards + 4 * (size_t)i, sizeof(key));+        key = croaring_letoh16(key);+        memcpy(&card_m1, keyscards + 4 * (size_t)i + 2, sizeof(card_m1));+        card_m1 = croaring_letoh16(card_m1);+        if ((int32_t)key <= last_key) {+            return false;+        }+        last_key = (int32_t)key;++        uint32_t cardinality = (uint32_t)card_m1 + 1;+        bool isbitmap = cardinality > DEFAULT_MAX_SIZE;+        bool isrun = false;+        if (hasrun && (run_flag_bitset[i / 8] & (1 << (i % 8))) != 0) {+            isbitmap = false;+            isrun = true;+        }++        const char *payload;+        if (offset_bytes != NULL) {+            uint32_t off;+            memcpy(&off, offset_bytes + (size_t)i * sizeof(uint32_t),+                   sizeof(off));+            off = croaring_letoh32(off);+            if ((size_t)off >= *consumed) {+                return false;+            }+            payload = start + off;+        } else {+            payload = buf;+        }++        uint8_t typecode;+        size_t payload_size;+        if (isbitmap) {+            typecode = BITSET_CONTAINER_TYPE;+            payload_size = BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+        } else if (isrun) {+            typecode = RUN_CONTAINER_TYPE;+            if ((size_t)(payload - start) + sizeof(uint16_t) > *consumed) {+                return false;+            }+            uint16_t n_runs;+            memcpy(&n_runs, payload, sizeof(n_runs));+            n_runs = croaring_letoh16(n_runs);+            payload_size = sizeof(uint16_t) + (size_t)n_runs * sizeof(rle16_t);+        } else {+            typecode = ARRAY_CONTAINER_TYPE;+            payload_size = (size_t)cardinality * sizeof(uint16_t);+        }+        if ((size_t)(payload - start) + payload_size > *consumed) {+            return false;+        }++        if (r->first_free >= r->capacity) {+            return false;+        }+        uint64_t index = allocate_index(r);+        frozen_container_header_t *header = headers + index;+        container_t *c;+        if (isbitmap) {+            header->bitset.cardinality = (int32_t)cardinality;+            header->bitset.words = (uint64_t *)payload;+            c = (container_t *)&header->bitset;+        } else if (isrun) {+            uint16_t n_runs;+            memcpy(&n_runs, payload, sizeof(n_runs));+            n_runs = croaring_letoh16(n_runs);+            header->run.n_runs = n_runs;+            header->run.capacity = n_runs;+            header->run.runs = (rle16_t *)(payload + sizeof(uint16_t));+            c = (container_t *)&header->run;+        } else {+            header->array.cardinality = (int32_t)cardinality;+            header->array.capacity = (int32_t)cardinality;+            header->array.array = (uint16_t *)payload;+            c = (container_t *)&header->array;+        }+        set_container_at(r, index, c, typecode);++        uint8_t high48[ART_KEY_BYTES];+        uint64_t high48_bits = key_base | ((uint64_t)key << 16);+        split_key(high48_bits, high48);+        art_insert(&r->art, high48, (art_val_t)create_leaf(index, typecode));++        if (offset_bytes == NULL) {+            buf += payload_size;+            remaining -= payload_size;+        }+    }+    return true;+}+#endif  // !CROARING_IS_BIG_ENDIAN++CROARING_ALLOW_UNALIGNED+roaring64_bitmap_t *roaring64_bitmap_portable_deserialize_frozen(+    const char *buf, size_t maxbytes) {+    if (buf == NULL) {+        return NULL;+    }+#if CROARING_IS_BIG_ENDIAN+    // The portable format is little-endian and this function uses the payload+    // bytes where they sit, so there is no correct view of them here. Refuse+    // rather than hand back a bitmap that silently reads byte-swapped values.+    (void)maxbytes;+    return NULL;+#else+    size_t remaining = maxbytes;++    if (remaining < sizeof(uint64_t)) {+        return NULL;+    }+    uint64_t buckets;+    memcpy(&buckets, buf, sizeof(buckets));+    buckets = croaring_letoh64(buckets);+    buf += sizeof(buckets);+    remaining -= sizeof(buckets);+    if (buckets > UINT32_MAX) {+        return NULL;+    }++    uint64_t ncontainers = 0;+    const char *count_buf = buf;+    size_t count_remaining = remaining;+    int64_t previous_high32 = -1;+    for (uint64_t bucket = 0; bucket < buckets; ++bucket) {+        if (count_remaining < sizeof(uint32_t)) {+            return NULL;+        }+        uint32_t high32;+        memcpy(&high32, count_buf, sizeof(high32));+        high32 = croaring_letoh32(high32);+        count_buf += sizeof(high32);+        count_remaining -= sizeof(high32);+        if (high32 <= previous_high32) {+            return NULL;+        }+        previous_high32 = high32;+        size_t bitmap32_size = roaring_bitmap_portable_deserialize_size(+            count_buf, count_remaining);+        if (bitmap32_size == 0) {+            return NULL;+        }+        uint32_t cookie;+        if (count_remaining < sizeof(cookie)) {+            return NULL;+        }+        memcpy(&cookie, count_buf, sizeof(cookie));+        cookie = croaring_letoh32(cookie);+        int32_t size;+        if ((cookie & 0xFFFF) == SERIAL_COOKIE) {+            size = (int32_t)(cookie >> 16) + 1;+        } else if (cookie == SERIAL_COOKIE_NO_RUNCONTAINER) {+            if (count_remaining < 2 * sizeof(uint32_t)) {+                return NULL;+            }+            uint32_t size_le;+            memcpy(&size_le, count_buf + sizeof(uint32_t), sizeof(size_le));+            size = (int32_t)croaring_letoh32(size_le);+        } else {+            return NULL;+        }+        if (size < 0) {+            return NULL;+        }+        ncontainers += (uint64_t)size;+        count_buf += bitmap32_size;+        count_remaining -= bitmap32_size;+    }++    frozen_container_header_t *headers = NULL;+    roaring64_bitmap_t *r = alloc_frozen_bitmap(ncontainers, &headers);+    if (r == NULL) {+        return NULL;+    }+    // ART is owned; payloads alias buf.+    r->flags = ROARING_FLAG_FROZEN;++    previous_high32 = -1;+    for (uint64_t bucket = 0; bucket < buckets; ++bucket) {+        if (remaining < sizeof(uint32_t)) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        uint32_t high32;+        memcpy(&high32, buf, sizeof(high32));+        high32 = croaring_letoh32(high32);+        buf += sizeof(high32);+        remaining -= sizeof(high32);+        if (high32 <= previous_high32) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        previous_high32 = high32;++        size_t consumed = 0;+        if (!view_one_portable32(r, headers, high32, buf, remaining,+                                 &consumed)) {+            roaring64_bitmap_free(r);+            return NULL;+        }+        buf += consumed;+        remaining -= consumed;+    }+    return r;+#endif+}++bool roaring64_bitmap_iterate(const roaring64_bitmap_t *r,+                              roaring_iterator64 iterator, void *ptr) {+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        uint64_t high48 = combine_key(it.key, 0);+        uint64_t high32 = high48 & 0xFFFFFFFF00000000ULL;+        uint32_t low32 = high48;+        leaf_t leaf = (leaf_t)*it.value;+        if (!container_iterate64(get_container(r, leaf), get_typecode(leaf),+                                 low32, iterator, high32, ptr)) {+            return false;+        }+        art_iterator_next(&it);+    }+    return true;+}++// Whether bitset_container_to_uint32_array picks a vectorized decoder for+// `bc` (keep in sync with it). Only then does decoding to 32 bits and widening+// beat decoding directly to 64 bits.+static inline bool roaring64_bitset_decodes_vectorized(+    const bitset_container_t *bc) {+#if CROARING_IS_X64+    int support = croaring_hardware_support();+#if CROARING_COMPILER_SUPPORTS_AVX512+    if (support & ROARING_SUPPORTS_AVX512) {+        return true;+    }+#endif+    return (support & ROARING_SUPPORTS_AVX2) && bc->cardinality >= 8192;+#else+    (void)bc;+    return false;+#endif+}++// The containers of a portable frozen view may be unaligned: see+// roaring64_bitmap_portable_deserialize_frozen.+CROARING_ALLOW_UNALIGNED+static uint64_t *roaring64_bitset_to_uint64_array(const bitset_container_t *bc,+                                                  uint64_t high48,+                                                  uint64_t *out) {+    for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i) {+        uint64_t w = bc->words[i];+        uint64_t base = high48 + (uint64_t)i * 64;+        while (w != 0) {+            *out++ = base + roaring_trailing_zeroes(w);+            w &= w - 1;+        }+    }+    return out;+}++static void roaring64_bitmap_to_uint64_array_iterator(+    const roaring64_bitmap_t *r, uint64_t *out) {+    roaring64_iterator_t it;  // gets initialized in the next line+    roaring64_iterator_init_at(r, &it, /*first=*/true);+    roaring64_iterator_read(&it, out, UINT64_MAX);+}++CROARING_ALLOW_UNALIGNED+void roaring64_bitmap_to_uint64_array(const roaring64_bitmap_t *r,+                                      uint64_t *out) {+    // Where vectorized 32-bit bitset decoders are available, decoding into a+    // scratch buffer and widening is several times faster than decoding value+    // by value.+    uint64_t *const start = out;+    uint32_t *scratch = NULL;+    art_iterator_t it = art_init_iterator((art_t *)&r->art, /*first=*/true);+    while (it.value != NULL) {+        leaf_t leaf = (leaf_t)*it.value;+        uint64_t high48 = combine_key(it.key, 0);+        uint8_t typecode = get_typecode(leaf);+        const container_t *c = get_container(r, leaf);+        switch (typecode) {+            case BITSET_CONTAINER_TYPE: {+                const bitset_container_t *bc = const_CAST_bitset(c);+                if (!roaring64_bitset_decodes_vectorized(bc)) {+                    out = roaring64_bitset_to_uint64_array(bc, high48, out);+                    break;+                }+                if (scratch == NULL) {+                    scratch = (uint32_t *)roaring_malloc((1 << 16) *+                                                         sizeof(uint32_t));+                    if (scratch == NULL) {+                        roaring64_bitmap_to_uint64_array_iterator(r, start);+                        return;+                    }+                }+                int n = bitset_container_to_uint32_array(scratch, bc, 0);+                for (int i = 0; i < n; ++i) {+                    out[i] = high48 | scratch[i];+                }+                out += n;+                break;+            }+            case ARRAY_CONTAINER_TYPE: {+                const array_container_t *ac = const_CAST_array(c);+                for (int32_t i = 0; i < ac->cardinality; ++i) {+                    out[i] = high48 | ac->array[i];+                }+                out += ac->cardinality;+                break;+            }+            case RUN_CONTAINER_TYPE: {+                const run_container_t *rc = const_CAST_run(c);+                for (int32_t i = 0; i < rc->n_runs; ++i) {+                    uint64_t value = high48 | rc->runs[i].value;+                    for (uint32_t j = 0; j <= rc->runs[i].length; ++j) {+                        *out++ = value + j;+                    }+                }+                break;+            }+            default:+                assert(false);+                roaring_unreachable;+        }+        art_iterator_next(&it);+    }+    roaring_free(scratch);+}++roaring64_iterator_t *roaring64_iterator_create(const roaring64_bitmap_t *r) {+    roaring64_iterator_t *it =+        (roaring64_iterator_t *)roaring_malloc(sizeof(roaring64_iterator_t));+    return roaring64_iterator_init_at(r, it, /*first=*/true);+}++roaring64_iterator_t *roaring64_iterator_create_last(+    const roaring64_bitmap_t *r) {+    roaring64_iterator_t *it =+        (roaring64_iterator_t *)roaring_malloc(sizeof(roaring64_iterator_t));+    return roaring64_iterator_init_at(r, it, /*first=*/false);+}++void roaring64_iterator_reinit(const roaring64_bitmap_t *r,+                               roaring64_iterator_t *it) {+    roaring64_iterator_init_at(r, it, /*first=*/true);+}++void roaring64_iterator_reinit_last(const roaring64_bitmap_t *r,+                                    roaring64_iterator_t *it) {+    roaring64_iterator_init_at(r, it, /*first=*/false);+}++roaring64_iterator_t *roaring64_iterator_copy(const roaring64_iterator_t *it) {+    roaring64_iterator_t *new_it =+        (roaring64_iterator_t *)roaring_malloc(sizeof(roaring64_iterator_t));+    memcpy(new_it, it, sizeof(*it));+    return new_it;+}++void roaring64_iterator_free(roaring64_iterator_t *it) { roaring_free(it); }++CROARING_STATIC_ASSERT(offsetof(roaring64_iterator_t, pub) == 0,+                       "the public members must be first in the iterator");++extern inline bool roaring64_iterator_has_value(const roaring64_iterator_t *it);+extern inline uint64_t roaring64_iterator_value(const roaring64_iterator_t *it);++// The current container is exhausted: step the ART to the next leaf.+static inline bool roaring64_iterator_next_leaf(roaring64_iterator_t *it) {+    if (art_iterator_next(&it->art_it)) {+        return roaring64_iterator_init_at_leaf_first(it);+    }+    it->saturated_forward = true;+    it->fast_type = 0;+    return (it->pub.has_value = false);+}++// Everything `advance` needs when the bitset cache does not describe where the+// iterator is: a restart, or a step through the container iterator.+static inline bool roaring64_iterator_advance_slow(roaring64_iterator_t *it) {+    if (it->art_it.value == NULL) {+        if (it->saturated_forward) {+            return (it->pub.has_value = false);+        }+        roaring64_iterator_init_at(it->r, it, /*first=*/true);+        return it->pub.has_value;+    }+    leaf_t leaf = (leaf_t)*it->art_it.value;+    uint8_t typecode = get_typecode(leaf);+    uint16_t low16 = (uint16_t)it->pub.value;+    if (container_iterator_next(get_container(it->r, leaf), typecode,+                                &it->container_it, &low16)) {+        it->pub.value = it->high48 | low16;+        it->pub.has_value = true;+        if (typecode == BITSET_CONTAINER_TYPE) {+            // Only reached when the cache was stale, i.e. something else moved+            // the cursor. Array and run containers never have a cache, and+            // priming them would cost a leaf load per value.+            roaring64_iterator_prime(it);+        }+        return true;+    }+    return roaring64_iterator_next_leaf(it);+}++bool roaring64_iterator_advance(roaring64_iterator_t *it) {+    // Matching both the leaf and `container_it.index` is enough to know the+    // cache describes where the iterator actually is: within a container the+    // index determines the position. Anything that moved the cursor --+    // `previous`, `move_equalorlarger`, any of the `read` variants -- moved one+    // of them, and drops through to the slow path, which rebuilds the cache.+    if (it->fast_type != 0 && it->fast_art_value == it->art_it.value &&+        it->fast_index == it->container_it.index) {+        uint64_t word = it->fast_word;+        uint32_t wi = it->fast_wordindex;+        while (word == 0) {+            if (++wi >= BITSET_CONTAINER_SIZE_IN_WORDS) {+                return roaring64_iterator_next_leaf(it);+            }+            word = it->fast_words[wi];+            it->fast_wordindex = wi;+        }+        uint32_t index = wi * 64 + (uint32_t)roaring_trailing_zeroes(word);+        it->fast_word = word & (word - 1);+        it->fast_index = (int32_t)index;+        it->container_it.index = (int32_t)index;+        it->pub.value = it->high48 | index;+        return (it->pub.has_value = true);+    }+    return roaring64_iterator_advance_slow(it);+}++bool roaring64_iterator_previous(roaring64_iterator_t *it) {+    if (it->art_it.value == NULL) {+        if (!it->saturated_forward) {+            // Saturated backward.+            return (it->pub.has_value = false);+        }+        roaring64_iterator_init_at(it->r, it, /*first=*/false);+        return it->pub.has_value;+    }+    leaf_t leaf = (leaf_t)*it->art_it.value;+    uint16_t low16 = (uint16_t)it->pub.value;+    if (container_iterator_prev(get_container(it->r, leaf), get_typecode(leaf),+                                &it->container_it, &low16)) {+        it->pub.value = it->high48 | low16;+        it->pub.has_value = true;+        it->fast_type = 0;+        return true;+    }+    if (art_iterator_prev(&it->art_it)) {+        return roaring64_iterator_init_at_leaf_last(it);+    }+    it->saturated_forward = false;  // Saturated backward.+    it->fast_type = 0;+    return (it->pub.has_value = false);+}++bool roaring64_iterator_move_equalorlarger(roaring64_iterator_t *it,+                                           uint64_t val) {+    uint8_t val_high48[ART_KEY_BYTES];+    uint16_t val_low16 = split_key(val, val_high48);+    if (!it->pub.has_value || it->high48 != (val & 0xFFFFFFFFFFFF0000)) {+        // The ART iterator is before or after the high48 bits of `val` (or+        // beyond the ART altogether), so we need to move to a leaf with a+        // key equal or greater.+        if (!art_iterator_lower_bound(&it->art_it, val_high48)) {+            // Only smaller keys found.+            it->saturated_forward = true;+            it->fast_type = 0;+            return (it->pub.has_value = false);+        }+        it->high48 = combine_key(it->art_it.key, 0);+        // Fall through to the next if statement.+    }++    if (it->high48 == (val & 0xFFFFFFFFFFFF0000)) {+        // We're at equal high bits, check if a suitable value can be found+        // in this container.+        leaf_t leaf = (leaf_t)*it->art_it.value;+        uint16_t low16 = (uint16_t)it->pub.value;+        if (container_iterator_lower_bound(+                get_container(it->r, leaf), get_typecode(leaf),+                &it->container_it, &low16, val_low16)) {+            it->pub.value = it->high48 | low16;+            it->pub.has_value = true;+            it->fast_type = 0;+            return true;+        }+        // Only smaller entries in this container, move to the next.+        if (!art_iterator_next(&it->art_it)) {+            it->saturated_forward = true;+            it->fast_type = 0;+            return (it->pub.has_value = false);+        }+    }++    // We're at a leaf with high bits greater than `val`, so the first entry+    // in this container is our result.+    return roaring64_iterator_init_at_leaf_first(it);+}++uint64_t roaring64_iterator_read(roaring64_iterator_t *it, uint64_t *buf,+                                 uint64_t count) {+    uint64_t consumed = 0;+    while (it->pub.has_value && consumed < count) {+        uint32_t container_consumed;+        leaf_t leaf = (leaf_t)*it->art_it.value;+        uint16_t low16 = (uint16_t)it->pub.value;+        uint32_t container_count = UINT32_MAX;+        if (count - consumed < (uint64_t)UINT32_MAX) {+            container_count = count - consumed;+        }+        bool has_value = container_iterator_read_into_uint64(+            get_container(it->r, leaf), get_typecode(leaf), &it->container_it,+            it->high48, buf, container_count, &container_consumed, &low16);+        consumed += container_consumed;+        buf += container_consumed;+        if (has_value) {+            it->pub.has_value = true;+            it->pub.value = it->high48 | low16;+            it->fast_type = 0;+            assert(consumed == count);+            return consumed;+        }+        it->pub.has_value = art_iterator_next(&it->art_it);+        if (it->pub.has_value) {+            roaring64_iterator_init_at_leaf_first(it);+        } else {+            it->saturated_forward = true;+            it->fast_type = 0;+        }+    }+    return consumed;+}++uint64_t roaring64_iterator_read_backward(roaring64_iterator_t *it,+                                          uint64_t *buf, uint64_t count) {+    uint64_t consumed = 0;+    while (it->pub.has_value && consumed < count) {+        uint32_t container_consumed;+        leaf_t leaf = *it->art_it.value;+        uint16_t low16 = (uint16_t)it->pub.value;+        uint32_t container_count = UINT32_MAX;+        if (count - consumed < (uint64_t)UINT32_MAX) {+            container_count = count - consumed;+        }+        bool has_value = container_iterator_read_backward_into_uint64(+            get_container(it->r, leaf), get_typecode(leaf), &it->container_it,+            it->high48, buf, container_count, &container_consumed, &low16);+        consumed += container_consumed;+        buf += container_consumed;+        if (has_value) {+            it->pub.has_value = true;+            it->pub.value = it->high48 | low16;+            assert(consumed == count);+            return consumed;+        }+        it->pub.has_value = art_iterator_prev(&it->art_it);+        if (it->pub.has_value) {+            roaring64_iterator_init_at_leaf_last(it);+        } else {+            it->saturated_forward = false;+        }+    }+    return consumed;+}++size_t roaring64_iterator_read_ranges(roaring64_iterator_t *it,+                                      roaring64_range_closed_t *buf,+                                      size_t count) {+    size_t ret = 0;+    while (it->pub.has_value && ret < count) {+        buf[ret].min = it->pub.value;+        for (;;) {+            uint16_t low16 = (uint16_t)it->pub.value;+            leaf_t leaf = (leaf_t)*it->art_it.value;+            bool container_has_more;+            uint16_t run_end_low16 = container_iterator_find_run_end(+                get_container(it->r, leaf), get_typecode(leaf),+                &it->container_it, &low16, &container_has_more);+            buf[ret].max = it->high48 | run_end_low16;++            if (container_has_more) {+                it->pub.value = it->high48 | low16;+                break;+            }+            // Move to next leaf+            it->pub.has_value = art_iterator_next(&it->art_it);+            if (it->pub.has_value) {+                roaring64_iterator_init_at_leaf_first(it);+            } else {+                it->saturated_forward = true;+                break;+            }+            // Continue merging only if the run reached the container+            // boundary and the next leaf starts exactly at max+1.+            if (run_end_low16 != UINT16_MAX ||+                it->pub.value != buf[ret].max + 1) {+                break;+            }+        }+        ret++;+    }+    return ret;+}++size_t roaring64_iterator_read_prev_ranges(roaring64_iterator_t *it,+                                           roaring64_range_closed_t *buf,+                                           size_t count) {+    size_t ret = 0;+    while (it->pub.has_value && ret < count) {+        buf[ret].max = it->pub.value;+        for (;;) {+            uint16_t low16 = (uint16_t)it->pub.value;+            leaf_t leaf = (leaf_t)*it->art_it.value;+            bool container_has_more;+            uint16_t run_start_low16 = container_iterator_find_run_start(+                get_container(it->r, leaf), get_typecode(leaf),+                &it->container_it, &low16, &container_has_more);+            buf[ret].min = it->high48 | run_start_low16;++            if (container_has_more) {+                it->pub.value = it->high48 | low16;+                break;+            }+            // Move to previous leaf+            it->pub.has_value = art_iterator_prev(&it->art_it);+            if (it->pub.has_value) {+                roaring64_iterator_init_at_leaf_last(it);+            } else {+                it->saturated_forward = false;+                break;+            }+            // Continue merging only if the run reached the container+            // boundary and the previous leaf ends exactly at min-1.+            if (run_start_low16 != 0 || it->pub.value != buf[ret].min - 1) {+                break;+            }+        }+        ret++;+    }+    return ret;+}++#ifdef __cplusplus+}  // extern "C"+}  // namespace roaring+}  // namespace api+#endif
+ cbits/CRoaring/src/roaring_array.c view
@@ -0,0 +1,826 @@+#include <assert.h>+#include <inttypes.h>+#include <stdbool.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include <roaring/containers/bitset.h>+#include <roaring/containers/containers.h>+#include <roaring/memory.h>+#include <roaring/roaring_array.h>++#ifdef __cplusplus+extern "C" {+namespace roaring {+namespace internal {+#endif++// Convention: [0,ra->size) all elements are initialized+//  [ra->size, ra->allocation_size) is junk and contains nothing needing freeing++extern inline int32_t ra_get_size(const roaring_array_t *ra);+extern inline int32_t ra_get_index(const roaring_array_t *ra, uint16_t x);++extern inline container_t *ra_get_container_at_index(const roaring_array_t *ra,+                                                     uint16_t i,+                                                     uint8_t *typecode);++extern inline void ra_unshare_container_at_index(roaring_array_t *ra,+                                                 uint16_t i);++extern inline void ra_replace_key_and_container_at_index(roaring_array_t *ra,+                                                         int32_t i,+                                                         uint16_t key,+                                                         container_t *c,+                                                         uint8_t typecode);++extern inline void ra_set_container_at_index(const roaring_array_t *ra,+                                             int32_t i, container_t *c,+                                             uint8_t typecode);++static bool realloc_array(roaring_array_t *ra, int32_t new_capacity) {+    //+    // Note: not implemented using C's realloc(), because the memory layout is+    // Struct-of-Arrays vs. Array-of-Structs:+    // https://github.com/RoaringBitmap/CRoaring/issues/256++    if (new_capacity == 0) {+        roaring_free(ra->containers);+        ra->containers = NULL;+        ra->keys = NULL;+        ra->typecodes = NULL;+        ra->allocation_size = 0;+        return true;+    }+    const size_t memoryneeded =+        new_capacity *+        (sizeof(uint16_t) + sizeof(container_t *) + sizeof(uint8_t));+    void *bigalloc = roaring_malloc(memoryneeded);+    if (!bigalloc) return false;+    void *oldbigalloc = ra->containers;+    container_t **newcontainers = (container_t **)bigalloc;+    uint16_t *newkeys = (uint16_t *)(newcontainers + new_capacity);+    uint8_t *newtypecodes = (uint8_t *)(newkeys + new_capacity);+    assert((char *)(newtypecodes + new_capacity) ==+           (char *)bigalloc + memoryneeded);+    if (ra->size > 0) {+        memcpy(newcontainers, ra->containers, sizeof(container_t *) * ra->size);+        memcpy(newkeys, ra->keys, sizeof(uint16_t) * ra->size);+        memcpy(newtypecodes, ra->typecodes, sizeof(uint8_t) * ra->size);+    }+    ra->containers = newcontainers;+    ra->keys = newkeys;+    ra->typecodes = newtypecodes;+    ra->allocation_size = new_capacity;+    roaring_free(oldbigalloc);+    return true;+}++bool ra_init_with_capacity(roaring_array_t *new_ra, uint32_t cap) {+    if (!new_ra) return false;+    ra_init(new_ra);++    // Containers hold 64Ki elements, so 64Ki containers is enough to hold+    // `0x10000 * 0x10000` (all 2^32) elements+    if (cap > 0x10000) {+        cap = 0x10000;+    }++    if (cap > 0) {+        void *bigalloc = roaring_malloc(+            cap * (sizeof(uint16_t) + sizeof(container_t *) + sizeof(uint8_t)));+        if (bigalloc == NULL) return false;+        new_ra->containers = (container_t **)bigalloc;+        new_ra->keys = (uint16_t *)(new_ra->containers + cap);+        new_ra->typecodes = (uint8_t *)(new_ra->keys + cap);+        // Narrowing is safe because of above check+        new_ra->allocation_size = (int32_t)cap;+    }+    return true;+}++int ra_shrink_to_fit(roaring_array_t *ra) {+    int32_t size = ra->size;+    int32_t allocation_size = ra->allocation_size;+    if (allocation_size == size) return 0;  // nothing to do+    if (!realloc_array(ra, size)) {+        return 0;+    }+    return (allocation_size - size) *+           (sizeof(uint16_t) + sizeof(container_t *) + sizeof(uint8_t));+}++void ra_init(roaring_array_t *new_ra) {+    if (!new_ra) {+        return;+    }+    new_ra->keys = NULL;+    new_ra->containers = NULL;+    new_ra->typecodes = NULL;++    new_ra->allocation_size = 0;+    new_ra->size = 0;+    new_ra->flags = 0;+}++bool ra_overwrite(const roaring_array_t *source, roaring_array_t *dest,+                  bool copy_on_write) {+    ra_clear_containers(dest);  // we are going to overwrite them+    if (source->size == 0) {    // Note: can't call memcpy(NULL), even w/size+        dest->size = 0;         // <--- This is important.+        return true;            // output was just cleared, so they match+    }+    if (dest->allocation_size < source->size) {+        if (!realloc_array(dest, source->size)) {+            return false;+        }+    }+    dest->size = source->size;+    memcpy(dest->keys, source->keys, dest->size * sizeof(uint16_t));+    // we go through the containers, turning them into shared containers...+    if (copy_on_write) {+        for (int32_t i = 0; i < dest->size; ++i) {+            source->containers[i] = get_copy_of_container(+                source->containers[i], &source->typecodes[i], copy_on_write);+        }+        // we do a shallow copy to the other bitmap+        memcpy(dest->containers, source->containers,+               dest->size * sizeof(container_t *));+        memcpy(dest->typecodes, source->typecodes,+               dest->size * sizeof(uint8_t));+    } else {+        memcpy(dest->typecodes, source->typecodes,+               dest->size * sizeof(uint8_t));+        for (int32_t i = 0; i < dest->size; i++) {+            dest->containers[i] =+                container_clone(source->containers[i], source->typecodes[i]);+            if (dest->containers[i] == NULL) {+                for (int32_t j = 0; j < i; j++) {+                    container_free(dest->containers[j], dest->typecodes[j]);+                }+                ra_clear_without_containers(dest);+                return false;+            }+        }+    }+    return true;+}++void ra_clear_containers(roaring_array_t *ra) {+    for (int32_t i = 0; i < ra->size; ++i) {+        container_free(ra->containers[i], ra->typecodes[i]);+    }+}++void ra_reset(roaring_array_t *ra) {+    ra_clear_containers(ra);+    ra->size = 0;+    ra_shrink_to_fit(ra);+}++void ra_clear_without_containers(roaring_array_t *ra) {+    roaring_free(+        ra->containers);  // keys and typecodes are allocated with containers+    ra->size = 0;+    ra->allocation_size = 0;+    ra->containers = NULL;+    ra->keys = NULL;+    ra->typecodes = NULL;+}++void ra_clear(roaring_array_t *ra) {+    ra_clear_containers(ra);+    ra_clear_without_containers(ra);+}++bool extend_array(roaring_array_t *ra, int32_t k) {+    int32_t desired_size = ra->size + k;+    const int32_t max_containers = 65536;+    assert(desired_size <= max_containers);+    if (desired_size > ra->allocation_size) {+        int32_t new_capacity =+            (ra->size < 1024) ? 2 * desired_size : 5 * desired_size / 4;+        if (new_capacity > max_containers) {+            new_capacity = max_containers;+        }++        return realloc_array(ra, new_capacity);+    }+    return true;+}++void ra_append(roaring_array_t *ra, uint16_t key, container_t *c,+               uint8_t typecode) {+    extend_array(ra, 1);+    const int32_t pos = ra->size;++    ra->keys[pos] = key;+    ra->containers[pos] = c;+    ra->typecodes[pos] = typecode;+    ra->size++;+}++void ra_append_copy(roaring_array_t *ra, const roaring_array_t *sa,+                    uint16_t index, bool copy_on_write) {+    extend_array(ra, 1);+    const int32_t pos = ra->size;++    // old contents is junk that does not need freeing+    ra->keys[pos] = sa->keys[index];+    // the shared container will be in two bitmaps+    if (copy_on_write) {+        sa->containers[index] = get_copy_of_container(+            sa->containers[index], &sa->typecodes[index], copy_on_write);+        ra->containers[pos] = sa->containers[index];+        ra->typecodes[pos] = sa->typecodes[index];+    } else {+        ra->containers[pos] =+            container_clone(sa->containers[index], sa->typecodes[index]);+        ra->typecodes[pos] = sa->typecodes[index];+    }+    ra->size++;+}++void ra_append_copies_until(roaring_array_t *ra, const roaring_array_t *sa,+                            uint16_t stopping_key, bool copy_on_write) {+    for (int32_t i = 0; i < sa->size; ++i) {+        if (sa->keys[i] >= stopping_key) break;+        ra_append_copy(ra, sa, (uint16_t)i, copy_on_write);+    }+}++void ra_append_copy_range(roaring_array_t *ra, const roaring_array_t *sa,+                          int32_t start_index, int32_t end_index,+                          bool copy_on_write) {+    extend_array(ra, end_index - start_index);+    for (int32_t i = start_index; i < end_index; ++i) {+        const int32_t pos = ra->size;+        ra->keys[pos] = sa->keys[i];+        if (copy_on_write) {+            sa->containers[i] = get_copy_of_container(+                sa->containers[i], &sa->typecodes[i], copy_on_write);+            ra->containers[pos] = sa->containers[i];+            ra->typecodes[pos] = sa->typecodes[i];+        } else {+            ra->containers[pos] =+                container_clone(sa->containers[i], sa->typecodes[i]);+            ra->typecodes[pos] = sa->typecodes[i];+        }+        ra->size++;+    }+}++void ra_append_copies_after(roaring_array_t *ra, const roaring_array_t *sa,+                            uint16_t before_start, bool copy_on_write) {+    int start_location = ra_get_index(sa, before_start);+    if (start_location >= 0)+        ++start_location;+    else+        start_location = -start_location - 1;+    ra_append_copy_range(ra, sa, start_location, sa->size, copy_on_write);+}++void ra_append_move_range(roaring_array_t *ra, roaring_array_t *sa,+                          int32_t start_index, int32_t end_index) {+    extend_array(ra, end_index - start_index);++    for (int32_t i = start_index; i < end_index; ++i) {+        const int32_t pos = ra->size;++        ra->keys[pos] = sa->keys[i];+        ra->containers[pos] = sa->containers[i];+        ra->typecodes[pos] = sa->typecodes[i];+        ra->size++;+    }+}++void ra_append_range(roaring_array_t *ra, roaring_array_t *sa,+                     int32_t start_index, int32_t end_index,+                     bool copy_on_write) {+    extend_array(ra, end_index - start_index);++    for (int32_t i = start_index; i < end_index; ++i) {+        const int32_t pos = ra->size;+        ra->keys[pos] = sa->keys[i];+        if (copy_on_write) {+            sa->containers[i] = get_copy_of_container(+                sa->containers[i], &sa->typecodes[i], copy_on_write);+            ra->containers[pos] = sa->containers[i];+            ra->typecodes[pos] = sa->typecodes[i];+        } else {+            ra->containers[pos] =+                container_clone(sa->containers[i], sa->typecodes[i]);+            ra->typecodes[pos] = sa->typecodes[i];+        }+        ra->size++;+    }+}++container_t *ra_get_container(roaring_array_t *ra, uint16_t x,+                              uint8_t *typecode) {+    int i = binarySearch(ra->keys, (int32_t)ra->size, x);+    if (i < 0) return NULL;+    *typecode = ra->typecodes[i];+    return ra->containers[i];+}++extern inline container_t *ra_get_container_at_index(const roaring_array_t *ra,+                                                     uint16_t i,+                                                     uint8_t *typecode);++extern inline uint16_t ra_get_key_at_index(const roaring_array_t *ra,+                                           uint16_t i);++extern inline int32_t ra_get_index(const roaring_array_t *ra, uint16_t x);++extern inline int32_t ra_advance_until(const roaring_array_t *ra, uint16_t x,+                                       int32_t pos);++// everything skipped over is freed+int32_t ra_advance_until_freeing(roaring_array_t *ra, uint16_t x, int32_t pos) {+    while (pos < ra->size && ra->keys[pos] < x) {+        container_free(ra->containers[pos], ra->typecodes[pos]);+        ++pos;+    }+    return pos;+}++void ra_insert_new_key_value_at(roaring_array_t *ra, int32_t i, uint16_t key,+                                container_t *c, uint8_t typecode) {+    extend_array(ra, 1);+    // May be an optimization opportunity with DIY memmove+    memmove(&(ra->keys[i + 1]), &(ra->keys[i]),+            sizeof(uint16_t) * (ra->size - i));+    memmove(&(ra->containers[i + 1]), &(ra->containers[i]),+            sizeof(container_t *) * (ra->size - i));+    memmove(&(ra->typecodes[i + 1]), &(ra->typecodes[i]),+            sizeof(uint8_t) * (ra->size - i));+    ra->keys[i] = key;+    ra->containers[i] = c;+    ra->typecodes[i] = typecode;+    ra->size++;+}++// note: Java routine set things to 0, enabling GC.+// Java called it "resize" but it was always used to downsize.+// Allowing upsize would break the conventions about+// valid containers below ra->size.++void ra_downsize(roaring_array_t *ra, int32_t new_length) {+    assert(new_length <= ra->size);+    ra->size = new_length;+}++void ra_remove_at_index(roaring_array_t *ra, int32_t i) {+    memmove(&(ra->containers[i]), &(ra->containers[i + 1]),+            sizeof(container_t *) * (ra->size - i - 1));+    memmove(&(ra->keys[i]), &(ra->keys[i + 1]),+            sizeof(uint16_t) * (ra->size - i - 1));+    memmove(&(ra->typecodes[i]), &(ra->typecodes[i + 1]),+            sizeof(uint8_t) * (ra->size - i - 1));+    ra->size--;+}++void ra_remove_at_index_and_free(roaring_array_t *ra, int32_t i) {+    container_free(ra->containers[i], ra->typecodes[i]);+    ra_remove_at_index(ra, i);+}++// used in inplace andNot only, to slide left the containers from+// the mutated RoaringBitmap that are after the largest container of+// the argument RoaringBitmap.  In use it should be followed by a call to+// downsize.+//+void ra_copy_range(roaring_array_t *ra, uint32_t begin, uint32_t end,+                   uint32_t new_begin) {+    assert(begin <= end);+    assert(new_begin < begin);++    const int range = end - begin;++    // We ensure to previously have freed overwritten containers+    // that are not copied elsewhere++    memmove(&(ra->containers[new_begin]), &(ra->containers[begin]),+            sizeof(container_t *) * range);+    memmove(&(ra->keys[new_begin]), &(ra->keys[begin]),+            sizeof(uint16_t) * range);+    memmove(&(ra->typecodes[new_begin]), &(ra->typecodes[begin]),+            sizeof(uint8_t) * range);+}++void ra_shift_tail(roaring_array_t *ra, int32_t count, int32_t distance) {+    if (distance > 0) {+        extend_array(ra, distance);+    }+    int32_t srcpos = ra->size - count;+    int32_t dstpos = srcpos + distance;+    memmove(&(ra->keys[dstpos]), &(ra->keys[srcpos]), sizeof(uint16_t) * count);+    memmove(&(ra->containers[dstpos]), &(ra->containers[srcpos]),+            sizeof(container_t *) * count);+    memmove(&(ra->typecodes[dstpos]), &(ra->typecodes[srcpos]),+            sizeof(uint8_t) * count);+    ra->size += distance;+}++void ra_to_uint32_array(const roaring_array_t *ra, uint32_t *ans) {+    size_t ctr = 0;+    for (int32_t i = 0; i < ra->size; ++i) {+        int num_added = container_to_uint32_array(+            ans + ctr, ra->containers[i], ra->typecodes[i],+            ((uint32_t)ra->keys[i]) << 16);+        ctr += num_added;+    }+}++bool ra_has_run_container(const roaring_array_t *ra) {+    for (int32_t k = 0; k < ra->size; ++k) {+        if (get_container_type(ra->containers[k], ra->typecodes[k]) ==+            RUN_CONTAINER_TYPE)+            return true;+    }+    return false;+}++uint32_t ra_portable_header_size(const roaring_array_t *ra) {+    return ra_portable_header_size_from_counts(ra->size,+                                               ra_has_run_container(ra));+}++uint32_t ra_portable_header_size_from_counts(int32_t num_containers,+                                             bool hasrun) {+    if (hasrun) {+        // for small bitmaps, we omit the offsets+        if (num_containers < NO_OFFSET_THRESHOLD) {+            return 4 + (num_containers + 7) / 8 + 4 * num_containers;+        }+        // - 4 because we pack the size with the cookie+        return 4 + (num_containers + 7) / 8 + 8 * num_containers;+    } else {+        return 4 + 4 + 8 * num_containers;+    }+}++size_t ra_portable_size_in_bytes(const roaring_array_t *ra) {+    size_t count = ra_portable_header_size(ra);++    for (int32_t k = 0; k < ra->size; ++k) {+        count += container_size_in_bytes(ra->containers[k], ra->typecodes[k]);+    }+    return count;+}++// The portable serialization format is little-endian. On big-endian hosts we+// byte-swap multi-byte fields before writing them to the buffer.+size_t ra_portable_serialize(const roaring_array_t *ra, char *buf) {+    char *initbuf = buf;+    uint32_t startOffset = 0;+    bool hasrun = ra_has_run_container(ra);+    if (hasrun) {+        uint32_t cookie = SERIAL_COOKIE | ((uint32_t)(ra->size - 1) << 16);+        uint32_t cookie_le = croaring_htole32(cookie);+        memcpy(buf, &cookie_le, sizeof(cookie_le));+        buf += sizeof(cookie_le);+        uint32_t s = (ra->size + 7) / 8;+        memset(buf, 0, s);+        for (int32_t i = 0; i < ra->size; ++i) {+            if (get_container_type(ra->containers[i], ra->typecodes[i]) ==+                RUN_CONTAINER_TYPE) {+                buf[i / 8] |= 1 << (i % 8);+            }+        }+        buf += s;+        if (ra->size < NO_OFFSET_THRESHOLD) {+            startOffset = 4 + 4 * ra->size + s;+        } else {+            startOffset = 4 + 8 * ra->size + s;+        }+    } else {  // backwards compatibility+        uint32_t cookie = SERIAL_COOKIE_NO_RUNCONTAINER;+        uint32_t cookie_le = croaring_htole32(cookie);+        memcpy(buf, &cookie_le, sizeof(cookie_le));+        buf += sizeof(cookie_le);+        uint32_t size_le = croaring_htole32((uint32_t)ra->size);+        memcpy(buf, &size_le, sizeof(size_le));+        buf += sizeof(size_le);++        startOffset = 4 + 4 + 4 * ra->size + 4 * ra->size;+    }+    for (int32_t k = 0; k < ra->size; ++k) {+        uint16_t key_le = croaring_htole16(ra->keys[k]);+        memcpy(buf, &key_le, sizeof(key_le));+        buf += sizeof(key_le);+        // get_cardinality returns a value in [1,1<<16], subtracting one+        // we get [0,1<<16 - 1] which fits in 16 bits+        uint16_t card = (uint16_t)(container_get_cardinality(ra->containers[k],+                                                             ra->typecodes[k]) -+                                   1);+        uint16_t card_le = croaring_htole16(card);+        memcpy(buf, &card_le, sizeof(card_le));+        buf += sizeof(card_le);+    }+    if ((!hasrun) || (ra->size >= NO_OFFSET_THRESHOLD)) {+        // writing the containers offsets+        for (int32_t k = 0; k < ra->size; k++) {+            uint32_t off_le = croaring_htole32(startOffset);+            memcpy(buf, &off_le, sizeof(off_le));+            buf += sizeof(off_le);+            startOffset =+                startOffset ++                container_size_in_bytes(ra->containers[k], ra->typecodes[k]);+        }+    }+    for (int32_t k = 0; k < ra->size; ++k) {+        buf += container_write(ra->containers[k], ra->typecodes[k], buf);+    }+    return buf - initbuf;+}++// Quickly checks whether there is a serialized bitmap at the pointer,+// not exceeding size "maxbytes" in bytes. This function does not allocate+// memory dynamically.+//+// This function returns 0 if and only if no valid bitmap is found.+// Otherwise, it returns how many bytes are occupied.+//+size_t ra_portable_deserialize_size(const char *buf, const size_t maxbytes) {+    size_t bytestotal = sizeof(int32_t);  // for cookie+    if (bytestotal > maxbytes) return 0;+    uint32_t cookie;+    memcpy(&cookie, buf, sizeof(int32_t));+    cookie = croaring_letoh32(cookie);+    buf += sizeof(uint32_t);+    if ((cookie & 0xFFFF) != SERIAL_COOKIE &&+        cookie != SERIAL_COOKIE_NO_RUNCONTAINER) {+        return 0;+    }+    int32_t size;++    if ((cookie & 0xFFFF) == SERIAL_COOKIE)+        size = (cookie >> 16) + 1;+    else {+        bytestotal += sizeof(int32_t);+        if (bytestotal > maxbytes) return 0;+        uint32_t size_le;+        memcpy(&size_le, buf, sizeof(int32_t));+        size = (int32_t)croaring_letoh32(size_le);+        buf += sizeof(uint32_t);+    }+    if (size > (1 << 16) || size < 0) {+        return 0;+    }+    char *bitmapOfRunContainers = NULL;+    bool hasrun = (cookie & 0xFFFF) == SERIAL_COOKIE;+    if (hasrun) {+        int32_t s = (size + 7) / 8;+        bytestotal += s;+        if (bytestotal > maxbytes) return 0;+        bitmapOfRunContainers = (char *)buf;+        buf += s;+    }+    bytestotal += size * 2 * sizeof(uint16_t);+    if (bytestotal > maxbytes) return 0;+    const char *keyscards = buf;+    buf += size * 2 * sizeof(uint16_t);+    if ((!hasrun) || (size >= NO_OFFSET_THRESHOLD)) {+        // skipping the offsets+        bytestotal += size * 4;+        if (bytestotal > maxbytes) return 0;+        buf += size * 4;+    }+    // Reading the containers+    for (int32_t k = 0; k < size; ++k) {+        uint16_t tmp;+        memcpy(&tmp, keyscards + 4 * k + 2, sizeof(tmp));+        tmp = croaring_letoh16(tmp);+        uint32_t thiscard = tmp + 1;+        bool isbitmap = (thiscard > DEFAULT_MAX_SIZE);+        bool isrun = false;+        if (hasrun) {+            if ((bitmapOfRunContainers[k / 8] & (1 << (k % 8))) != 0) {+                isbitmap = false;+                isrun = true;+            }+        }+        if (isbitmap) {+            size_t containersize =+                BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+            bytestotal += containersize;+            if (bytestotal > maxbytes) return 0;+            buf += containersize;+        } else if (isrun) {+            bytestotal += sizeof(uint16_t);+            if (bytestotal > maxbytes) return 0;+            uint16_t n_runs;+            memcpy(&n_runs, buf, sizeof(uint16_t));+            n_runs = croaring_letoh16(n_runs);+            buf += sizeof(uint16_t);+            size_t containersize = n_runs * sizeof(rle16_t);+            bytestotal += containersize;+            if (bytestotal > maxbytes) return 0;+            buf += containersize;+        } else {+            size_t containersize = thiscard * sizeof(uint16_t);+            bytestotal += containersize;+            if (bytestotal > maxbytes) return 0;+            buf += containersize;+        }+    }+    return bytestotal;+}++// This function populates answer from the content of buf (reading up to+// maxbytes bytes). The function returns false if a properly serialized bitmap+// cannot be found. If it returns true, readbytes is populated by how many bytes+// were read, we have that *readbytes <= maxbytes.+//+// The portable serialization format is little-endian. On big-endian hosts we+// byte-swap multi-byte fields after reading them from the buffer.+bool ra_portable_deserialize(roaring_array_t *answer, const char *buf,+                             const size_t maxbytes, size_t *readbytes) {+    *readbytes = sizeof(int32_t);  // for cookie+    if (*readbytes > maxbytes) {+        // Ran out of bytes while reading first 4 bytes.+        return false;+    }+    uint32_t cookie;+    memcpy(&cookie, buf, sizeof(int32_t));+    cookie = croaring_letoh32(cookie);+    buf += sizeof(uint32_t);+    if ((cookie & 0xFFFF) != SERIAL_COOKIE &&+        cookie != SERIAL_COOKIE_NO_RUNCONTAINER) {+        // "I failed to find one of the right cookies.+        return false;+    }+    int32_t size;++    if ((cookie & 0xFFFF) == SERIAL_COOKIE)+        size = (cookie >> 16) + 1;+    else {+        *readbytes += sizeof(int32_t);+        if (*readbytes > maxbytes) {+            // Ran out of bytes while reading second part of the cookie.+            return false;+        }+        uint32_t size_le;+        memcpy(&size_le, buf, sizeof(int32_t));+        size = (int32_t)croaring_letoh32(size_le);+        buf += sizeof(uint32_t);+    }+    if (size < 0) {+        // You cannot have a negative number of containers, the data must be+        // corrupted.+        return false;+    }+    if (size > (1 << 16)) {+        // You cannot have so many containers, the data must be corrupted.+        return false;+    }+    const char *bitmapOfRunContainers = NULL;+    bool hasrun = (cookie & 0xFFFF) == SERIAL_COOKIE;+    if (hasrun) {+        int32_t s = (size + 7) / 8;+        *readbytes += s;+        if (*readbytes > maxbytes) {  // data is corrupted?+            // Ran out of bytes while reading run bitmap.+            return false;+        }+        bitmapOfRunContainers = buf;+        buf += s;+    }+    const char *keyscards = buf;++    *readbytes += size * 2 * sizeof(uint16_t);+    if (*readbytes > maxbytes) {+        // Ran out of bytes while reading key-cardinality array.+        return false;+    }+    buf += size * 2 * sizeof(uint16_t);++    bool is_ok = ra_init_with_capacity(answer, size);+    if (!is_ok) {+        // Failed to allocate memory for roaring array. Bailing out.+        return false;+    }++    for (int32_t k = 0; k < size; ++k) {+        uint16_t tmp;+        memcpy(&tmp, keyscards + 4 * k, sizeof(tmp));+        answer->keys[k] = croaring_letoh16(tmp);+    }+    if ((!hasrun) || (size >= NO_OFFSET_THRESHOLD)) {+        *readbytes += size * 4;+        if (*readbytes > maxbytes) {  // data is corrupted?+            // Ran out of bytes while reading offsets.+            ra_clear(answer);  // we need to clear the containers already+                               // allocated, and the roaring array+            return false;+        }++        // skipping the offsets+        buf += size * 4;+    }+    // Reading the containers+    for (int32_t k = 0; k < size; ++k) {+        uint16_t tmp;+        memcpy(&tmp, keyscards + 4 * k + 2, sizeof(tmp));+        tmp = croaring_letoh16(tmp);+        uint32_t thiscard = tmp + 1;+        bool isbitmap = (thiscard > DEFAULT_MAX_SIZE);+        bool isrun = false;+        if (hasrun) {+            if ((bitmapOfRunContainers[k / 8] & (1 << (k % 8))) != 0) {+                isbitmap = false;+                isrun = true;+            }+        }+        if (isbitmap) {+            // we check that the read is allowed+            size_t containersize =+                BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t);+            *readbytes += containersize;+            if (*readbytes > maxbytes) {+                // Running out of bytes while reading a bitset container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            // it is now safe to read+            bitset_container_t *c = bitset_container_create_uninitialized();+            if (c == NULL) {  // memory allocation failure+                // Failed to allocate memory for a bitset container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            answer->size++;+            buf += bitset_container_read(thiscard, c, buf);+            answer->containers[k] = c;+            answer->typecodes[k] = BITSET_CONTAINER_TYPE;+        } else if (isrun) {+            // we check that the read is allowed+            *readbytes += sizeof(uint16_t);+            if (*readbytes > maxbytes) {+                // Running out of bytes while reading a run container (header).+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            uint16_t n_runs;+            memcpy(&n_runs, buf, sizeof(uint16_t));+            n_runs = croaring_letoh16(n_runs);+            size_t containersize = n_runs * sizeof(rle16_t);+            *readbytes += containersize;+            if (*readbytes > maxbytes) {  // data is corrupted?+                // Running out of bytes while reading a run container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            // it is now safe to read++            run_container_t *c = run_container_create();+            if (c == NULL) {  // memory allocation failure+                // Failed to allocate memory for a run container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            answer->size++;+            buf += run_container_read(thiscard, c, buf);+            answer->containers[k] = c;+            answer->typecodes[k] = RUN_CONTAINER_TYPE;+        } else {+            // we check that the read is allowed+            size_t containersize = thiscard * sizeof(uint16_t);+            *readbytes += containersize;+            if (*readbytes > maxbytes) {  // data is corrupted?+                // Running out of bytes while reading an array container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            // it is now safe to read+            array_container_t *c =+                array_container_create_given_capacity(thiscard);+            if (c == NULL) {  // memory allocation failure+                // Failed to allocate memory for an array container.+                ra_clear(answer);  // we need to clear the containers already+                                   // allocated, and the roaring array+                return false;+            }+            answer->size++;+            buf += array_container_read(thiscard, c, buf);+            answer->containers[k] = c;+            answer->typecodes[k] = ARRAY_CONTAINER_TYPE;+        }+    }+    return true;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace internal {+#endif
+ cbits/CRoaring/src/roaring_priority_queue.c view
@@ -0,0 +1,253 @@+#include <roaring/roaring.h>+#include <roaring/roaring_array.h>++#ifdef __cplusplus+using namespace ::roaring::internal;++extern "C" {+namespace roaring {+namespace api {+#endif++struct roaring_pq_element_s {+    uint64_t size;+    bool is_temporary;+    roaring_bitmap_t *bitmap;+};++typedef struct roaring_pq_element_s roaring_pq_element_t;++struct roaring_pq_s {+    roaring_pq_element_t *elements;+    uint64_t size;+};++typedef struct roaring_pq_s roaring_pq_t;++static inline bool compare(roaring_pq_element_t *t1, roaring_pq_element_t *t2) {+    return t1->size < t2->size;+}++static void pq_add(roaring_pq_t *pq, roaring_pq_element_t *t) {+    uint64_t i = pq->size;+    pq->elements[pq->size++] = *t;+    while (i > 0) {+        uint64_t p = (i - 1) >> 1;+        roaring_pq_element_t ap = pq->elements[p];+        if (!compare(t, &ap)) break;+        pq->elements[i] = ap;+        i = p;+    }+    pq->elements[i] = *t;+}++static void pq_free(roaring_pq_t *pq) { roaring_free(pq); }++static void percolate_down(roaring_pq_t *pq, uint32_t i) {+    uint32_t size = (uint32_t)pq->size;+    uint32_t hsize = size >> 1;+    roaring_pq_element_t ai = pq->elements[i];+    while (i < hsize) {+        uint32_t l = (i << 1) + 1;+        uint32_t r = l + 1;+        roaring_pq_element_t bestc = pq->elements[l];+        if (r < size) {+            if (compare(pq->elements + r, &bestc)) {+                l = r;+                bestc = pq->elements[r];+            }+        }+        if (!compare(&bestc, &ai)) {+            break;+        }+        pq->elements[i] = bestc;+        i = l;+    }+    pq->elements[i] = ai;+}++static roaring_pq_t *create_pq(const roaring_bitmap_t **arr, uint32_t length) {+    size_t alloc_size =+        sizeof(roaring_pq_t) + sizeof(roaring_pq_element_t) * length;+    roaring_pq_t *answer = (roaring_pq_t *)roaring_malloc(alloc_size);+    answer->elements = (roaring_pq_element_t *)(answer + 1);+    answer->size = length;+    for (uint32_t i = 0; i < length; i++) {+        answer->elements[i].bitmap = (roaring_bitmap_t *)arr[i];+        answer->elements[i].is_temporary = false;+        answer->elements[i].size =+            roaring_bitmap_portable_size_in_bytes(arr[i]);+    }+    for (int32_t i = (length >> 1); i >= 0; i--) {+        percolate_down(answer, i);+    }+    return answer;+}++static roaring_pq_element_t pq_poll(roaring_pq_t *pq) {+    roaring_pq_element_t ans = *pq->elements;+    if (pq->size > 1) {+        pq->elements[0] = pq->elements[--pq->size];+        percolate_down(pq, 0);+    } else+        --pq->size;+    // memmove(pq->elements,pq->elements+1,(pq->size-1)*sizeof(roaring_pq_element_t));--pq->size;+    return ans;+}++// this function consumes and frees the inputs+static roaring_bitmap_t *lazy_or_from_lazy_inputs(roaring_bitmap_t *x1,+                                                  roaring_bitmap_t *x2) {+    uint8_t result_type = 0;+    const int length1 = ra_get_size(&x1->high_low_container),+              length2 = ra_get_size(&x2->high_low_container);+    if (0 == length1) {+        roaring_bitmap_free(x1);+        return x2;+    }+    if (0 == length2) {+        roaring_bitmap_free(x2);+        return x1;+    }+    uint32_t neededcap = length1 > length2 ? length2 : length1;+    roaring_bitmap_t *answer = roaring_bitmap_create_with_capacity(neededcap);+    int pos1 = 0, pos2 = 0;+    uint8_t type1, type2;+    uint16_t s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+    uint16_t s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+    while (true) {+        if (s1 == s2) {+            // todo: unsharing can be inefficient as it may create a clone where+            // none+            // is needed, but it has the benefit of being easy to reason about.++            ra_unshare_container_at_index(&x1->high_low_container,+                                          (uint16_t)pos1);+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            assert(type1 != SHARED_CONTAINER_TYPE);++            ra_unshare_container_at_index(&x2->high_low_container,+                                          (uint16_t)pos2);+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            assert(type2 != SHARED_CONTAINER_TYPE);++            container_t *c;++            if ((type2 == BITSET_CONTAINER_TYPE) &&+                (type1 != BITSET_CONTAINER_TYPE)) {+                c = container_lazy_ior(c2, type2, c1, type1, &result_type);+                container_free(c1, type1);+                if (c != c2) {+                    container_free(c2, type2);+                }+            } else {+                c = container_lazy_ior(c1, type1, c2, type2, &result_type);+                container_free(c2, type2);+                if (c != c1) {+                    container_free(c1, type1);+                }+            }+            // since we assume that the initial containers are non-empty, the+            // result here+            // can only be non-empty+            ra_append(&answer->high_low_container, s1, c, result_type);+            ++pos1;+            ++pos2;+            if (pos1 == length1) break;+            if (pos2 == length2) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);++        } else if (s1 < s2) {  // s1 < s2+            container_t *c1 = ra_get_container_at_index(&x1->high_low_container,+                                                        (uint16_t)pos1, &type1);+            ra_append(&answer->high_low_container, s1, c1, type1);+            pos1++;+            if (pos1 == length1) break;+            s1 = ra_get_key_at_index(&x1->high_low_container, (uint16_t)pos1);++        } else {  // s1 > s2+            container_t *c2 = ra_get_container_at_index(&x2->high_low_container,+                                                        (uint16_t)pos2, &type2);+            ra_append(&answer->high_low_container, s2, c2, type2);+            pos2++;+            if (pos2 == length2) break;+            s2 = ra_get_key_at_index(&x2->high_low_container, (uint16_t)pos2);+        }+    }+    if (pos1 == length1) {+        ra_append_move_range(&answer->high_low_container,+                             &x2->high_low_container, pos2, length2);+    } else if (pos2 == length2) {+        ra_append_move_range(&answer->high_low_container,+                             &x1->high_low_container, pos1, length1);+    }+    ra_clear_without_containers(&x1->high_low_container);+    ra_clear_without_containers(&x2->high_low_container);+    roaring_free(x1);+    roaring_free(x2);+    return answer;+}++/**+ * Compute the union of 'number' bitmaps using a heap. This can+ * sometimes be faster than roaring_bitmap_or_many which uses+ * a naive algorithm. Caller is responsible for freeing the+ * result.+ */+roaring_bitmap_t *roaring_bitmap_or_many_heap(uint32_t number,+                                              const roaring_bitmap_t **x) {+    if (number == 0) {+        return roaring_bitmap_create();+    }+    if (number == 1) {+        return roaring_bitmap_copy(x[0]);+    }+    roaring_pq_t *pq = create_pq(x, number);+    while (pq->size > 1) {+        roaring_pq_element_t x1 = pq_poll(pq);+        roaring_pq_element_t x2 = pq_poll(pq);++        if (x1.is_temporary && x2.is_temporary) {+            roaring_bitmap_t *newb =+                lazy_or_from_lazy_inputs(x1.bitmap, x2.bitmap);+            // should normally return a fresh new bitmap *except* that+            // it can return x1.bitmap or x2.bitmap in degenerate cases+            bool temporary = !((newb == x1.bitmap) && (newb == x2.bitmap));+            uint64_t bsize = roaring_bitmap_portable_size_in_bytes(newb);+            roaring_pq_element_t newelement = {+                .size = bsize, .is_temporary = temporary, .bitmap = newb};+            pq_add(pq, &newelement);+        } else if (x2.is_temporary) {+            roaring_bitmap_lazy_or_inplace(x2.bitmap, x1.bitmap, false);+            x2.size = roaring_bitmap_portable_size_in_bytes(x2.bitmap);+            pq_add(pq, &x2);+        } else if (x1.is_temporary) {+            roaring_bitmap_lazy_or_inplace(x1.bitmap, x2.bitmap, false);+            x1.size = roaring_bitmap_portable_size_in_bytes(x1.bitmap);++            pq_add(pq, &x1);+        } else {+            roaring_bitmap_t *newb =+                roaring_bitmap_lazy_or(x1.bitmap, x2.bitmap, false);+            uint64_t bsize = roaring_bitmap_portable_size_in_bytes(newb);+            roaring_pq_element_t newelement = {+                .size = bsize, .is_temporary = true, .bitmap = newb};++            pq_add(pq, &newelement);+        }+    }+    roaring_pq_element_t X = pq_poll(pq);+    roaring_bitmap_t *answer = X.bitmap;+    roaring_bitmap_repair_after_lazy(answer);+    pq_free(pq);+    return answer;+}++#ifdef __cplusplus+}+}+}  // extern "C" { namespace roaring { namespace api {+#endif
+ cbits/croaring-ffi.c view
@@ -0,0 +1,5 @@+#include "croaring-ffi.h"++void croaring_ffi_init_memory_hook(const roaring_memory_t *hook) {+    roaring_init_memory_hook(*hook);+}
+ cbits/croaring-ffi.h view
@@ -0,0 +1,8 @@+#ifndef CROARING_FFI_H+#define CROARING_FFI_H++#include <roaring/memory.h>++void croaring_ffi_init_memory_hook(const roaring_memory_t *hook);++#endif
+ croaring-ffi.cabal view
@@ -0,0 +1,201 @@+cabal-version: 2.2++-- This file has been generated from package.yaml by hpack version 0.39.6.+--+-- see: https://github.com/sol/hpack++name:           croaring-ffi+version:        1.0+synopsis:       Raw FFI bindings to CRoaring compressed bitmaps+description:    Please see the README on GitLab at <https://gitlab.com/dpwiz/croaring-ffi#croaring-ffi>+category:       Data, FFI+homepage:       https://gitlab.com/dpwiz/croaring-ffi+bug-reports:    https://gitlab.com/dpwiz/croaring-ffi/-/issues+author:         IC Rainbow+maintainer:     aenor.realm@gmail.com+copyright:      2026 IC Rainbow+license:        BSD-3-Clause AND (Apache-2.0 OR MIT)+license-files:  LICENSE,+                cbits/CRoaring/LICENSE+build-type:     Simple+extra-source-files:+    cbits/croaring-ffi.h+    cbits/CRoaring/include/roaring/array_util.h+    cbits/CRoaring/include/roaring/bitset_util.h+    cbits/CRoaring/include/roaring/isadetection.h+    cbits/CRoaring/include/roaring/memory.h+    cbits/CRoaring/include/roaring/portability.h+    cbits/CRoaring/include/roaring/roaring.h+    cbits/CRoaring/include/roaring/roaring64.h+    cbits/CRoaring/include/roaring/roaring_array.h+    cbits/CRoaring/include/roaring/roaring_types.h+    cbits/CRoaring/include/roaring/roaring_version.h+    cbits/CRoaring/include/roaring/utilasm.h+    cbits/CRoaring/include/roaring/art/art.h+    cbits/CRoaring/include/roaring/bitset/bitset.h+    cbits/CRoaring/include/roaring/containers/array.h+    cbits/CRoaring/include/roaring/containers/bitset.h+    cbits/CRoaring/include/roaring/containers/container_defs.h+    cbits/CRoaring/include/roaring/containers/containers.h+    cbits/CRoaring/include/roaring/containers/convert.h+    cbits/CRoaring/include/roaring/containers/mixed_andnot.h+    cbits/CRoaring/include/roaring/containers/mixed_equal.h+    cbits/CRoaring/include/roaring/containers/mixed_intersection.h+    cbits/CRoaring/include/roaring/containers/mixed_negation.h+    cbits/CRoaring/include/roaring/containers/mixed_subset.h+    cbits/CRoaring/include/roaring/containers/mixed_union.h+    cbits/CRoaring/include/roaring/containers/mixed_xor.h+    cbits/CRoaring/include/roaring/containers/perfparameters.h+    cbits/CRoaring/include/roaring/containers/run.h+    cbits/CRoaring/include/roaring/misc/configreport.h+    cbits/CRoaring/src/license-comment.h+extra-doc-files:+    README.md+    CHANGELOG.md++source-repository head+  type: git+  location: https://gitlab.com/dpwiz/croaring-ffi++library+  exposed-modules:+      Data.Bitmap.Roaring.FFI.Bitmap+      Data.Bitmap.Roaring.FFI.Bitmap.BulkContext+      Data.Bitmap.Roaring.FFI.Bitmap.Iterator+      Data.Bitmap.Roaring.FFI.Bitmap.Statistics+      Data.Bitmap.Roaring.FFI.Bitmap64+      Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext+      Data.Bitmap.Roaring.FFI.Bitmap64.Iterator+      Data.Bitmap.Roaring.FFI.Bitmap64.Statistics+      Data.Bitmap.Roaring.FFI.Bitset+      Data.Bitmap.Roaring.FFI.Callback+      Data.Bitmap.Roaring.FFI.Memory+      Data.Bitmap.Roaring.FFI.NullHandle+      Data.Bitmap.Roaring.FFI.Version+  other-modules:+      Paths_croaring_ffi+  autogen-modules:+      Paths_croaring_ffi+  hs-source-dirs:+      src+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints+  cc-options: -O3 -DNDEBUG+  include-dirs:+      cbits+      cbits/CRoaring/include+  c-sources:+      cbits/croaring-ffi.c+      cbits/CRoaring/src/array_util.c+      cbits/CRoaring/src/bitset.c+      cbits/CRoaring/src/bitset_util.c+      cbits/CRoaring/src/isadetection.c+      cbits/CRoaring/src/memory.c+      cbits/CRoaring/src/roaring.c+      cbits/CRoaring/src/roaring64.c+      cbits/CRoaring/src/roaring_array.c+      cbits/CRoaring/src/roaring_priority_queue.c+      cbits/CRoaring/src/art/art.c+      cbits/CRoaring/src/containers/array.c+      cbits/CRoaring/src/containers/bitset.c+      cbits/CRoaring/src/containers/containers.c+      cbits/CRoaring/src/containers/convert.c+      cbits/CRoaring/src/containers/mixed_andnot.c+      cbits/CRoaring/src/containers/mixed_equal.c+      cbits/CRoaring/src/containers/mixed_intersection.c+      cbits/CRoaring/src/containers/mixed_negation.c+      cbits/CRoaring/src/containers/mixed_subset.c+      cbits/CRoaring/src/containers/mixed_union.c+      cbits/CRoaring/src/containers/mixed_xor.c+      cbits/CRoaring/src/containers/run.c+  build-depends:+      base >=4.7 && <5+  default-language: GHC2021++test-suite croaring-ffi-test+  type: exitcode-stdio-1.0+  main-is: Spec.hs+  other-modules:+      Test.Bitmap+      Test.Bitmap.Iterator+      Test.Bitmap64+      Test.Bitmap64.Iterator+      Test.Bitset+      Test.Memory+      Test.Property+      Test.Version+      Paths_croaring_ffi+  autogen-modules:+      Paths_croaring_ffi+  hs-source-dirs:+      test+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts -with-rtsopts=-N+  c-sources:+      test/cbits/counting-allocator.c+  build-depends:+      base >=4.7 && <5+    , containers+    , croaring-ffi+    , hedgehog+    , mmorph+    , resourcet+    , tasty+    , tasty-hedgehog+  default-language: GHC2021++benchmark croaring-ffi-bench+  type: exitcode-stdio-1.0+  main-is: Main.hs+  other-modules:+      Bench.Baseline+      Bench.Bitmap+      Bench.Bitmap64+      Bench.Bitset+      Bench.Comparison+      Bench.Unfair+      Bench.Values+      Paths_croaring_ffi+  autogen-modules:+      Paths_croaring_ffi+  hs-source-dirs:+      bench+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -O2 -rtsopts -with-rtsopts=-A32m -fproc-alignment=64+  build-depends:+      base >=4.7 && <5+    , binary+    , bitvec+    , bytestring+    , containers+    , croaring-ffi+    , deepseq+    , tasty+    , tasty-bench+    , vector+  default-language: GHC2021++benchmark croaring-ffi-sizes+  type: exitcode-stdio-1.0+  main-is: Sizes.hs+  other-modules:+      Bench.Values+  hs-source-dirs:+      bench/sizes+      bench+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -O2+  include-dirs:+      cbits/CRoaring/include+  c-sources:+      bench/sizes/allocation-counter.c+  build-depends:+      base >=4.7 && <5+    , binary+    , bitvec+    , bytestring+    , containers+    , croaring-ffi+    , ghc-compact+    , tasty+    , tasty-bench+    , vector+  default-language: GHC2021+  if os(windows)+    buildable: False
+ src/Data/Bitmap/Roaring/FFI/Bitmap.hsc view
@@ -0,0 +1,791 @@+{-# LANGUAGE CApiFFI #-}++{-| Raw bindings to @roaring/roaring.h@ for 32-bit Roaring bitmaps, @roaring_bitmap_t@.++Each binding is the C function without its @roaring_bitmap_@ (or @roaring_@) prefix, camelCased.+The @_safe@ loaders only bound how far they read, so they are named @*Bounded@ here.+The @*Validated@ loaders also run 'internalValidate'.+Use them for bytes from outside the program.+Handles carry no ownership.+Release the bitmaps that C returns with 'free' or 'finalizerFree'.+A bitmap is not synchronised.+Concurrent use needs external locking.+-}+module Data.Bitmap.Roaring.FFI.Bitmap+  ( Bitmap (..)+  , size+  , alignment+  , create+  , createWithCapacity+  , initWithCapacity+  , initCleared+  , free+  , finalizerFree+  , with+  , fromRange+  , ofPtr+  , copy+  , overwrite+  , clear+  , getCopyOnWrite+  , setCopyOnWrite+  , containsShared+  , unshareAll+  , add+  , addChecked+  , addMany+  , addBulk+  , addRange+  , addRangeClosed+  , remove+  , removeChecked+  , removeMany+  , removeRange+  , removeRangeClosed+  , contains+  , containsBulk+  , containsRange+  , containsRangeClosed+  , getCardinality+  , rangeCardinality+  , rangeCardinalityClosed+  , isEmpty+  , minimum+  , maximum+  , select+  , rank+  , rankMany+  , getIndex+  , equals+  , isSubset+  , isStrictSubset+  , intersect+  , intersectWithRange+  , jaccardIndex+  , and+  , andInplace+  , andCardinality+  , or+  , orInplace+  , orCardinality+  , orMany+  , orManyHeap+  , xor+  , xorInplace+  , xorCardinality+  , xorMany+  , andnot+  , andnotInplace+  , andnotCardinality+  , lazyOr+  , lazyOrInplace+  , lazyXor+  , lazyXorInplace+  , repairAfterLazy+  , flip+  , flipClosed+  , flipInplace+  , flipInplaceClosed+  , addOffset+  , toUint32Array+  , rangeUint32Array+  , toBitset+  , wordRangeToBitset+  , runOptimize+  , removeRunCompression+  , shrinkToFit+  , statistics+  , getStatistics+  , internalValidate+  , sizeInBytes+  , serialize+  , deserialize+  , deserializeBounded+  , deserializeValidated+  , portableSizeInBytes+  , portableSerialize+  , portableDeserialize+  , portableDeserializeBounded+  , portableDeserializeValidated+  , portableDeserializeSize+  , portableDeserializeFrozen+  , frozenSizeInBytes+  , frozenSerialize+  , frozenView+  , withPortableFrozen+  , withFrozenView+  , Callback+  , wrapCallback+  , iterate+  , iterateWith+  , Callback64+  , wrapCallback64+  , iterate64+  , iterate64With+  , printf+  , printfDescribe+  ) where++#include <roaring/roaring.h>+#include <roaring/roaring_array.h>++import Prelude hiding (and, flip, iterate, maximum, minimum, or)++import Control.Exception (bracket)+import Data.Bitmap.Roaring.FFI.Bitmap.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap.Statistics (Statistics)+import Data.Bitmap.Roaring.FFI.Bitset (Bitset (..))+import Data.Bitmap.Roaring.FFI.Callback qualified as Callback+import Data.Bitmap.Roaring.FFI.NullHandle qualified as NullHandle+import Data.Bits (shiftL, shiftR, testBit, (.&.), (.|.))+import Data.Int (Int64)+import Data.Word (Word32, Word64, Word8)+import Foreign.C.ConstPtr (ConstPtr (..))+import Foreign.C.Types (CChar, CSize (..))+import Foreign.ForeignPtr (FinalizerPtr)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Marshal.Array (peekArray)+import Foreign.Ptr (FunPtr, Ptr, nullPtr, plusPtr)+import Foreign.Storable (Storable, peek)++{- | @roaring_bitmap_t@.++Its 'Storable' instance stores the pointer itself, so a @'Ptr' t'Bitmap'@ is a C array of bitmaps, as taken by 'orMany'.+-}+newtype Bitmap = Bitmap (Ptr Bitmap)+  deriving (Eq, Ord, Show, Storable)++-- | @sizeof(roaring_bitmap_t)@+size :: Int+size = #{size roaring_bitmap_t}+{-# INLINE size #-}++-- | @alignof(roaring_bitmap_t)@+alignment :: Int+alignment = #{alignment roaring_bitmap_t}+{-# INLINE alignment #-}++-- | @roaring_bitmap_create@: NULL on allocation failure.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_create"+  create :: IO Bitmap++-- | @roaring_bitmap_create_with_capacity@: the capacity is a number of containers. NULL on allocation failure.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_create_with_capacity"+  createWithCapacity :: Word32 -> IO Bitmap++{- | @roaring_bitmap_init_with_capacity@: initialises a bitmap in caller memory of 'size' bytes.+'False' if allocating the container array failed.++Such a bitmap must be released with 'clear', never with 'free'.+-}+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_init_with_capacity"+  initWithCapacity :: Bitmap -> Word32 -> IO Bool++-- | @roaring_bitmap_init_cleared@: initialises a bitmap in caller memory without allocating. Release it with 'clear', never with 'free'.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_init_cleared"+  initCleared :: Bitmap -> IO ()++-- | @roaring_bitmap_free@: also releases read-only views.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_free"+  free :: Bitmap -> IO ()++-- | @&roaring_bitmap_free@+foreign import capi unsafe "roaring/roaring.h &roaring_bitmap_free"+  finalizerFree :: FinalizerPtr Bitmap++-- | Frees the bitmap when the action ends. Throws t'NullHandle.NullHandle' if the constructor returns NULL.+with :: IO Bitmap -> (Bitmap -> IO a) -> IO a+with = NullHandle.bracket "Data.Bitmap.Roaring.FFI.Bitmap.with" free+{-# INLINE with #-}++{- | @roaring_bitmap_from_range@: @[min, max)@ every @step@ from @min@, @max@ capped at @2^32@.+NULL on failure, when @step@ is 0, and when the range is empty.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_from_range"+  fromRange :: Word64 -> Word64 -> Word32 -> IO Bitmap++-- | @roaring_bitmap_of_ptr@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_of_ptr"+  ofPtr :: CSize -> Ptr Word32 -> IO Bitmap++-- | @roaring_bitmap_copy@: NULL on failure. With copy-on-write enabled, containers are shared rather than copied.+foreign import capi safe "roaring/roaring.h roaring_bitmap_copy"+  copy :: Bitmap -> IO Bitmap++{- | @roaring_bitmap_overwrite@: makes the first bitmap a copy of the second, copy-on-write flag included.+'False' on allocation failure.++After 'False', the first bitmap must not be used, cleared or freed.+C releases the first bitmap's containers before it allocates anything.+If growing its container array then fails, the bitmap keeps its old size over the released containers.++Passing the same bitmap twice leaves it unchanged and returns 'True'.+The C function would read the containers it has just released, so this binding does not call it.+-}+overwrite :: Bitmap -> Bitmap -> IO Bool+overwrite target source+  | target == source = pure True+  | otherwise = overwriteDistinct target source+{-# INLINE overwrite #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_overwrite"+  overwriteDistinct :: Bitmap -> Bitmap -> IO Bool++-- | @roaring_bitmap_clear@: also releases every container.+foreign import capi safe "roaring/roaring.h roaring_bitmap_clear"+  clear :: Bitmap -> IO ()++-- | @roaring_bitmap_get_copy_on_write@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_get_copy_on_write"+  getCopyOnWrite :: Bitmap -> IO Bool++{- | @roaring_bitmap_set_copy_on_write@: enables or disables copy-on-write.+Disabling it unshares (copies) any shared containers.++Bitmaps that share containers must all have it enabled.+Such bitmaps look independent but touch the same memory, which matters across threads.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_set_copy_on_write"+  setCopyOnWrite :: Bitmap -> Bool -> IO ()++-- | @roaring_contains_shared@+foreign import capi unsafe "roaring/roaring.h roaring_contains_shared"+  containsShared :: Bitmap -> IO Bool++-- | @roaring_unshare_all@: 'True' if anything was unshared.+foreign import capi safe "roaring/roaring.h roaring_unshare_all"+  unshareAll :: Bitmap -> IO Bool++-- | @roaring_bitmap_add@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_add"+  add :: Bitmap -> Word32 -> IO ()++-- | @roaring_bitmap_add_checked@: 'True' if the value was absent.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_add_checked"+  addChecked :: Bitmap -> Word32 -> IO Bool++-- | @roaring_bitmap_add_many@+foreign import capi safe "roaring/roaring.h roaring_bitmap_add_many"+  addMany :: Bitmap -> CSize -> Ptr Word32 -> IO ()++-- | @roaring_bitmap_add_bulk@: see "Data.Bitmap.Roaring.FFI.Bitmap.BulkContext" for the context's rules.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_add_bulk"+  addBulk :: Bitmap -> BulkContext -> Word32 -> IO ()++-- | @roaring_bitmap_add_range@: @[min, max)@, @max@ capped at @2^32@.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_add_range"+  addRange :: Bitmap -> Word64 -> Word64 -> IO ()++-- | @roaring_bitmap_add_range_closed@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_add_range_closed"+  addRangeClosed :: Bitmap -> Word32 -> Word32 -> IO ()++-- | @roaring_bitmap_remove@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_remove"+  remove :: Bitmap -> Word32 -> IO ()++-- | @roaring_bitmap_remove_checked@: 'True' if the value was present.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_remove_checked"+  removeChecked :: Bitmap -> Word32 -> IO Bool++-- | @roaring_bitmap_remove_many@+foreign import capi safe "roaring/roaring.h roaring_bitmap_remove_many"+  removeMany :: Bitmap -> CSize -> Ptr Word32 -> IO ()++-- | @roaring_bitmap_remove_range@: @[min, max)@, @max@ capped at @2^32@.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_remove_range"+  removeRange :: Bitmap -> Word64 -> Word64 -> IO ()++-- | @roaring_bitmap_remove_range_closed@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_remove_range_closed"+  removeRangeClosed :: Bitmap -> Word32 -> Word32 -> IO ()++-- | @roaring_bitmap_contains@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_contains"+  contains :: Bitmap -> Word32 -> IO Bool++-- | @roaring_bitmap_contains_bulk@: see "Data.Bitmap.Roaring.FFI.Bitmap.BulkContext" for the context's rules.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_contains_bulk"+  containsBulk :: Bitmap -> BulkContext -> Word32 -> IO Bool++-- | @roaring_bitmap_contains_range@: @[start, end)@, @end@ capped at @2^32@.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_contains_range"+  containsRange :: Bitmap -> Word64 -> Word64 -> IO Bool++-- | @roaring_bitmap_contains_range_closed@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_contains_range_closed"+  containsRangeClosed :: Bitmap -> Word32 -> Word32 -> IO Bool++-- | @roaring_bitmap_get_cardinality@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_get_cardinality"+  getCardinality :: Bitmap -> IO Word64++-- | @roaring_bitmap_range_cardinality@: @[start, end)@, @end@ capped at @2^32@.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_range_cardinality"+  rangeCardinality :: Bitmap -> Word64 -> Word64 -> IO Word64++-- | @roaring_bitmap_range_cardinality_closed@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_range_cardinality_closed"+  rangeCardinalityClosed :: Bitmap -> Word32 -> Word32 -> IO Word64++-- | @roaring_bitmap_is_empty@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_is_empty"+  isEmpty :: Bitmap -> IO Bool++-- | @roaring_bitmap_minimum@: @UINT32_MAX@ when empty.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_minimum"+  minimum :: Bitmap -> IO Word32++-- | @roaring_bitmap_maximum@: 0 when empty.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_maximum"+  maximum :: Bitmap -> IO Word32++-- | @roaring_bitmap_select@: the rank is 0-based. 'False' when it is not below the cardinality.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_select"+  select :: Bitmap -> Word32 -> Ptr Word32 -> IO Bool++-- | @roaring_bitmap_rank@: the number of values less than or equal to the given one.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_rank"+  rank :: Bitmap -> Word32 -> IO Word64++-- | @roaring_bitmap_rank_many@: 'rank' of each value in the ascending array @[begin, end)@, written to the output array.+foreign import capi safe "roaring/roaring.h roaring_bitmap_rank_many"+  rankMany :: Bitmap -> Ptr Word32 -> Ptr Word32 -> Ptr Word64 -> IO ()++-- | @roaring_bitmap_get_index@: 0-based, or -1 when the value is absent.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_get_index"+  getIndex :: Bitmap -> Word32 -> IO Int64++-- | @roaring_bitmap_equals@+foreign import capi safe "roaring/roaring.h roaring_bitmap_equals"+  equals :: Bitmap -> Bitmap -> IO Bool++-- | @roaring_bitmap_is_subset@: whether the first is a subset of the second.+foreign import capi safe "roaring/roaring.h roaring_bitmap_is_subset"+  isSubset :: Bitmap -> Bitmap -> IO Bool++-- | @roaring_bitmap_is_strict_subset@: whether the first is a strict subset of the second.+foreign import capi safe "roaring/roaring.h roaring_bitmap_is_strict_subset"+  isStrictSubset :: Bitmap -> Bitmap -> IO Bool++-- | @roaring_bitmap_intersect@+foreign import capi safe "roaring/roaring.h roaring_bitmap_intersect"+  intersect :: Bitmap -> Bitmap -> IO Bool++-- | @roaring_bitmap_intersect_with_range@: @[x, y)@, @y@ capped at @2^32@.+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_intersect_with_range"+  intersectWithRange :: Bitmap -> Word64 -> Word64 -> IO Bool++-- | @roaring_bitmap_jaccard_index@: NaN when both are empty.+foreign import capi safe "roaring/roaring.h roaring_bitmap_jaccard_index"+  jaccardIndex :: Bitmap -> Bitmap -> IO Double++-- | @roaring_bitmap_and@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_and"+  and :: Bitmap -> Bitmap -> IO Bitmap++-- | @roaring_bitmap_and_inplace@: the two may be the same bitmap.+foreign import capi safe "roaring/roaring.h roaring_bitmap_and_inplace"+  andInplace :: Bitmap -> Bitmap -> IO ()++-- | @roaring_bitmap_and_cardinality@+foreign import capi safe "roaring/roaring.h roaring_bitmap_and_cardinality"+  andCardinality :: Bitmap -> Bitmap -> IO Word64++-- | @roaring_bitmap_or@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_or"+  or :: Bitmap -> Bitmap -> IO Bitmap++-- | @roaring_bitmap_or_inplace@: passing the same bitmap twice leaves it unchanged.+orInplace :: Bitmap -> Bitmap -> IO ()+orInplace target other+  | target == other = pure ()+  | otherwise = orInplaceDistinct target other+{-# INLINE orInplace #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_or_inplace"+  orInplaceDistinct :: Bitmap -> Bitmap -> IO ()++-- | @roaring_bitmap_or_cardinality@+foreign import capi safe "roaring/roaring.h roaring_bitmap_or_cardinality"+  orCardinality :: Bitmap -> Bitmap -> IO Word64++-- | @roaring_bitmap_or_many@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_or_many"+  orMany :: CSize -> Ptr Bitmap -> IO Bitmap++-- | @roaring_bitmap_or_many_heap@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_or_many_heap"+  orManyHeap :: Word32 -> Ptr Bitmap -> IO Bitmap++-- | @roaring_bitmap_xor@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_xor"+  xor :: Bitmap -> Bitmap -> IO Bitmap++-- | @roaring_bitmap_xor_inplace@: passing the same bitmap twice clears it.+xorInplace :: Bitmap -> Bitmap -> IO ()+xorInplace target other+  | target == other = clear target+  | otherwise = xorInplaceDistinct target other+{-# INLINE xorInplace #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_xor_inplace"+  xorInplaceDistinct :: Bitmap -> Bitmap -> IO ()++-- | @roaring_bitmap_xor_cardinality@+foreign import capi safe "roaring/roaring.h roaring_bitmap_xor_cardinality"+  xorCardinality :: Bitmap -> Bitmap -> IO Word64++-- | @roaring_bitmap_xor_many@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_xor_many"+  xorMany :: CSize -> Ptr Bitmap -> IO Bitmap++-- | @roaring_bitmap_andnot@: first minus second. NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_andnot"+  andnot :: Bitmap -> Bitmap -> IO Bitmap++-- | @roaring_bitmap_andnot_inplace@: first minus second. Passing the same bitmap twice clears it.+andnotInplace :: Bitmap -> Bitmap -> IO ()+andnotInplace target other+  | target == other = clear target+  | otherwise = andnotInplaceDistinct target other+{-# INLINE andnotInplace #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_andnot_inplace"+  andnotInplaceDistinct :: Bitmap -> Bitmap -> IO ()++-- | @roaring_bitmap_andnot_cardinality@: the size of first minus second.+foreign import capi safe "roaring/roaring.h roaring_bitmap_andnot_cardinality"+  andnotCardinality :: Bitmap -> Bitmap -> IO Word64++{- | @roaring_bitmap_lazy_or@: NULL on failure.+The flag forces container results into bitsets.++The result is inconsistent until 'repairAfterLazy'.+Until then, pass it only to the lazy functions.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_lazy_or"+  lazyOr :: Bitmap -> Bitmap -> Bool -> IO Bitmap++-- | @roaring_bitmap_lazy_or_inplace@: the first bitmap is inconsistent until 'repairAfterLazy'. Passing the same bitmap twice leaves it unchanged.+lazyOrInplace :: Bitmap -> Bitmap -> Bool -> IO ()+lazyOrInplace target other bitsetConversion+  | target == other = pure ()+  | otherwise = lazyOrInplaceDistinct target other bitsetConversion+{-# INLINE lazyOrInplace #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_lazy_or_inplace"+  lazyOrInplaceDistinct :: Bitmap -> Bitmap -> Bool -> IO ()++-- | @roaring_bitmap_lazy_xor@: NULL on failure. The result is inconsistent until 'repairAfterLazy'.+foreign import capi safe "roaring/roaring.h roaring_bitmap_lazy_xor"+  lazyXor :: Bitmap -> Bitmap -> IO Bitmap++-- | @roaring_bitmap_lazy_xor_inplace@: the first bitmap is inconsistent until 'repairAfterLazy'. Passing the same bitmap twice clears it.+lazyXorInplace :: Bitmap -> Bitmap -> IO ()+lazyXorInplace target other+  | target == other = clear target+  | otherwise = lazyXorInplaceDistinct target other+{-# INLINE lazyXorInplace #-}++foreign import capi safe "roaring/roaring.h roaring_bitmap_lazy_xor_inplace"+  lazyXorInplaceDistinct :: Bitmap -> Bitmap -> IO ()++-- | @roaring_bitmap_repair_after_lazy@+foreign import capi safe "roaring/roaring.h roaring_bitmap_repair_after_lazy"+  repairAfterLazy :: Bitmap -> IO ()++-- | @roaring_bitmap_flip@: @[start, end)@, @end@ capped at @2^32@. NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_flip"+  flip :: Bitmap -> Word64 -> Word64 -> IO Bitmap++-- | @roaring_bitmap_flip_closed@: NULL on failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_flip_closed"+  flipClosed :: Bitmap -> Word32 -> Word32 -> IO Bitmap++-- | @roaring_bitmap_flip_inplace@: @[start, end)@, @end@ capped at @2^32@.+foreign import capi safe "roaring/roaring.h roaring_bitmap_flip_inplace"+  flipInplace :: Bitmap -> Word64 -> Word64 -> IO ()++-- | @roaring_bitmap_flip_inplace_closed@+foreign import capi safe "roaring/roaring.h roaring_bitmap_flip_inplace_closed"+  flipInplaceClosed :: Bitmap -> Word32 -> Word32 -> IO ()++-- | @roaring_bitmap_add_offset@: NULL on failure. Values shifted outside @[0, 2^32)@ are dropped.+foreign import capi safe "roaring/roaring.h roaring_bitmap_add_offset"+  addOffset :: Bitmap -> Int64 -> IO Bitmap++-- | @roaring_bitmap_to_uint32_array@: the output must hold 'getCardinality' values.+foreign import capi safe "roaring/roaring.h roaring_bitmap_to_uint32_array"+  toUint32Array :: Bitmap -> Ptr Word32 -> IO ()++-- | @roaring_bitmap_range_uint32_array@: up to @limit@ values from the 0-based @offset@, both truncated to 32 bits. Always returns 'True', so it does not tell how many values it wrote.+foreign import capi safe "roaring/roaring.h roaring_bitmap_range_uint32_array"+  rangeUint32Array :: Bitmap -> CSize -> CSize -> Ptr Word32 -> IO Bool++-- | @roaring_bitmap_to_bitset@: the bitset must be empty. 'False' on allocation failure.+foreign import capi safe "roaring/roaring.h roaring_bitmap_to_bitset"+  toBitset :: Bitmap -> Bitset -> IO Bool++{- | @roaring_bitmap_word_range_to_bitset@: writes @n_words@ 64-bit words, starting at word @first_word@.+Value @64 * (first_word + k) + i@ sets bit @i@ of word @k@.+All words are overwritten.+The sum @first_word + n_words@ must not exceed @2^26@.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_word_range_to_bitset"+  wordRangeToBitset :: Bitmap -> Word32 -> Word32 -> Ptr Word64 -> IO ()++{- | @roaring_bitmap_run_optimize@: converts containers to or from runs, whichever is smaller.+'True' if the result has a run container.++Adding or removing single values never converts a run container to another kind.+Call it again after many such edits to recover the space.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_run_optimize"+  runOptimize :: Bitmap -> IO Bool++-- | @roaring_bitmap_remove_run_compression@: 'True' if anything changed.+foreign import capi safe "roaring/roaring.h roaring_bitmap_remove_run_compression"+  removeRunCompression :: Bitmap -> IO Bool++-- | @roaring_bitmap_shrink_to_fit@: returns the bytes saved.+foreign import capi safe "roaring/roaring.h roaring_bitmap_shrink_to_fit"+  shrinkToFit :: Bitmap -> IO CSize++-- | @roaring_bitmap_statistics@+foreign import capi safe "roaring/roaring.h roaring_bitmap_statistics"+  statistics :: Bitmap -> Ptr Statistics -> IO ()++getStatistics :: Bitmap -> IO Statistics+getStatistics bitmap =+  alloca $ \ptr -> do+    statistics bitmap ptr+    peek ptr++{- | @roaring_bitmap_internal_validate@: checks the internal invariants.+'True' if the bitmap is consistent.+If a check fails and the pointer is not NULL, the pointer receives a static C string that names the first problem.+The caller does not free that string.++Unless the bytes are known to come from a serialised bitmap, the loaded bitmap must pass this before any other use.+Lazy results fail it until 'repairAfterLazy'.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_internal_validate"+  internalValidate :: Bitmap -> Ptr (ConstPtr CChar) -> IO Bool++-- | @roaring_bitmap_size_in_bytes@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_size_in_bytes"+  sizeInBytes :: Bitmap -> IO CSize++-- | @roaring_bitmap_serialize@: CRoaring's own format, which other Roaring implementations cannot read. The buffer must hold 'sizeInBytes'.+foreign import capi safe "roaring/roaring.h roaring_bitmap_serialize"+  serialize :: Bitmap -> Ptr CChar -> IO CSize++{- | @roaring_bitmap_deserialize@: NULL on failure.++It takes no length, so a malformed or truncated buffer can make it read past the end.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_deserialize"+  deserialize :: Ptr () -> IO Bitmap++{- | @roaring_bitmap_deserialize_safe@: reads at most the given number of bytes.+NULL on failure.++Only the reads are bounded.+Bytes that did not come from 'serialize' can still yield a broken bitmap.+Until 'internalValidate' accepts it, such a bitmap can give wrong answers, crash or corrupt memory.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_deserialize_safe"+  deserializeBounded :: Ptr () -> CSize -> IO Bitmap++{- | Runs 'deserializeBounded', then 'internalValidate'.+'Nothing' when the bytes are truncated, malformed or fail validation.+After 'Nothing', there is nothing to free.+-}+deserializeValidated :: Ptr () -> CSize -> IO (Maybe Bitmap)+deserializeValidated buffer limit = deserializeBounded buffer limit >>= validated++-- | @roaring_bitmap_portable_size_in_bytes@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_portable_size_in_bytes"+  portableSizeInBytes :: Bitmap -> IO CSize++-- | @roaring_bitmap_portable_serialize@: the buffer must hold 'portableSizeInBytes'.+foreign import capi safe "roaring/roaring.h roaring_bitmap_portable_serialize"+  portableSerialize :: Bitmap -> Ptr CChar -> IO CSize++{- | @roaring_bitmap_portable_deserialize@: NULL on failure.++It takes no length, so a malformed or truncated buffer can make it read past the end.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_portable_deserialize"+  portableDeserialize :: Ptr CChar -> IO Bitmap++{- | @roaring_bitmap_portable_deserialize_safe@: reads at most the given number of bytes.+NULL on failure.++Only the reads are bounded.+Bytes that did not come from a portable serialiser can still yield a broken bitmap.+Until 'internalValidate' accepts it, such a bitmap can give wrong answers, crash or corrupt memory.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_portable_deserialize_safe"+  portableDeserializeBounded :: Ptr CChar -> CSize -> IO Bitmap++{- | Runs 'portableDeserializeBounded', then 'internalValidate'.+'Nothing' when the bytes are truncated, malformed or fail validation.+After 'Nothing', there is nothing to free.+-}+portableDeserializeValidated :: Ptr CChar -> CSize -> IO (Maybe Bitmap)+portableDeserializeValidated buffer limit = portableDeserializeBounded buffer limit >>= validated++validated :: Bitmap -> IO (Maybe Bitmap)+validated bitmap+  | bitmap == Bitmap nullPtr = pure Nothing+  | otherwise = do+      valid <- internalValidate bitmap nullPtr+      if valid then pure $ Just bitmap else Nothing <$ free bitmap++-- | @roaring_bitmap_portable_deserialize_size@: reads at most the given number of bytes. Returns 0 if the whole bitmap does not fit in them or is malformed.+foreign import capi safe "roaring/roaring.h roaring_bitmap_portable_deserialize_size"+  portableDeserializeSize :: Ptr CChar -> CSize -> IO CSize++{- | @roaring_bitmap_portable_deserialize_frozen@: a read-only view into the portable-format buffer.+NULL on failure and on big-endian hosts.++It takes no length, so a malformed or truncated buffer can make it read past the end.+The buffer must stay alive and unmodified until 'free' releases the view.+The view must not be modified.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_portable_deserialize_frozen"+  portableDeserializeFrozen :: Ptr CChar -> IO Bitmap++-- | @roaring_bitmap_frozen_size_in_bytes@+foreign import capi unsafe "roaring/roaring.h roaring_bitmap_frozen_size_in_bytes"+  frozenSizeInBytes :: Bitmap -> IO CSize++{- | @roaring_bitmap_frozen_serialize@: the format mirrors the in-memory layout, so it depends on the CRoaring version and host endianness.+The buffer must hold 'frozenSizeInBytes'.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_frozen_serialize"+  frozenSerialize :: Bitmap -> Ptr CChar -> IO ()++{- | @roaring_bitmap_frozen_view@: a @const@ bitmap viewing a 'frozenSerialize' buffer of exactly the given length.+NULL on failure, including when the buffer is not 32-byte aligned.++Only the header is checked, so a buffer that did not come from 'frozenSerialize' can yield a bitmap that misbehaves.+Pass @v'Bitmap' ('unConstPtr' view)@ to read-only functions and to 'free'.+Modifying the view is undefined behaviour.+The buffer must stay alive and unmodified until 'free' releases the view.+-}+foreign import capi safe "roaring/roaring.h roaring_bitmap_frozen_view"+  frozenView :: Ptr CChar -> CSize -> IO (ConstPtr Bitmap)++{- | Validates a 'portableDeserializeFrozen' view of a whole bitmap within the given number of bytes, and frees it when the action ends.+'Nothing' when the bytes do not hold a valid bitmap, and on big-endian hosts.++Before it makes the view, it checks that the container offsets in the header stay within those bytes.+C follows these offsets without checks.++The buffer must stay unmodified during the action.+The view must not be modified, and must not escape the action.+-}+withPortableFrozen :: Ptr CChar -> CSize -> (Bitmap -> IO a) -> IO (Maybe a)+withPortableFrozen buffer limit action = do+  whole <- portableDeserializeSize buffer limit+  contained <- if whole == 0 then pure False else payloadsWithin buffer $ fromIntegral whole+  if contained then viewing (portableDeserializeFrozen buffer) action else pure Nothing++payloadsWithin :: Ptr CChar -> Word64 -> IO Bool+payloadsWithin buffer whole = do+  cookie <- littleEndian buffer 0 4+  let runFlagged = cookie .&. 0xFFFF == #{const SERIAL_COOKIE}+  count <- if runFlagged then pure $ cookie `shiftR` 16 + 1 else littleEndian buffer 4 4+  let+    descriptors = if runFlagged then 4 + (count + 7) `div` 8 else 8+    offsets = descriptors + 4 * count+    containersFrom index+      | index == count = pure True+      | otherwise = do+          within <- payloadWithin index+          if within then containersFrom $ index + 1 else pure False+    payloadWithin index = do+      cardinality <- (+ 1) <$> littleEndian buffer (descriptors + 4 * index + 2) 2+      offset <- littleEndian buffer (offsets + 4 * index) 4+      run <- isRun index+      if run then runsWithin offset else pure $ offset + arrayOrBitsetBytes cardinality <= whole+    isRun index+      | runFlagged = (`testBit` fromIntegral (index `mod` 8)) <$> littleEndian buffer (4 + index `div` 8) 1+      | otherwise = pure False+    runsWithin offset+      | offset + 2 > whole = pure False+      | otherwise = do+          runs <- littleEndian buffer offset 2+          pure $ offset + 2 + 4 * runs <= whole+  if runFlagged && count < #{const NO_OFFSET_THRESHOLD} then pure True else containersFrom 0+  where+    arrayOrBitsetBytes cardinality+      | cardinality > #{const DEFAULT_MAX_SIZE} = 8 * #{const BITSET_CONTAINER_SIZE_IN_WORDS}+      | otherwise = 2 * cardinality++littleEndian :: Ptr CChar -> Word64 -> Int -> IO Word64+littleEndian buffer at width = foldr prepend 0 <$> peekArray width (buffer `plusPtr` fromIntegral at :: Ptr Word8)+  where+    prepend byte rest = rest `shiftL` 8 .|. fromIntegral byte++{- | Validates a 'frozenView' of a buffer of exactly the given length, and frees it when the action ends.++'Nothing' when the bytes do not hold a valid bitmap or the buffer is not 32-byte aligned.+The buffer must stay unmodified during the action.+The view must not be modified, and must not escape the action.+-}+withFrozenView :: Ptr CChar -> CSize -> (Bitmap -> IO a) -> IO (Maybe a)+withFrozenView buffer bytes = viewing $ Bitmap . unConstPtr <$> frozenView buffer bytes++viewing :: IO Bitmap -> (Bitmap -> IO a) -> IO (Maybe a)+viewing construct action = bracket (construct >>= validated) (mapM_ free) $ traverse action++-- | @roaring_iterator@: returning 'False' stops the iteration.+type Callback = Word32 -> Ptr () -> IO Bool++-- | Release the result with 'Foreign.Ptr.freeHaskellFunPtr'.+foreign import ccall "wrapper"+  wrapCallback :: Callback -> IO (FunPtr Callback)++-- | @roaring_iterate@: 'True' if the callback never stopped the iteration.+foreign import capi safe "roaring/roaring.h roaring_iterate"+  iterate :: Bitmap -> FunPtr Callback -> Ptr () -> IO Bool++-- | @roaring_iterator64@: returning 'False' stops the iteration.+type Callback64 = Word64 -> Ptr () -> IO Bool++-- | Release the result with 'Foreign.Ptr.freeHaskellFunPtr'.+foreign import ccall "wrapper"+  wrapCallback64 :: Callback64 -> IO (FunPtr Callback64)++{- | @roaring_iterate64@: each value is widened and OR-ed with the given word.+The word is not shifted.+To place a key in the upper half, pass @key * 2^32@.+-}+foreign import capi safe "roaring/roaring.h roaring_iterate64"+  iterate64 :: Bitmap -> FunPtr Callback64 -> Word64 -> Ptr () -> IO Bool++{- | Runs 'iterate' with a Haskell function.+'True' if the function never returned 'False'.++An exception from the function stops the iteration and is rethrown after it, instead of unwinding through C.+Each value costs a call into Haskell, about 100 ns.+For reading in bulk, 'Data.Bitmap.Roaring.FFI.Bitmap.Iterator.read' is much faster.+-}+iterateWith :: Bitmap -> (Word32 -> IO Bool) -> IO Bool+iterateWith bitmap visit = Callback.with wrapCallback visit $ \callback -> iterate bitmap callback nullPtr++-- | Like 'iterateWith', but for 'iterate64'. Each value is OR-ed with the given word.+iterate64With :: Bitmap -> Word64 -> (Word64 -> IO Bool) -> IO Bool+iterate64With bitmap highBits visit = Callback.with wrapCallback64 visit $ \callback -> iterate64 bitmap callback highBits nullPtr++-- | @roaring_bitmap_printf@: writes to C's @stdout@.+foreign import capi safe "roaring/roaring.h roaring_bitmap_printf"+  printf :: Bitmap -> IO ()++-- | @roaring_bitmap_printf_describe@: writes to C's @stdout@.+foreign import capi safe "roaring/roaring.h roaring_bitmap_printf_describe"+  printfDescribe :: Bitmap -> IO ()
+ src/Data/Bitmap/Roaring/FFI/Bitmap/BulkContext.hsc view
@@ -0,0 +1,44 @@+{-| C struct @roaring_bulk_context_t@ from @roaring/roaring.h@.+It is the cursor for 'Data.Bitmap.Roaring.FFI.Bitmap.addBulk' and 'Data.Bitmap.Roaring.FFI.Bitmap.containsBulk'.++The caller allocates it with 'size' and 'alignment' and zero-fills it before the first use.+A context belongs to one bitmap.+Any other change to that bitmap invalidates the context.+Passing an invalidated context is undefined behaviour.+Zero-fill it again before the next use.+-}+module Data.Bitmap.Roaring.FFI.Bitmap.BulkContext+  ( BulkContext (..)+  , size+  , alignment+  , with+  ) where++#include <roaring/roaring.h>++import Foreign.Marshal.Alloc (allocaBytesAligned)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable)++-- | @roaring_bulk_context_t@ in caller-owned memory.+newtype BulkContext = BulkContext (Ptr BulkContext)+  deriving (Eq, Ord, Show, Storable)++-- | @sizeof(roaring_bulk_context_t)@+size :: Int+size = #{size roaring_bulk_context_t}+{-# INLINE size #-}++-- | @alignof(roaring_bulk_context_t)@+alignment :: Int+alignment = #{alignment roaring_bulk_context_t}+{-# INLINE alignment #-}++-- | A zero-filled context in temporary memory, valid only during the action.+with :: (BulkContext -> IO a) -> IO a+with action =+  allocaBytesAligned size alignment $ \ptr -> do+    fillBytes ptr 0 size+    action $ BulkContext ptr+{-# INLINE with #-}
+ src/Data/Bitmap/Roaring/FFI/Bitmap/Iterator.hsc view
@@ -0,0 +1,179 @@+{-# LANGUAGE CApiFFI #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++{-| Raw bindings to the iterator part of @roaring/roaring.h@.+A @roaring_uint32_iterator_t@ iterates over a 32-bit bitmap.++An iterator lives either in caller memory or on the C heap.+In caller memory, allocate it with 'size' and 'alignment', then position it with 'init' or 'initLast'.+On the C heap, make it with 'create' and release it with 'free'.++An iterator points into its bitmap.+The bitmap must outlive the iterator, and any change to the bitmap invalidates the iterator.+-}+module Data.Bitmap.Roaring.FFI.Bitmap.Iterator+  ( Iterator (..)+  , size+  , alignment+  , currentValue+  , hasValue+  , init+  , initLast+  , with+  , withLast+  , create+  , copy+  , free+  , finalizerFree+  , advance+  , previous+  , moveEqualOrLarger+  , read+  , readBackward+  , skip+  , skipBackward+  , Range (..)+  , readRanges+  , readPrevRanges+  ) where++#include <roaring/roaring.h>++import Prelude hiding (init, read)++import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap (..))+import Data.Word (Word32)+import Foreign.C.Types (CBool (..), CSize (..))+import Foreign.ForeignPtr (FinalizerPtr)+import Foreign.Marshal.Alloc (allocaBytesAligned)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable (peek, peekByteOff, poke, pokeByteOff, sizeOf))+import Foreign.Storable qualified as Storable++-- | @roaring_uint32_iterator_t@+newtype Iterator = Iterator (Ptr Iterator)+  deriving (Eq, Ord, Show, Storable)++-- | @sizeof(roaring_uint32_iterator_t)@+size :: Int+size = #{size roaring_uint32_iterator_t}+{-# INLINE size #-}++-- | @alignof(roaring_uint32_iterator_t)@+alignment :: Int+alignment = #{alignment roaring_uint32_iterator_t}+{-# INLINE alignment #-}++-- | The @current_value@ field, meaningful only while 'hasValue' is 'True'.+currentValue :: Iterator -> IO Word32+currentValue (Iterator ptr) = #{peek roaring_uint32_iterator_t, current_value} ptr+{-# INLINE currentValue #-}++-- | The @has_value@ field.+hasValue :: Iterator -> IO Bool+hasValue (Iterator ptr) = (/= 0) <$> (#{peek roaring_uint32_iterator_t, has_value} ptr :: IO CBool)+{-# INLINE hasValue #-}++-- | @roaring_iterator_init@: positions an iterator in caller memory at the smallest value. It holds no allocation and needs no 'free'.+foreign import capi unsafe "roaring/roaring.h roaring_iterator_init"+  init :: Bitmap -> Iterator -> IO ()++-- | @roaring_iterator_init_last@+foreign import capi unsafe "roaring/roaring.h roaring_iterator_init_last"+  initLast :: Bitmap -> Iterator -> IO ()++{- | An iterator in temporary memory at the smallest value ('init'), valid only during the action.++The bitmap must not be modified while the action runs, and the iterator must not escape the action.+-}+with :: Bitmap -> (Iterator -> IO a) -> IO a+with = positioned init+{-# INLINE with #-}++-- | Like 'with', but from the largest value. The same rules apply.+withLast :: Bitmap -> (Iterator -> IO a) -> IO a+withLast = positioned initLast+{-# INLINE withLast #-}++positioned :: (Bitmap -> Iterator -> IO ()) -> Bitmap -> (Iterator -> IO a) -> IO a+positioned start bitmap action =+  allocaBytesAligned size alignment $ \ptr -> do+    let iterator = Iterator ptr+    start bitmap iterator+    action iterator+{-# INLINE positioned #-}++-- | @roaring_iterator_create@: positioned at the smallest value. NULL on allocation failure.+foreign import capi unsafe "roaring/roaring.h roaring_iterator_create"+  create :: Bitmap -> IO Iterator++-- | @roaring_uint32_iterator_copy@: NULL on allocation failure.+foreign import capi unsafe "roaring/roaring.h roaring_uint32_iterator_copy"+  copy :: Iterator -> IO Iterator++-- | @roaring_uint32_iterator_free@: only for iterators from 'create' or 'copy'.+foreign import capi unsafe "roaring/roaring.h roaring_uint32_iterator_free"+  free :: Iterator -> IO ()++-- | @&roaring_uint32_iterator_free@+foreign import capi unsafe "roaring/roaring.h &roaring_uint32_iterator_free"+  finalizerFree :: FinalizerPtr Iterator++-- | @roaring_uint32_iterator_advance@: returns 'hasValue'. Do not call it again after 'False'.+foreign import capi unsafe "roaring/roaring.h roaring_uint32_iterator_advance"+  advance :: Iterator -> IO Bool++-- | @roaring_uint32_iterator_previous@: returns 'hasValue'. Do not call it again after 'False'.+foreign import capi unsafe "roaring/roaring.h roaring_uint32_iterator_previous"+  previous :: Iterator -> IO Bool++-- | @roaring_uint32_iterator_move_equalorlarger@: returns 'hasValue'.+foreign import capi unsafe "roaring/roaring.h roaring_uint32_iterator_move_equalorlarger"+  moveEqualOrLarger :: Iterator -> Word32 -> IO Bool++-- | @roaring_uint32_iterator_read@: returns how many values it wrote. Fewer than asked means the iterator is exhausted.+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_read"+  read :: Iterator -> Ptr Word32 -> Word32 -> IO Word32++-- | @roaring_uint32_iterator_read_backward@+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_read_backward"+  readBackward :: Iterator -> Ptr Word32 -> Word32 -> IO Word32++-- | @roaring_uint32_iterator_skip@+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_skip"+  skip :: Iterator -> Word32 -> IO Word32++-- | @roaring_uint32_iterator_skip_backward@+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_skip_backward"+  skipBackward :: Iterator -> Word32 -> IO Word32++-- | @roaring_uint32_range_closed_t@: a maximal run of consecutive values @[min, max]@.+data Range = Range+  { min :: Word32+  , max :: Word32+  }+  deriving (Eq, Ord, Show)++instance Storable Range where+  sizeOf _ = #{size roaring_uint32_range_closed_t}+  {-# INLINE sizeOf #-}+  alignment _ = #{alignment roaring_uint32_range_closed_t}+  {-# INLINE alignment #-}+  peek ptr =+    Range+      <$> #{peek roaring_uint32_range_closed_t, min} ptr+      <*> #{peek roaring_uint32_range_closed_t, max} ptr+  {-# INLINE peek #-}+  poke ptr range = do+    #{poke roaring_uint32_range_closed_t, min} ptr range.min+    #{poke roaring_uint32_range_closed_t, max} ptr range.max+  {-# INLINE poke #-}++-- | @roaring_uint32_iterator_read_ranges@+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_read_ranges"+  readRanges :: Iterator -> Ptr Range -> CSize -> IO CSize++-- | @roaring_uint32_iterator_read_prev_ranges@+foreign import capi safe "roaring/roaring.h roaring_uint32_iterator_read_prev_ranges"+  readPrevRanges :: Iterator -> Ptr Range -> CSize -> IO CSize
+ src/Data/Bitmap/Roaring/FFI/Bitmap/Statistics.hsc view
@@ -0,0 +1,71 @@+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++-- | C struct @roaring_statistics_t@ from @roaring/roaring_types.h@, filled in by 'Data.Bitmap.Roaring.FFI.Bitmap.statistics'.+module Data.Bitmap.Roaring.FFI.Bitmap.Statistics+  ( Statistics (..)+  ) where++#include <roaring/roaring.h>++import Data.Word (Word32, Word64)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Storable (Storable (..))++-- | @roaring_statistics_t@ without the deprecated, always-zero @sum_value@ field, which 'poke' zero-fills.+data Statistics = Statistics+  { containers :: Word32+  , arrayContainers :: Word32+  , runContainers :: Word32+  , bitsetContainers :: Word32+  , valuesArrayContainers :: Word32+  , valuesRunContainers :: Word32+  , valuesBitsetContainers :: Word32+  , bytesArrayContainers :: Word32+  , bytesRunContainers :: Word32+  , bytesBitsetContainers :: Word32+  , maxValue :: Word32+  -- ^ Undefined when the cardinality is zero.+  , minValue :: Word32+  -- ^ Undefined when the cardinality is zero.+  , cardinality :: Word64+  }+  deriving (Eq, Ord, Show)++instance Storable Statistics where+  sizeOf _ = #{size roaring_statistics_t}+  {-# INLINE sizeOf #-}+  alignment _ = #{alignment roaring_statistics_t}+  {-# INLINE alignment #-}+  peek ptr =+    Statistics+      <$> #{peek roaring_statistics_t, n_containers} ptr+      <*> #{peek roaring_statistics_t, n_array_containers} ptr+      <*> #{peek roaring_statistics_t, n_run_containers} ptr+      <*> #{peek roaring_statistics_t, n_bitset_containers} ptr+      <*> #{peek roaring_statistics_t, n_values_array_containers} ptr+      <*> #{peek roaring_statistics_t, n_values_run_containers} ptr+      <*> #{peek roaring_statistics_t, n_values_bitset_containers} ptr+      <*> #{peek roaring_statistics_t, n_bytes_array_containers} ptr+      <*> #{peek roaring_statistics_t, n_bytes_run_containers} ptr+      <*> #{peek roaring_statistics_t, n_bytes_bitset_containers} ptr+      <*> #{peek roaring_statistics_t, max_value} ptr+      <*> #{peek roaring_statistics_t, min_value} ptr+      <*> #{peek roaring_statistics_t, cardinality} ptr+  {-# INLINE peek #-}+  poke ptr statistics = do+    fillBytes ptr 0 #{size roaring_statistics_t}+    #{poke roaring_statistics_t, n_containers} ptr statistics.containers+    #{poke roaring_statistics_t, n_array_containers} ptr statistics.arrayContainers+    #{poke roaring_statistics_t, n_run_containers} ptr statistics.runContainers+    #{poke roaring_statistics_t, n_bitset_containers} ptr statistics.bitsetContainers+    #{poke roaring_statistics_t, n_values_array_containers} ptr statistics.valuesArrayContainers+    #{poke roaring_statistics_t, n_values_run_containers} ptr statistics.valuesRunContainers+    #{poke roaring_statistics_t, n_values_bitset_containers} ptr statistics.valuesBitsetContainers+    #{poke roaring_statistics_t, n_bytes_array_containers} ptr statistics.bytesArrayContainers+    #{poke roaring_statistics_t, n_bytes_run_containers} ptr statistics.bytesRunContainers+    #{poke roaring_statistics_t, n_bytes_bitset_containers} ptr statistics.bytesBitsetContainers+    #{poke roaring_statistics_t, max_value} ptr statistics.maxValue+    #{poke roaring_statistics_t, min_value} ptr statistics.minValue+    #{poke roaring_statistics_t, cardinality} ptr statistics.cardinality+  {-# INLINE poke #-}
+ src/Data/Bitmap/Roaring/FFI/Bitmap64.hs view
@@ -0,0 +1,513 @@+{-# LANGUAGE CApiFFI #-}++{-| Raw bindings to @roaring/roaring64.h@ for 64-bit Roaring bitmaps, @roaring64_bitmap_t@.++Each binding is the C function without its @roaring64_bitmap_@ prefix, camelCased.+The @_safe@ loader only bounds how far it reads, so it is named 'portableDeserializeBounded' here.+'portableDeserializeValidated' also runs 'internalValidate'.+Use it for bytes from outside the program.+Handles carry no ownership.+Release the bitmaps that C returns with 'free' or 'finalizerFree'.+A bitmap is not synchronised.+Concurrent use needs external locking.+-}+module Data.Bitmap.Roaring.FFI.Bitmap64+  ( Bitmap64 (..)+  , create+  , free+  , finalizerFree+  , with+  , fromRange+  , ofPtr+  , moveFromRoaring32+  , copy+  , overwrite+  , clear+  , add+  , addChecked+  , addMany+  , addBulk+  , addRange+  , addRangeClosed+  , remove+  , removeChecked+  , removeMany+  , removeBulk+  , removeRange+  , removeRangeClosed+  , contains+  , containsBulk+  , containsRange+  , containsRangeClosed+  , getCardinality+  , rangeCardinality+  , rangeClosedCardinality+  , isEmpty+  , minimum+  , maximum+  , select+  , rank+  , getIndex+  , equals+  , isSubset+  , isStrictSubset+  , intersect+  , intersectWithRange+  , jaccardIndex+  , and+  , andInplace+  , andCardinality+  , or+  , orInplace+  , orCardinality+  , xor+  , xorInplace+  , xorCardinality+  , andnot+  , andnotInplace+  , andnotCardinality+  , flip+  , flipClosed+  , flipInplace+  , flipClosedInplace+  , addOffsetSigned+  , addOffset+  , subOffset+  , toUint64Array+  , runOptimize+  , removeRunCompression+  , shrinkToFit+  , statistics+  , getStatistics+  , internalValidate+  , portableSizeInBytes+  , portableSerialize+  , portableDeserializeBounded+  , portableDeserializeValidated+  , portableDeserializeSize+  , portableDeserializeFrozen+  , frozenSizeInBytes+  , frozenSerialize+  , frozenView+  , withPortableFrozen+  , withFrozenView+  , iterate+  , iterateWith+  ) where++import Prelude hiding (and, flip, iterate, maximum, minimum, or)++import Control.Exception (bracket)+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap (..), Callback64, wrapCallback64)+import Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap64.Statistics (Statistics)+import Data.Bitmap.Roaring.FFI.Callback qualified as Callback+import Data.Bitmap.Roaring.FFI.NullHandle qualified as NullHandle+import Data.Word (Word64)+import Foreign.C.ConstPtr (ConstPtr (..))+import Foreign.C.Types (CChar, CSize (..))+import Foreign.ForeignPtr (FinalizerPtr)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Ptr (FunPtr, Ptr, nullPtr)+import Foreign.Storable (Storable, peek)++-- | @roaring64_bitmap_t@. Its 'Storable' instance stores the pointer itself.+newtype Bitmap64 = Bitmap64 (Ptr Bitmap64)+  deriving (Eq, Ord, Show, Storable)++-- | @roaring64_bitmap_create@: NULL on allocation failure.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_create"+  create :: IO Bitmap64++-- | @roaring64_bitmap_free@: also releases read-only views.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_free"+  free :: Bitmap64 -> IO ()++-- | @&roaring64_bitmap_free@+foreign import capi unsafe "roaring/roaring64.h &roaring64_bitmap_free"+  finalizerFree :: FinalizerPtr Bitmap64++-- | Frees the bitmap when the action ends. Throws t'NullHandle.NullHandle' if the constructor returns NULL.+with :: IO Bitmap64 -> (Bitmap64 -> IO a) -> IO a+with = NullHandle.bracket "Data.Bitmap.Roaring.FFI.Bitmap64.with" free+{-# INLINE with #-}++-- | @roaring64_bitmap_from_range@: @[min, max)@ every @step@ from @min@. NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_from_range"+  fromRange :: Word64 -> Word64 -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_of_ptr@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_of_ptr"+  ofPtr :: CSize -> Ptr Word64 -> IO Bitmap64++-- | @roaring64_bitmap_move_from_roaring32@: empties the 32-bit bitmap, which still has to be freed.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_move_from_roaring32"+  moveFromRoaring32 :: Bitmap -> IO Bitmap64++-- | @roaring64_bitmap_copy@: NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_copy"+  copy :: Bitmap64 -> IO Bitmap64++-- | @roaring64_bitmap_overwrite@: the two may be the same bitmap.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_overwrite"+  overwrite :: Bitmap64 -> Bitmap64 -> IO ()++-- | @roaring64_bitmap_clear@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_clear"+  clear :: Bitmap64 -> IO ()++-- | @roaring64_bitmap_add@+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_add"+  add :: Bitmap64 -> Word64 -> IO ()++-- | @roaring64_bitmap_add_checked@: 'True' if the value was absent.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_add_checked"+  addChecked :: Bitmap64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_add_many@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_add_many"+  addMany :: Bitmap64 -> CSize -> Ptr Word64 -> IO ()++-- | @roaring64_bitmap_add_bulk@: see "Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext" for the context's rules.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_add_bulk"+  addBulk :: Bitmap64 -> BulkContext -> Word64 -> IO ()++-- | @roaring64_bitmap_add_range@: @[min, max)@.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_add_range"+  addRange :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_add_range_closed@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_add_range_closed"+  addRangeClosed :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_remove@+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_remove"+  remove :: Bitmap64 -> Word64 -> IO ()++-- | @roaring64_bitmap_remove_checked@: 'True' if the value was present.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_remove_checked"+  removeChecked :: Bitmap64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_remove_many@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_remove_many"+  removeMany :: Bitmap64 -> CSize -> Ptr Word64 -> IO ()++-- | @roaring64_bitmap_remove_bulk@: see "Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext" for the context's rules.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_remove_bulk"+  removeBulk :: Bitmap64 -> BulkContext -> Word64 -> IO ()++-- | @roaring64_bitmap_remove_range@: @[min, max)@.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_remove_range"+  removeRange :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_remove_range_closed@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_remove_range_closed"+  removeRangeClosed :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_contains@+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_contains"+  contains :: Bitmap64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_contains_bulk@: see "Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext" for the context's rules.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_contains_bulk"+  containsBulk :: Bitmap64 -> BulkContext -> Word64 -> IO Bool++-- | @roaring64_bitmap_contains_range@: @[min, max)@.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_contains_range"+  containsRange :: Bitmap64 -> Word64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_contains_range_closed@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_contains_range_closed"+  containsRangeClosed :: Bitmap64 -> Word64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_get_cardinality@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_get_cardinality"+  getCardinality :: Bitmap64 -> IO Word64++-- | @roaring64_bitmap_range_cardinality@: @[min, max)@.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_range_cardinality"+  rangeCardinality :: Bitmap64 -> Word64 -> Word64 -> IO Word64++-- | @roaring64_bitmap_range_closed_cardinality@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_range_closed_cardinality"+  rangeClosedCardinality :: Bitmap64 -> Word64 -> Word64 -> IO Word64++-- | @roaring64_bitmap_is_empty@+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_is_empty"+  isEmpty :: Bitmap64 -> IO Bool++-- | @roaring64_bitmap_minimum@: @UINT64_MAX@ when empty.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_minimum"+  minimum :: Bitmap64 -> IO Word64++-- | @roaring64_bitmap_maximum@: 0 when empty.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_maximum"+  maximum :: Bitmap64 -> IO Word64++-- | @roaring64_bitmap_select@: the rank is 0-based. 'False' when it is not below the cardinality.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_select"+  select :: Bitmap64 -> Word64 -> Ptr Word64 -> IO Bool++-- | @roaring64_bitmap_rank@: the number of values less than or equal to the given one.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_rank"+  rank :: Bitmap64 -> Word64 -> IO Word64++-- | @roaring64_bitmap_get_index@: the index is 0-based. 'False' when the value is absent.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_get_index"+  getIndex :: Bitmap64 -> Word64 -> Ptr Word64 -> IO Bool++-- | @roaring64_bitmap_equals@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_equals"+  equals :: Bitmap64 -> Bitmap64 -> IO Bool++-- | @roaring64_bitmap_is_subset@: whether the first is a subset of the second.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_is_subset"+  isSubset :: Bitmap64 -> Bitmap64 -> IO Bool++-- | @roaring64_bitmap_is_strict_subset@: whether the first is a strict subset of the second.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_is_strict_subset"+  isStrictSubset :: Bitmap64 -> Bitmap64 -> IO Bool++-- | @roaring64_bitmap_intersect@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_intersect"+  intersect :: Bitmap64 -> Bitmap64 -> IO Bool++-- | @roaring64_bitmap_intersect_with_range@: @[min, max)@.+foreign import capi unsafe "roaring/roaring64.h roaring64_bitmap_intersect_with_range"+  intersectWithRange :: Bitmap64 -> Word64 -> Word64 -> IO Bool++-- | @roaring64_bitmap_jaccard_index@: NaN when both are empty.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_jaccard_index"+  jaccardIndex :: Bitmap64 -> Bitmap64 -> IO Double++-- | @roaring64_bitmap_and@: NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_and"+  and :: Bitmap64 -> Bitmap64 -> IO Bitmap64++-- | @roaring64_bitmap_and_inplace@: the two may be the same bitmap.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_and_inplace"+  andInplace :: Bitmap64 -> Bitmap64 -> IO ()++-- | @roaring64_bitmap_and_cardinality@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_and_cardinality"+  andCardinality :: Bitmap64 -> Bitmap64 -> IO Word64++-- | @roaring64_bitmap_or@: NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_or"+  or :: Bitmap64 -> Bitmap64 -> IO Bitmap64++-- | @roaring64_bitmap_or_inplace@: passing the same bitmap twice leaves it unchanged.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_or_inplace"+  orInplace :: Bitmap64 -> Bitmap64 -> IO ()++-- | @roaring64_bitmap_or_cardinality@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_or_cardinality"+  orCardinality :: Bitmap64 -> Bitmap64 -> IO Word64++-- | @roaring64_bitmap_xor@: NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_xor"+  xor :: Bitmap64 -> Bitmap64 -> IO Bitmap64++-- | @roaring64_bitmap_xor_inplace@: passing the same bitmap twice clears it.+xorInplace :: Bitmap64 -> Bitmap64 -> IO ()+xorInplace target other+  | target == other = clear target+  | otherwise = xorInplaceDistinct target other+{-# INLINE xorInplace #-}++foreign import capi safe "roaring/roaring64.h roaring64_bitmap_xor_inplace"+  xorInplaceDistinct :: Bitmap64 -> Bitmap64 -> IO ()++-- | @roaring64_bitmap_xor_cardinality@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_xor_cardinality"+  xorCardinality :: Bitmap64 -> Bitmap64 -> IO Word64++-- | @roaring64_bitmap_andnot@: first minus second. NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_andnot"+  andnot :: Bitmap64 -> Bitmap64 -> IO Bitmap64++-- | @roaring64_bitmap_andnot_inplace@: first minus second. Passing the same bitmap twice clears it.+andnotInplace :: Bitmap64 -> Bitmap64 -> IO ()+andnotInplace target other+  | target == other = clear target+  | otherwise = andnotInplaceDistinct target other+{-# INLINE andnotInplace #-}++foreign import capi safe "roaring/roaring64.h roaring64_bitmap_andnot_inplace"+  andnotInplaceDistinct :: Bitmap64 -> Bitmap64 -> IO ()++-- | @roaring64_bitmap_andnot_cardinality@: the size of first minus second.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_andnot_cardinality"+  andnotCardinality :: Bitmap64 -> Bitmap64 -> IO Word64++-- | @roaring64_bitmap_flip@: @[min, max)@. NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_flip"+  flip :: Bitmap64 -> Word64 -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_flip_closed@: NULL on failure.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_flip_closed"+  flipClosed :: Bitmap64 -> Word64 -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_flip_inplace@: @[min, max)@.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_flip_inplace"+  flipInplace :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_flip_closed_inplace@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_flip_closed_inplace"+  flipClosedInplace :: Bitmap64 -> Word64 -> Word64 -> IO ()++-- | @roaring64_bitmap_add_offset_signed@: shifts up when 'True', down when 'False'. Values leaving the 64-bit range are dropped.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_add_offset_signed"+  addOffsetSigned :: Bitmap64 -> Bool -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_add_offset@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_add_offset"+  addOffset :: Bitmap64 -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_sub_offset@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_sub_offset"+  subOffset :: Bitmap64 -> Word64 -> IO Bitmap64++-- | @roaring64_bitmap_to_uint64_array@: the output must hold 'getCardinality' values.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_to_uint64_array"+  toUint64Array :: Bitmap64 -> Ptr Word64 -> IO ()++{- | @roaring64_bitmap_run_optimize@: converts containers to or from runs, whichever is smaller.+'True' if the result has a run container.++Adding or removing single values never converts a run container to another kind.+Call it again after many such edits to recover the space.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_run_optimize"+  runOptimize :: Bitmap64 -> IO Bool++-- | @roaring64_bitmap_remove_run_compression@: 'True' if anything changed.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_remove_run_compression"+  removeRunCompression :: Bitmap64 -> IO Bool++-- | @roaring64_bitmap_shrink_to_fit@: returns the bytes freed.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_shrink_to_fit"+  shrinkToFit :: Bitmap64 -> IO CSize++-- | @roaring64_bitmap_statistics@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_statistics"+  statistics :: Bitmap64 -> Ptr Statistics -> IO ()++getStatistics :: Bitmap64 -> IO Statistics+getStatistics bitmap =+  alloca $ \ptr -> do+    statistics bitmap ptr+    peek ptr++{- | @roaring64_bitmap_internal_validate@: checks the internal invariants.+'True' if the bitmap is consistent.+If a check fails and the pointer is not NULL, the pointer receives a static C string that names the first problem.+The caller does not free that string.++Unless the bytes are known to come from a serialised bitmap, the loaded bitmap must pass this before any other use.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_internal_validate"+  internalValidate :: Bitmap64 -> Ptr (ConstPtr CChar) -> IO Bool++-- | @roaring64_bitmap_portable_size_in_bytes@+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_portable_size_in_bytes"+  portableSizeInBytes :: Bitmap64 -> IO CSize++-- | @roaring64_bitmap_portable_serialize@: the buffer must hold 'portableSizeInBytes'.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_portable_serialize"+  portableSerialize :: Bitmap64 -> Ptr CChar -> IO CSize++{- | @roaring64_bitmap_portable_deserialize_safe@: reads at most the given number of bytes.+NULL on failure.++Only the reads are bounded.+Bytes that did not come from a portable serialiser can still yield a broken bitmap.+Until 'internalValidate' accepts it, such a bitmap can give wrong answers, crash or corrupt memory.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_portable_deserialize_safe"+  portableDeserializeBounded :: Ptr CChar -> CSize -> IO Bitmap64++{- | Runs 'portableDeserializeBounded', then 'internalValidate'.+'Nothing' when the bytes are truncated, malformed or fail validation.+After 'Nothing', there is nothing to free.+-}+portableDeserializeValidated :: Ptr CChar -> CSize -> IO (Maybe Bitmap64)+portableDeserializeValidated buffer limit = portableDeserializeBounded buffer limit >>= validated++validated :: Bitmap64 -> IO (Maybe Bitmap64)+validated bitmap+  | bitmap == Bitmap64 nullPtr = pure Nothing+  | otherwise = do+      valid <- internalValidate bitmap nullPtr+      if valid then pure $ Just bitmap else Nothing <$ free bitmap++-- | @roaring64_bitmap_portable_deserialize_size@: reads at most the given number of bytes. Returns 0 if no whole bitmap is found.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_portable_deserialize_size"+  portableDeserializeSize :: Ptr CChar -> CSize -> IO CSize++{- | @roaring64_bitmap_portable_deserialize_frozen@: a read-only view into the portable-format buffer, reading at most the given number of bytes.+NULL on failure and on big-endian hosts.++Only the reads are bounded.+Bytes that did not come from a portable serialiser can still yield a bitmap that misbehaves until 'internalValidate' accepts it.+The buffer must stay alive and unmodified until 'free' releases the view.+The view must not be modified.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_portable_deserialize_frozen"+  portableDeserializeFrozen :: Ptr CChar -> CSize -> IO Bitmap64++-- | @roaring64_bitmap_frozen_size_in_bytes@: call 'shrinkToFit' first.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_frozen_size_in_bytes"+  frozenSizeInBytes :: Bitmap64 -> IO CSize++{- | @roaring64_bitmap_frozen_serialize@: the format mirrors the in-memory layout, so it depends on the CRoaring version and host endianness.+Call 'shrinkToFit' first.+The buffer must hold 'frozenSizeInBytes'.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_frozen_serialize"+  frozenSerialize :: Bitmap64 -> Ptr CChar -> IO CSize++{- | @roaring64_bitmap_frozen_view@: a read-only bitmap viewing a 'frozenSerialize' buffer, reading at most the given number of bytes.+NULL on failure, including when the buffer is not 64-byte aligned.++Bytes that did not come from 'frozenSerialize' can yield a bitmap that misbehaves.+The buffer must stay alive and unmodified until 'free' releases the view.+The view must not be modified.+-}+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_frozen_view"+  frozenView :: Ptr CChar -> CSize -> IO Bitmap64++{- | Validates a 'portableDeserializeFrozen' view, reading at most the given number of bytes, and frees it when the action ends.++'Nothing' when the bytes do not hold a valid bitmap, and on big-endian hosts.+The buffer must stay unmodified during the action.+The view must not be modified, and must not escape the action.+-}+withPortableFrozen :: Ptr CChar -> CSize -> (Bitmap64 -> IO a) -> IO (Maybe a)+withPortableFrozen buffer limit = viewing $ portableDeserializeFrozen buffer limit++{- | Validates a 'frozenView', reading at most the given number of bytes, and frees it when the action ends.++'Nothing' when the bytes do not hold a valid bitmap or the buffer is not 64-byte aligned.+The buffer must stay unmodified during the action.+The view must not be modified, and must not escape the action.+-}+withFrozenView :: Ptr CChar -> CSize -> (Bitmap64 -> IO a) -> IO (Maybe a)+withFrozenView buffer limit = viewing $ frozenView buffer limit++viewing :: IO Bitmap64 -> (Bitmap64 -> IO a) -> IO (Maybe a)+viewing construct action = bracket (construct >>= validated) (mapM_ free) $ traverse action++-- | @roaring64_bitmap_iterate@: 'True' if the callback never stopped the iteration.+foreign import capi safe "roaring/roaring64.h roaring64_bitmap_iterate"+  iterate :: Bitmap64 -> FunPtr Callback64 -> Ptr () -> IO Bool++{- | Runs 'iterate' with a Haskell function.+'True' if the function never returned 'False'.++An exception from the function stops the iteration and is rethrown after it, instead of unwinding through C.+Each value costs a call into Haskell, about 100 ns.+For reading in bulk, 'Data.Bitmap.Roaring.FFI.Bitmap64.Iterator.read' is much faster.+-}+iterateWith :: Bitmap64 -> (Word64 -> IO Bool) -> IO Bool+iterateWith bitmap visit = Callback.with wrapCallback64 visit $ \callback -> iterate bitmap callback nullPtr
+ src/Data/Bitmap/Roaring/FFI/Bitmap64/BulkContext.hsc view
@@ -0,0 +1,44 @@+{-| C struct @roaring64_bulk_context_t@ from @roaring/roaring64.h@.+It is the cursor for 'Data.Bitmap.Roaring.FFI.Bitmap64.addBulk', 'Data.Bitmap.Roaring.FFI.Bitmap64.removeBulk' and 'Data.Bitmap.Roaring.FFI.Bitmap64.containsBulk'.++The caller allocates it with 'size' and 'alignment' and zero-fills it before the first use.+A context belongs to one bitmap.+Any other change to that bitmap invalidates the context.+Passing an invalidated context is undefined behaviour.+Zero-fill it again before the next use.+-}+module Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext+  ( BulkContext (..)+  , size+  , alignment+  , with+  ) where++#include <roaring/roaring64.h>++import Foreign.Marshal.Alloc (allocaBytesAligned)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable)++-- | @roaring64_bulk_context_t@ in caller-owned memory.+newtype BulkContext = BulkContext (Ptr BulkContext)+  deriving (Eq, Ord, Show, Storable)++-- | @sizeof(roaring64_bulk_context_t)@+size :: Int+size = #{size roaring64_bulk_context_t}+{-# INLINE size #-}++-- | @alignof(roaring64_bulk_context_t)@+alignment :: Int+alignment = #{alignment roaring64_bulk_context_t}+{-# INLINE alignment #-}++-- | A zero-filled context in temporary memory, valid only during the action.+with :: (BulkContext -> IO a) -> IO a+with action =+  allocaBytesAligned size alignment $ \ptr -> do+    fillBytes ptr 0 size+    action $ BulkContext ptr+{-# INLINE with #-}
+ src/Data/Bitmap/Roaring/FFI/Bitmap64/Iterator.hsc view
@@ -0,0 +1,151 @@+{-# LANGUAGE CApiFFI #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++{-| Raw bindings to the iterator part of @roaring/roaring64.h@.+The opaque @roaring64_iterator_t@ iterates over a 64-bit bitmap.++Iterators live on the C heap.+Release each one with 'free'.++An iterator points into its bitmap.+The bitmap must outlive the iterator, and any change to the bitmap invalidates the iterator.+-}+module Data.Bitmap.Roaring.FFI.Bitmap64.Iterator+  ( Iterator (..)+  , create+  , createLast+  , with+  , withLast+  , reinit+  , reinitLast+  , copy+  , free+  , finalizerFree+  , hasValue+  , value+  , advance+  , previous+  , moveEqualOrLarger+  , read+  , readBackward+  , Range (..)+  , readRanges+  , readPrevRanges+  ) where++#include <roaring/roaring64.h>++import Prelude hiding (read)++import Data.Bitmap.Roaring.FFI.Bitmap64 (Bitmap64 (..))+import Data.Bitmap.Roaring.FFI.NullHandle qualified as NullHandle+import Data.Word (Word64)+import Foreign.C.Types (CSize (..))+import Foreign.ForeignPtr (FinalizerPtr)+import Foreign.Ptr (Ptr)+import Foreign.Storable (Storable (..))++-- | @roaring64_iterator_t@+newtype Iterator = Iterator (Ptr Iterator)+  deriving (Eq, Ord, Show, Storable)++-- | @roaring64_iterator_create@: positioned at the smallest value. NULL on allocation failure.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_create"+  create :: Bitmap64 -> IO Iterator++-- | @roaring64_iterator_create_last@: NULL on allocation failure.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_create_last"+  createLast :: Bitmap64 -> IO Iterator++-- | @roaring64_iterator_reinit@: repositions at the smallest value of the given bitmap, without allocating.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_reinit"+  reinit :: Bitmap64 -> Iterator -> IO ()++-- | @roaring64_iterator_reinit_last@: without allocating.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_reinit_last"+  reinitLast :: Bitmap64 -> Iterator -> IO ()++-- | @roaring64_iterator_copy@: NULL on allocation failure.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_copy"+  copy :: Iterator -> IO Iterator++-- | @roaring64_iterator_free@+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_free"+  free :: Iterator -> IO ()++-- | @&roaring64_iterator_free@+foreign import capi unsafe "roaring/roaring64.h &roaring64_iterator_free"+  finalizerFree :: FinalizerPtr Iterator++{- | An iterator at the smallest value ('create'), freed when the action ends.++Throws t'NullHandle.NullHandle' if 'create' returns NULL.+The bitmap must not be modified while the action runs, and the iterator must not escape the action.+-}+with :: Bitmap64 -> (Iterator -> IO a) -> IO a+with bitmap = NullHandle.bracket "Data.Bitmap.Roaring.FFI.Bitmap64.Iterator.with" free $ create bitmap+{-# INLINE with #-}++-- | Like 'with', but from the largest value. The same rules apply.+withLast :: Bitmap64 -> (Iterator -> IO a) -> IO a+withLast bitmap = NullHandle.bracket "Data.Bitmap.Roaring.FFI.Bitmap64.Iterator.withLast" free $ createLast bitmap+{-# INLINE withLast #-}++-- | @roaring64_iterator_has_value@+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_has_value"+  hasValue :: Iterator -> IO Bool++-- | @roaring64_iterator_value@: meaningful only while 'hasValue' is 'True'.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_value"+  value :: Iterator -> IO Word64++-- | @roaring64_iterator_advance@: returns 'hasValue'. Do not call it again after 'False'.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_advance"+  advance :: Iterator -> IO Bool++-- | @roaring64_iterator_previous@: returns 'hasValue'. Do not call it again after 'False'.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_previous"+  previous :: Iterator -> IO Bool++-- | @roaring64_iterator_move_equalorlarger@: returns 'hasValue'.+foreign import capi unsafe "roaring/roaring64.h roaring64_iterator_move_equalorlarger"+  moveEqualOrLarger :: Iterator -> Word64 -> IO Bool++-- | @roaring64_iterator_read@: returns how many values it wrote. Fewer than asked means the iterator is exhausted.+foreign import capi safe "roaring/roaring64.h roaring64_iterator_read"+  read :: Iterator -> Ptr Word64 -> Word64 -> IO Word64++-- | @roaring64_iterator_read_backward@+foreign import capi safe "roaring/roaring64.h roaring64_iterator_read_backward"+  readBackward :: Iterator -> Ptr Word64 -> Word64 -> IO Word64++-- | @roaring64_range_closed_t@: a maximal run of consecutive values @[min, max]@.+data Range = Range+  { min :: Word64+  , max :: Word64+  }+  deriving (Eq, Ord, Show)++instance Storable Range where+  sizeOf _ = #{size roaring64_range_closed_t}+  {-# INLINE sizeOf #-}+  alignment _ = #{alignment roaring64_range_closed_t}+  {-# INLINE alignment #-}+  peek ptr =+    Range+      <$> #{peek roaring64_range_closed_t, min} ptr+      <*> #{peek roaring64_range_closed_t, max} ptr+  {-# INLINE peek #-}+  poke ptr range = do+    #{poke roaring64_range_closed_t, min} ptr range.min+    #{poke roaring64_range_closed_t, max} ptr range.max+  {-# INLINE poke #-}++-- | @roaring64_iterator_read_ranges@+foreign import capi safe "roaring/roaring64.h roaring64_iterator_read_ranges"+  readRanges :: Iterator -> Ptr Range -> CSize -> IO CSize++-- | @roaring64_iterator_read_prev_ranges@+foreign import capi safe "roaring/roaring64.h roaring64_iterator_read_prev_ranges"+  readPrevRanges :: Iterator -> Ptr Range -> CSize -> IO CSize
+ src/Data/Bitmap/Roaring/FFI/Bitmap64/Statistics.hsc view
@@ -0,0 +1,69 @@+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++-- | C struct @roaring64_statistics_t@ from @roaring/roaring_types.h@, filled in by 'Data.Bitmap.Roaring.FFI.Bitmap64.statistics'.+module Data.Bitmap.Roaring.FFI.Bitmap64.Statistics+  ( Statistics (..)+  ) where++#include <roaring/roaring.h>++import Data.Word (Word64)+import Foreign.Storable (Storable (..))++-- | @roaring64_statistics_t@+data Statistics = Statistics+  { containers :: Word64+  , arrayContainers :: Word64+  , runContainers :: Word64+  , bitsetContainers :: Word64+  , valuesArrayContainers :: Word64+  , valuesRunContainers :: Word64+  , valuesBitsetContainers :: Word64+  , bytesArrayContainers :: Word64+  , bytesRunContainers :: Word64+  , bytesBitsetContainers :: Word64+  , maxValue :: Word64+  -- ^ Undefined when the cardinality is zero.+  , minValue :: Word64+  -- ^ Undefined when the cardinality is zero.+  , cardinality :: Word64+  }+  deriving (Eq, Ord, Show)++instance Storable Statistics where+  sizeOf _ = #{size roaring64_statistics_t}+  {-# INLINE sizeOf #-}+  alignment _ = #{alignment roaring64_statistics_t}+  {-# INLINE alignment #-}+  peek ptr =+    Statistics+      <$> #{peek roaring64_statistics_t, n_containers} ptr+      <*> #{peek roaring64_statistics_t, n_array_containers} ptr+      <*> #{peek roaring64_statistics_t, n_run_containers} ptr+      <*> #{peek roaring64_statistics_t, n_bitset_containers} ptr+      <*> #{peek roaring64_statistics_t, n_values_array_containers} ptr+      <*> #{peek roaring64_statistics_t, n_values_run_containers} ptr+      <*> #{peek roaring64_statistics_t, n_values_bitset_containers} ptr+      <*> #{peek roaring64_statistics_t, n_bytes_array_containers} ptr+      <*> #{peek roaring64_statistics_t, n_bytes_run_containers} ptr+      <*> #{peek roaring64_statistics_t, n_bytes_bitset_containers} ptr+      <*> #{peek roaring64_statistics_t, max_value} ptr+      <*> #{peek roaring64_statistics_t, min_value} ptr+      <*> #{peek roaring64_statistics_t, cardinality} ptr+  {-# INLINE peek #-}+  poke ptr statistics = do+    #{poke roaring64_statistics_t, n_containers} ptr statistics.containers+    #{poke roaring64_statistics_t, n_array_containers} ptr statistics.arrayContainers+    #{poke roaring64_statistics_t, n_run_containers} ptr statistics.runContainers+    #{poke roaring64_statistics_t, n_bitset_containers} ptr statistics.bitsetContainers+    #{poke roaring64_statistics_t, n_values_array_containers} ptr statistics.valuesArrayContainers+    #{poke roaring64_statistics_t, n_values_run_containers} ptr statistics.valuesRunContainers+    #{poke roaring64_statistics_t, n_values_bitset_containers} ptr statistics.valuesBitsetContainers+    #{poke roaring64_statistics_t, n_bytes_array_containers} ptr statistics.bytesArrayContainers+    #{poke roaring64_statistics_t, n_bytes_run_containers} ptr statistics.bytesRunContainers+    #{poke roaring64_statistics_t, n_bytes_bitset_containers} ptr statistics.bytesBitsetContainers+    #{poke roaring64_statistics_t, max_value} ptr statistics.maxValue+    #{poke roaring64_statistics_t, min_value} ptr statistics.minValue+    #{poke roaring64_statistics_t, cardinality} ptr statistics.cardinality+  {-# INLINE poke #-}
+ src/Data/Bitmap/Roaring/FFI/Bitset.hsc view
@@ -0,0 +1,294 @@+{-# LANGUAGE CApiFFI #-}++{-| Raw bindings to @roaring/bitset/bitset.h@ for CRoaring's growable plain bitset, @bitset_t@.++Each binding is the C function without its @bitset_@ or @bitsets_@ prefix, camelCased.+C creates and frees bitsets.+The handles here carry no ownership.+-}+module Data.Bitmap.Roaring.FFI.Bitset+  ( Bitset (..)+  , array+  , arraySize+  , capacity+  , create+  , createWithCapacity+  , free+  , finalizerFree+  , with+  , copy+  , clear+  , fill+  , resize+  , grow+  , trim+  , sizeInBytes+  , sizeInBits+  , sizeInWords+  , shiftLeft+  , shiftRight+  , set+  , setToValue+  , get+  , count+  , empty+  , minimum+  , maximum+  , inplaceUnion+  , unionCount+  , inplaceIntersection+  , intersectionCount+  , disjoint+  , intersect+  , containsAll+  , inplaceDifference+  , differenceCount+  , inplaceSymmetricDifference+  , symmetricDifferenceCount+  , nextSetBit+  , nextSetBits+  , Callback+  , wrapCallback+  , forEach+  , forEachWith+  , print+  ) where++#include <roaring/bitset/bitset.h>++import Prelude hiding (maximum, minimum, print)++import Data.Bitmap.Roaring.FFI.Callback qualified as Callback+import Data.Bitmap.Roaring.FFI.NullHandle qualified as NullHandle+import Data.Word (Word64)+import Foreign.C.Types (CSize (..))+import Foreign.ForeignPtr (FinalizerPtr)+import Foreign.Ptr (FunPtr, Ptr, nullPtr)+import Foreign.Storable (Storable (..))++-- | @bitset_t@+newtype Bitset = Bitset (Ptr Bitset)+  deriving (Eq, Ord, Show, Storable)++{- | The @array@ field holds the backing buffer of 64-bit words.+Bit @i@ lives in bit @i mod 64@ of word @i div 64@.++The pointer is invalidated by any call that resizes the bitset.+-}+array :: Bitset -> IO (Ptr Word64)+array (Bitset ptr) = #{peek bitset_t, array} ptr+{-# INLINE array #-}++-- | The @arraysize@ field holds the number of words in use.+arraySize :: Bitset -> IO CSize+arraySize (Bitset ptr) = #{peek bitset_t, arraysize} ptr+{-# INLINE arraySize #-}++-- | The @capacity@ field holds the number of words allocated.+capacity :: Bitset -> IO CSize+capacity (Bitset ptr) = #{peek bitset_t, capacity} ptr+{-# INLINE capacity #-}++-- | @bitset_create@: NULL on allocation failure.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_create"+  create :: IO Bitset++-- | @bitset_create_with_capacity@: a bitset of the given number of bits, all zero. NULL on allocation failure.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_create_with_capacity"+  createWithCapacity :: CSize -> IO Bitset++-- | @bitset_free@+foreign import capi unsafe "roaring/bitset/bitset.h bitset_free"+  free :: Bitset -> IO ()++-- | @&bitset_free@+foreign import capi unsafe "roaring/bitset/bitset.h &bitset_free"+  finalizerFree :: FinalizerPtr Bitset++-- | Frees the bitset when the action ends. Throws t'NullHandle.NullHandle' if the constructor returns NULL.+with :: IO Bitset -> (Bitset -> IO a) -> IO a+with = NullHandle.bracket "Data.Bitmap.Roaring.FFI.Bitset.with" free+{-# INLINE with #-}++-- | @bitset_copy@: NULL on allocation failure.+foreign import capi safe "roaring/bitset/bitset.h bitset_copy"+  copy :: Bitset -> IO Bitset++-- | @bitset_clear@: keeps the size.+foreign import capi safe "roaring/bitset/bitset.h bitset_clear"+  clear :: Bitset -> IO ()++-- | @bitset_fill@: sets every bit within the current size.+foreign import capi safe "roaring/bitset/bitset.h bitset_fill"+  fill :: Bitset -> IO ()++{- | @bitset_resize@: resizes to the given number of words.+New words are zeroed when the flag is set.+'False' on allocation failure.+-}+foreign import capi safe "roaring/bitset/bitset.h bitset_resize"+  resize :: Bitset -> CSize -> Bool -> IO Bool++{- | @bitset_grow@: grows to at least the given number of words, zero-padded.+'False' on allocation failure, and when the bitset already has more words.+-}+foreign import capi safe "roaring/bitset/bitset.h bitset_grow"+  grow :: Bitset -> CSize -> IO Bool++-- | @bitset_trim@: 'False' on reallocation failure.+foreign import capi safe "roaring/bitset/bitset.h bitset_trim"+  trim :: Bitset -> IO Bool++-- | @bitset_size_in_bytes@+foreign import capi unsafe "roaring/bitset/bitset.h bitset_size_in_bytes"+  sizeInBytes :: Bitset -> IO CSize++-- | @bitset_size_in_bits@: bits accessible without growing.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_size_in_bits"+  sizeInBits :: Bitset -> IO CSize++-- | @bitset_size_in_words@+foreign import capi unsafe "roaring/bitset/bitset.h bitset_size_in_words"+  sizeInWords :: Bitset -> IO CSize++-- | @bitset_shift_left@: grows the bitset. If growing fails, the bitset silently stays unchanged.+foreign import capi safe "roaring/bitset/bitset.h bitset_shift_left"+  shiftLeft :: Bitset -> CSize -> IO ()++-- | @bitset_shift_right@+foreign import capi safe "roaring/bitset/bitset.h bitset_shift_right"+  shiftRight :: Bitset -> CSize -> IO ()++-- | @bitset_set@: grows the bitset if needed. If growing fails, the bit is silently not set.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_set"+  set :: Bitset -> CSize -> IO ()++-- | @bitset_set_to_value@: grows the bitset if needed. If growing fails, the call silently does nothing.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_set_to_value"+  setToValue :: Bitset -> CSize -> Bool -> IO ()++-- | @bitset_get@: bits beyond the size read as 'False'.+foreign import capi unsafe "roaring/bitset/bitset.h bitset_get"+  get :: Bitset -> CSize -> IO Bool++-- | @bitset_count@+foreign import capi safe "roaring/bitset/bitset.h bitset_count"+  count :: Bitset -> IO CSize++-- | @bitset_empty@+foreign import capi safe "roaring/bitset/bitset.h bitset_empty"+  empty :: Bitset -> IO Bool++-- | @bitset_minimum@: @SIZE_MAX@ when no bit is set.+foreign import capi safe "roaring/bitset/bitset.h bitset_minimum"+  minimum :: Bitset -> IO CSize++-- | @bitset_maximum@: 0 when no bit is set.+foreign import capi safe "roaring/bitset/bitset.h bitset_maximum"+  maximum :: Bitset -> IO CSize++-- | @bitset_inplace_union@: 'False' if growing the first failed. Passing the same bitset twice leaves it unchanged.+inplaceUnion :: Bitset -> Bitset -> IO Bool+inplaceUnion target other+  | target == other = pure True+  | otherwise = inplaceUnionDistinct target other+{-# INLINE inplaceUnion #-}++foreign import capi safe "roaring/bitset/bitset.h bitset_inplace_union"+  inplaceUnionDistinct :: Bitset -> Bitset -> IO Bool++-- | @bitset_union_count@+foreign import capi safe "roaring/bitset/bitset.h bitset_union_count"+  unionCount :: Bitset -> Bitset -> IO CSize++-- | @bitset_inplace_intersection@: passing the same bitset twice leaves it unchanged.+inplaceIntersection :: Bitset -> Bitset -> IO ()+inplaceIntersection target other+  | target == other = pure ()+  | otherwise = inplaceIntersectionDistinct target other+{-# INLINE inplaceIntersection #-}++foreign import capi safe "roaring/bitset/bitset.h bitset_inplace_intersection"+  inplaceIntersectionDistinct :: Bitset -> Bitset -> IO ()++-- | @bitset_intersection_count@+foreign import capi safe "roaring/bitset/bitset.h bitset_intersection_count"+  intersectionCount :: Bitset -> Bitset -> IO CSize++-- | @bitsets_disjoint@+foreign import capi safe "roaring/bitset/bitset.h bitsets_disjoint"+  disjoint :: Bitset -> Bitset -> IO Bool++-- | @bitsets_intersect@+foreign import capi safe "roaring/bitset/bitset.h bitsets_intersect"+  intersect :: Bitset -> Bitset -> IO Bool++-- | @bitset_contains_all@: whether the first has every bit set in the second.+foreign import capi safe "roaring/bitset/bitset.h bitset_contains_all"+  containsAll :: Bitset -> Bitset -> IO Bool++-- | @bitset_inplace_difference@: first minus second. Passing the same bitset twice clears it.+inplaceDifference :: Bitset -> Bitset -> IO ()+inplaceDifference target other+  | target == other = clear target+  | otherwise = inplaceDifferenceDistinct target other+{-# INLINE inplaceDifference #-}++foreign import capi safe "roaring/bitset/bitset.h bitset_inplace_difference"+  inplaceDifferenceDistinct :: Bitset -> Bitset -> IO ()++-- | @bitset_difference_count@: the size of first minus second.+foreign import capi safe "roaring/bitset/bitset.h bitset_difference_count"+  differenceCount :: Bitset -> Bitset -> IO CSize++-- | @bitset_inplace_symmetric_difference@: 'False' if growing the first failed. Passing the same bitset twice clears it.+inplaceSymmetricDifference :: Bitset -> Bitset -> IO Bool+inplaceSymmetricDifference target other+  | target == other = True <$ clear target+  | otherwise = inplaceSymmetricDifferenceDistinct target other+{-# INLINE inplaceSymmetricDifference #-}++foreign import capi safe "roaring/bitset/bitset.h bitset_inplace_symmetric_difference"+  inplaceSymmetricDifferenceDistinct :: Bitset -> Bitset -> IO Bool++-- | @bitset_symmetric_difference_count@+foreign import capi safe "roaring/bitset/bitset.h bitset_symmetric_difference_count"+  symmetricDifferenceCount :: Bitset -> Bitset -> IO CSize++{- | @bitset_next_set_bit@: moves the index behind the pointer to the first set bit at or after it.+'False' when there is none.+The index then stays unchanged.+-}+foreign import capi safe "roaring/bitset/bitset.h bitset_next_set_bit"+  nextSetBit :: Bitset -> Ptr CSize -> IO Bool++{- | @bitset_next_set_bits@: writes the indices of set bits into the buffer, up to the given capacity.+It starts at the index behind the last pointer.+It returns how many indices it wrote, and stores the last index it wrote at that pointer.+To continue, start from one past that index.+-}+foreign import capi safe "roaring/bitset/bitset.h bitset_next_set_bits"+  nextSetBits :: Bitset -> Ptr CSize -> CSize -> Ptr CSize -> IO CSize++-- | @bitset_iterator@: returning 'False' stops the iteration.+type Callback = CSize -> Ptr () -> IO Bool++-- | Release the result with 'Foreign.Ptr.freeHaskellFunPtr'.+foreign import ccall "wrapper"+  wrapCallback :: Callback -> IO (FunPtr Callback)++-- | @bitset_for_each@: 'True' if the callback never stopped the iteration.+foreign import capi safe "roaring/bitset/bitset.h bitset_for_each"+  forEach :: Bitset -> FunPtr Callback -> Ptr () -> IO Bool++{- | Runs 'forEach' with a Haskell function.+'True' if the function never returned 'False'.++An exception from the function stops the iteration and is rethrown after it, instead of unwinding through C.+-}+forEachWith :: Bitset -> (CSize -> IO Bool) -> IO Bool+forEachWith bitset visit = Callback.with wrapCallback visit $ \callback -> forEach bitset callback nullPtr++-- | @bitset_print@: writes to C's @stdout@.+foreign import capi safe "roaring/bitset/bitset.h bitset_print"+  print :: Bitset -> IO ()
+ src/Data/Bitmap/Roaring/FFI/Callback.hs view
@@ -0,0 +1,21 @@+-- | Running Haskell functions as CRoaring iteration callbacks without leaking the function pointer or unwinding an exception through C.+module Data.Bitmap.Roaring.FFI.Callback (with) where++import Control.Exception (SomeException, bracket, catch, evaluate, throwIO)+import Data.IORef (IORef, newIORef, readIORef, writeIORef)+import Foreign.Ptr (FunPtr, Ptr, freeHaskellFunPtr)++-- | An exception from the visiting function stops the iteration as if it had returned 'False', and is rethrown once the iteration has returned.+with+  :: ((element -> Ptr () -> IO Bool) -> IO (FunPtr (element -> Ptr () -> IO Bool)))+  -> (element -> IO Bool)+  -> (FunPtr (element -> Ptr () -> IO Bool) -> IO Bool)+  -> IO Bool+with wrap visit run = do+  failure <- newIORef Nothing+  completed <- bracket (wrap $ callback failure) freeHaskellFunPtr run+  readIORef failure >>= maybe (pure completed) throwIO+  where+    callback failure element _ = (visit element >>= evaluate) `catch` stop failure+    stop :: IORef (Maybe SomeException) -> SomeException -> IO Bool+    stop failure exception = False <$ writeIORef failure (Just exception)
+ src/Data/Bitmap/Roaring/FFI/Memory.hsc view
@@ -0,0 +1,94 @@+{-# LANGUAGE CApiFFI #-}+{-# LANGUAGE OverloadedRecordDot #-}+{-# LANGUAGE NoFieldSelectors #-}++{-| Raw bindings to @roaring/memory.h@.+It sets the allocator that CRoaring uses for every bitmap, container, iterator and bitset.++The hook is process-wide.+Install it before CRoaring allocates anything.+Never change it while any CRoaring structure is alive.+Otherwise a different allocator frees memory that the old one allocated.++Every hook must be a C function, not a Haskell function wrapped with @foreign import ccall "wrapper"@.+CRoaring allocates inside @unsafe@ foreign calls, which must not re-enter Haskell.+-}+module Data.Bitmap.Roaring.FFI.Memory+  ( Hooks (..)+  , Malloc+  , Realloc+  , Calloc+  , Free+  , AlignedMalloc+  , AlignedFree+  , initMemoryHook+  ) where++#include <roaring/memory.h>++import Foreign.C.Types (CSize)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Ptr (FunPtr, Ptr)+import Foreign.Storable (Storable (..))++-- | @roaring_malloc_p@+type Malloc = CSize -> IO (Ptr ())++-- | @roaring_realloc_p@+type Realloc = Ptr () -> CSize -> IO (Ptr ())++-- | @roaring_calloc_p@+type Calloc = CSize -> CSize -> IO (Ptr ())++-- | @roaring_free_p@: must accept and ignore NULL.+type Free = Ptr () -> IO ()++-- | @roaring_aligned_malloc_p@: alignment first, size second.+type AlignedMalloc = CSize -> CSize -> IO (Ptr ())++-- | @roaring_aligned_free_p@: frees blocks from 'AlignedMalloc'. It must accept and ignore NULL.+type AlignedFree = Ptr () -> IO ()++-- | @roaring_memory_t@+data Hooks = Hooks+  { malloc :: FunPtr Malloc+  , realloc :: FunPtr Realloc+  , calloc :: FunPtr Calloc+  , free :: FunPtr Free+  , alignedMalloc :: FunPtr AlignedMalloc+  , alignedFree :: FunPtr AlignedFree+  }+  deriving (Eq, Show)++instance Storable Hooks where+  sizeOf _ = #{size roaring_memory_t}+  {-# INLINE sizeOf #-}+  alignment _ = #{alignment roaring_memory_t}+  {-# INLINE alignment #-}+  peek ptr =+    Hooks+      <$> #{peek roaring_memory_t, malloc} ptr+      <*> #{peek roaring_memory_t, realloc} ptr+      <*> #{peek roaring_memory_t, calloc} ptr+      <*> #{peek roaring_memory_t, free} ptr+      <*> #{peek roaring_memory_t, aligned_malloc} ptr+      <*> #{peek roaring_memory_t, aligned_free} ptr+  {-# INLINE peek #-}+  poke ptr hooks = do+    #{poke roaring_memory_t, malloc} ptr hooks.malloc+    #{poke roaring_memory_t, realloc} ptr hooks.realloc+    #{poke roaring_memory_t, calloc} ptr hooks.calloc+    #{poke roaring_memory_t, free} ptr hooks.free+    #{poke roaring_memory_t, aligned_malloc} ptr hooks.alignedMalloc+    #{poke roaring_memory_t, aligned_free} ptr hooks.alignedFree+  {-# INLINE poke #-}++-- | @roaring_init_memory_hook@: see the module header for when it is safe to call.+initMemoryHook :: Hooks -> IO ()+initMemoryHook hooks =+  alloca $ \ptr -> do+    poke ptr hooks+    initMemoryHookFrom ptr++foreign import capi unsafe "croaring-ffi.h croaring_ffi_init_memory_hook"+  initMemoryHookFrom :: Ptr Hooks -> IO ()
+ src/Data/Bitmap/Roaring/FFI/NullHandle.hs view
@@ -0,0 +1,37 @@+{-# LANGUAGE NoFieldSelectors #-}++-- | The exception the @with@ helpers throw when their constructor returns a NULL handle.+module Data.Bitmap.Roaring.FFI.NullHandle+  ( NullHandle (..)+  , bracket+  ) where++import Control.Exception (Exception (..), throwIO)+import Control.Exception qualified as Exception+import Data.Coerce (Coercible, coerce)+import Foreign.Ptr (Ptr, nullPtr)++{- | A @with@ helper's constructor returned NULL.++The helper cannot tell why.+Depending on the constructor, NULL means failed allocation, rejected arguments or malformed bytes.+-}+newtype NullHandle = NullHandle+  { function :: String+  -- ^ The helper that received the NULL handle, such as @Data.Bitmap.Roaring.FFI.Bitmap.with@.+  }+  deriving (Eq, Show)++instance Exception NullHandle where+  displayException (NullHandle function) = function <> ": the constructor returned NULL"++{- | Runs the action on the constructed handle and releases the handle when the action ends.+If the constructor returns NULL, throws t'NullHandle' with the given name instead of running the action.+-}+bracket :: forall handle a. (Coercible handle (Ptr handle)) => String -> (handle -> IO ()) -> IO handle -> (handle -> IO a) -> IO a+bracket function release construct = Exception.bracket (construct >>= checked) release+  where+    checked handle+      | coerce handle == (nullPtr :: Ptr handle) = throwIO $ NullHandle function+      | otherwise = pure handle+{-# INLINE bracket #-}
+ src/Data/Bitmap/Roaring/FFI/Version.hsc view
@@ -0,0 +1,28 @@+-- | Version of the bundled CRoaring library, from @roaring/roaring_version.h@.+module Data.Bitmap.Roaring.FFI.Version+  ( version+  , major+  , minor+  , revision+  ) where++#include <roaring/roaring_version.h>++-- | @ROARING_VERSION@, e.g. @"5.5.5"@.+version :: String+version = #{const_str ROARING_VERSION}++-- | @ROARING_VERSION_MAJOR@+major :: Int+major = #{const ROARING_VERSION_MAJOR}+{-# INLINE major #-}++-- | @ROARING_VERSION_MINOR@+minor :: Int+minor = #{const ROARING_VERSION_MINOR}+{-# INLINE minor #-}++-- | @ROARING_VERSION_REVISION@+revision :: Int+revision = #{const ROARING_VERSION_REVISION}+{-# INLINE revision #-}
+ test/Spec.hs view
@@ -0,0 +1,26 @@+module Main (main) where++import Data.Bitmap.Roaring.FFI.Memory qualified as Memory+import Test.Bitmap qualified+import Test.Bitmap.Iterator qualified+import Test.Bitmap64 qualified+import Test.Bitmap64.Iterator qualified+import Test.Bitset qualified+import Test.Memory qualified+import Test.Tasty (defaultMain, testGroup)+import Test.Version qualified++main :: IO ()+main = do+  Memory.initMemoryHook Test.Memory.countingHooks+  defaultMain $+    testGroup+      "croaring-ffi"+      [ Test.Bitmap.tests+      , Test.Bitmap.Iterator.tests+      , Test.Bitmap64.tests+      , Test.Bitmap64.Iterator.tests+      , Test.Bitset.tests+      , Test.Memory.tests+      , Test.Version.tests+      ]
+ test/Test/Bitmap.hs view
@@ -0,0 +1,637 @@+{-# LANGUAGE OverloadedRecordDot #-}++module Test.Bitmap+  ( tests+  , genValues+  , fromValues+  , modelRank+  , symmetricDifference+  , collect+  , visiting+  ) where++import Control.Exception (IOException, throw, try)+import Control.Monad (forM, forM_)+import Control.Monad.Trans.Resource (allocate)+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap (..))+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap.BulkContext qualified as BulkContext+import Data.Bitmap.Roaring.FFI.Bitmap.Statistics (Statistics (..))+import Data.Bitmap.Roaring.FFI.Bitset (Bitset (..))+import Data.Bitmap.Roaring.FFI.Bitset qualified as Bitset+import Data.Bitmap.Roaring.FFI.NullHandle (NullHandle (..))+import Data.IORef (IORef, modifyIORef', newIORef, readIORef)+import Data.Maybe (fromMaybe)+import Data.Set (Set)+import Data.Set qualified as Set+import Data.Word (Word32, Word64, Word8)+import Foreign.C.ConstPtr (ConstPtr (..))+import Foreign.C.String (peekCString)+import Foreign.C.Types (CChar, CSize)+import Foreign.ForeignPtr (finalizeForeignPtr, newForeignPtr, withForeignPtr)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Marshal.Array (advancePtr, allocaArray, peekArray, pokeArray, withArrayLen)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (FunPtr, Ptr, castPtr, freeHaskellFunPtr, nullPtr, plusPtr)+import Foreign.Storable (peek, poke)+import Hedgehog (Gen, Property, annotate, assert, evalIO, forAll, (===))+import Hedgehog.Gen qualified as Gen+import Hedgehog.Range qualified as Range+import Test.Property (Test, returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Bitmap"+    [ testGroup+        "marshalling"+        [ testProperty "bool results in both polarities" boolResults+        , testProperty "bool arguments through copy-on-write" boolArguments+        ]+    , testGroup+        "model"+        [ testProperty "contents and queries" contentsAndQueries+        , testProperty "set algebra" setAlgebra+        , testProperty "in-place set algebra" inplaceSetAlgebra+        , testProperty "many-bitmap operations" manyBitmaps+        , testProperty "lazy union after repair" lazyUnion+        , testProperty "ranges" ranges+        , testProperty "offset" offset+        , testProperty "rankMany and rangeUint32Array" bulkQueries+        , testProperty "copy-on-write shares and unshares containers" copyOnWrite+        ]+    , testGroup+        "serialization"+        [ testProperty "native round-trip" nativeRoundTrip+        , testProperty "portable round-trip" portableRoundTrip+        , testProperty "portable frozen round-trip" portableFrozenRoundTrip+        , testProperty "frozen view round-trip" frozenRoundTrip+        , testProperty "bounded portable load rejects a truncated buffer" truncatedPortable+        , testProperty "bounded load accepts unsorted values that validation rejects" corruptedPortable+        ]+    , testGroup+        "layout"+        [ testProperty "statistics of known containers" knownStatistics+        , testProperty "statistics poke and peek agree" statisticsStorable+        , testProperty "bitmap in caller memory" callerMemory+        , testProperty "bulk context in caller memory" bulkContext+        ]+    , testGroup+        "callbacks"+        [ testProperty "iterate visits every value" iterateAll+        , testProperty "iterate stops when the callback says so" iterateStop+        , testProperty "iterate64 ORs in the high bits" iterate64HighBits+        ]+    , testGroup+        "helpers"+        [ testProperty "with frees the bitmap and rejects NULL" scoped+        , testProperty "withPortableFrozen views valid bytes in place" portableFrozenScope+        , testProperty "withPortableFrozen rejects offsets past the bytes" portableFrozenOffsets+        , testProperty "withFrozenView views aligned bytes in place" frozenViewScope+        , testProperty "iterateWith visits, stops and rethrows" callbackHelpers+        , testProperty "in-place operations with the same bitmap twice" selfAliasing+        , testProperty "BulkContext.with" scopedBulkContext+        ]+    , testGroup+        "interop"+        [ testProperty "toBitset and wordRangeToBitset" bitsets+        , testProperty "finalizerFree releases through a ForeignPtr" finalizer+        ]+    ]++genValues :: Gen [Word32]+genValues = concat <$> Gen.list (Range.linear 0 60) (Gen.choice [pure <$> genValue, genRun])+  where+    genRun = do+      start <- genValue+      len <- Gen.int $ Range.linear 0 3000+      pure . take len $ enumFrom start++genValue :: Gen Word32+genValue =+  Gen.choice+    [ Gen.word32 $ Range.constant 0 200000+    , Gen.word32 Range.constantBounded+    , Gen.word32 $ Range.constant (maxBound - 200000) maxBound+    ]++adopt :: IO Bitmap -> Test Bitmap+adopt = Property.adopt (Bitmap nullPtr) Bitmap.free++owned :: IO Bitmap -> Test Bitmap+owned = Property.owned (Bitmap nullPtr) Bitmap.free++fromValues :: [Word32] -> Test Bitmap+fromValues values = owned . withArrayLen values $ \count ptr -> Bitmap.ofPtr (fromIntegral count) ptr++elements :: Bitmap -> IO [Word32]+elements bitmap = do+  count <- fromIntegral <$> Bitmap.getCardinality bitmap+  allocaArray count $ \ptr -> do+    Bitmap.toUint32Array bitmap ptr+    peekArray count ptr++toList :: Bitmap -> Test [Word32]+toList = evalIO . elements++visiting :: ((value -> IO Bool) -> IO Bool) -> (value -> IO Bool) -> IO (Either IOException Bool, [value])+visiting iteration decide = do+  seen <- newIORef []+  outcome <- try . iteration $ \value -> modifyIORef' seen (value :) >> decide value+  (,) outcome . reverse <$> readIORef seen++modelRank :: (Ord a, Num rank) => Set a -> a -> rank+modelRank model value = fromIntegral . Set.size $ Set.takeWhileAntitone (<= value) model++symmetricDifference :: (Ord a) => Set a -> Set a -> Set a+symmetricDifference left right = Set.union left right `Set.difference` Set.intersection left right++collect :: ((value -> Ptr () -> IO Bool) -> IO (FunPtr callback)) -> (value -> Bool) -> Test (IORef [value], FunPtr callback)+collect wrap continue = do+  seen <- evalIO $ newIORef []+  (_, callback) <- allocate (wrap $ \value _ -> modifyIORef' seen (value :) >> pure (continue value)) freeHaskellFunPtr+  pure (seen, callback)++boolResults :: Property+boolResults =+  Property.example $ do+    bitmap <- owned Bitmap.create+    Bitmap.addChecked bitmap 7 `returns` True+    Bitmap.addChecked bitmap 7 `returns` False+    Bitmap.removeChecked bitmap 7 `returns` True+    Bitmap.removeChecked bitmap 7 `returns` False+    Bitmap.isEmpty bitmap `returns` True++boolArguments :: Property+boolArguments =+  Property.example $ do+    bitmap <- owned Bitmap.create+    Bitmap.getCopyOnWrite bitmap `returns` False+    evalIO $ Bitmap.setCopyOnWrite bitmap True+    Bitmap.getCopyOnWrite bitmap `returns` True+    evalIO $ Bitmap.setCopyOnWrite bitmap False+    Bitmap.getCopyOnWrite bitmap `returns` False++contentsAndQueries :: Property+contentsAndQueries =+  Property.generated $ do+    values <- forAll genValues+    probes <- forAll $ Gen.list (Range.linear 0 50) genValue+    bitmap <- fromValues values+    let model = Set.fromList values+    toList bitmap >>= (=== Set.toAscList model)+    Bitmap.getCardinality bitmap `returns` fromIntegral (Set.size model)+    Bitmap.isEmpty bitmap `returns` Set.null model+    Bitmap.minimum bitmap `returns` maybe maxBound fst (Set.minView model)+    Bitmap.maximum bitmap `returns` maybe 0 fst (Set.maxView model)+    Bitmap.internalValidate bitmap nullPtr `returns` True+    forM (values <> probes) (probe bitmap) `returns` map (modelProbe model) (values <> probes)+    forM [0 .. fromIntegral (Set.size model)] (select bitmap) `returns` (map Just (Set.toAscList model) <> [Nothing])+  where+    probe bitmap value = (,,) <$> Bitmap.contains bitmap value <*> Bitmap.rank bitmap value <*> Bitmap.getIndex bitmap value+    modelProbe model value =+      ( Set.member value model+      , modelRank model value+      , maybe (-1) fromIntegral $ Set.lookupIndex value model+      )+    select bitmap rank =+      alloca $ \out -> do+        found <- Bitmap.select bitmap rank out+        if found then Just <$> peek out else pure Nothing++setAlgebra :: Property+setAlgebra =+  Property.generated $ do+    leftValues <- forAll genValues+    rightValues <- forAll genValues+    left <- fromValues leftValues+    right <- fromValues rightValues+    let+      leftModel = Set.fromList leftValues+      rightModel = Set.fromList rightValues+      intersection = Set.intersection leftModel rightModel+      union = Set.union leftModel rightModel+      difference = Set.difference leftModel rightModel+      symmetric = symmetricDifference leftModel rightModel+    owned (Bitmap.and left right) >>= toList >>= (=== Set.toAscList intersection)+    owned (Bitmap.or left right) >>= toList >>= (=== Set.toAscList union)+    owned (Bitmap.andnot left right) >>= toList >>= (=== Set.toAscList difference)+    owned (Bitmap.xor left right) >>= toList >>= (=== Set.toAscList symmetric)+    Bitmap.andCardinality left right `returns` fromIntegral (Set.size intersection)+    Bitmap.orCardinality left right `returns` fromIntegral (Set.size union)+    Bitmap.andnotCardinality left right `returns` fromIntegral (Set.size difference)+    Bitmap.xorCardinality left right `returns` fromIntegral (Set.size symmetric)+    Bitmap.intersect left right `returns` not (Set.disjoint leftModel rightModel)+    Bitmap.isSubset left right `returns` Set.isSubsetOf leftModel rightModel+    Bitmap.isStrictSubset left right `returns` Set.isProperSubsetOf leftModel rightModel+    Bitmap.equals left right `returns` (leftModel == rightModel)++inplaceSetAlgebra :: Property+inplaceSetAlgebra =+  Property.generated $ do+    leftValues <- forAll genValues+    rightValues <- forAll genValues+    right <- fromValues rightValues+    let+      leftModel = Set.fromList leftValues+      rightModel = Set.fromList rightValues+      applied :: (Bitmap -> Bitmap -> IO ()) -> Test [Word32]+      applied operation = do+        left <- fromValues leftValues+        evalIO $ operation left right+        toList left+    applied Bitmap.andInplace >>= (=== Set.toAscList (Set.intersection leftModel rightModel))+    applied Bitmap.orInplace >>= (=== Set.toAscList (Set.union leftModel rightModel))+    applied Bitmap.andnotInplace >>= (=== Set.toAscList (Set.difference leftModel rightModel))+    applied Bitmap.xorInplace >>= (=== Set.toAscList (symmetricDifference leftModel rightModel))+    self <- fromValues leftValues+    evalIO $ Bitmap.andInplace self self+    toList self >>= (=== Set.toAscList leftModel)++manyBitmaps :: Property+manyBitmaps =+  Property.generated $ do+    valueLists <- forAll $ Gen.list (Range.linear 0 10) genValues+    bitmaps <- mapM fromValues valueLists+    handles <- Property.array $ length bitmaps+    evalIO $ pokeArray handles bitmaps+    let+      count = length bitmaps+      union = Set.toAscList . Set.unions $ map Set.fromList valueLists+      symmetric = Set.toAscList . foldr (symmetricDifference . Set.fromList) Set.empty $ valueLists+    owned (Bitmap.orMany (fromIntegral count) handles) >>= toList >>= (=== union)+    owned (Bitmap.orManyHeap (fromIntegral count) handles) >>= toList >>= (=== union)+    owned (Bitmap.xorMany (fromIntegral count) handles) >>= toList >>= (=== symmetric)++lazyUnion :: Property+lazyUnion =+  Property.generated $ do+    leftValues <- forAll genValues+    rightValues <- forAll genValues+    left <- fromValues leftValues+    right <- fromValues rightValues+    union <- owned $ Bitmap.lazyOr left right False+    evalIO $ do+      Bitmap.lazyOrInplace union left True+      Bitmap.repairAfterLazy union+    Bitmap.internalValidate union nullPtr `returns` True+    toList union >>= (=== Set.toAscList (Set.fromList leftValues `Set.union` Set.fromList rightValues))++ranges :: Property+ranges =+  Property.generated $ do+    values <- forAll genValues+    start <- forAll genValue+    len <- forAll . Gen.word64 $ Range.linear 0 100000+    bitmap <- fromValues values+    let+      model = Set.fromList values+      end = min (fromIntegral start + len) $ 2 ^ (32 :: Int)+      spanned = Set.fromList . take (fromIntegral end - fromIntegral start) $ enumFrom start+      inRange = Set.intersection model spanned+      modified :: (Bitmap -> Word64 -> Word64 -> IO ()) -> Test [Word32]+      modified operation = do+        copy <- fromValues values+        evalIO $ operation copy (fromIntegral start) end+        toList copy+    Bitmap.rangeCardinality bitmap (fromIntegral start) end `returns` fromIntegral (Set.size inRange)+    Bitmap.containsRange bitmap (fromIntegral start) end `returns` (inRange == spanned)+    Bitmap.intersectWithRange bitmap (fromIntegral start) end `returns` not (Set.null inRange)+    owned (Bitmap.flip bitmap (fromIntegral start) end) >>= toList >>= (=== Set.toAscList (symmetricDifference model spanned))+    modified Bitmap.flipInplace >>= (=== Set.toAscList (symmetricDifference model spanned))+    modified Bitmap.addRange >>= (=== Set.toAscList (Set.union model spanned))+    modified Bitmap.removeRange >>= (=== Set.toAscList (Set.difference model spanned))++offset :: Property+offset =+  Property.generated $ do+    values <- forAll genValues+    shift <- forAll . Gen.int64 $ Range.constant (-300000) 300000+    bitmap <- fromValues values+    let moved = map ((+ shift) . fromIntegral) . Set.toAscList $ Set.fromList values+    owned (Bitmap.addOffset bitmap shift) >>= toList >>= (=== [fromIntegral value | value <- moved, value >= 0, value < 2 ^ (32 :: Int)])++bulkQueries :: Property+bulkQueries =+  Property.generated $ do+    values <- forAll genValues+    probes <- forAll $ Gen.list (Range.linear 0 50) genValue+    start <- forAll . Gen.int $ Range.linear 0 5000+    limit <- forAll . Gen.int $ Range.linear 0 5000+    bitmap <- fromValues values+    let+      model = Set.fromList values+      sortedProbes = Set.toAscList $ Set.fromList probes+      window = take limit . drop start $ Set.toAscList model+      ranks = withArrayLen sortedProbes $ \count begin ->+        allocaArray count $ \out -> do+          Bitmap.rankMany bitmap begin (advancePtr begin count) out+          peekArray count out+      windowed =+        allocaArray (length window) $ \out -> do+          done <- Bitmap.rangeUint32Array bitmap (fromIntegral start) (fromIntegral limit) out+          found <- peekArray (length window) out+          pure (done, found)+    ranks `returns` map (modelRank model) sortedProbes+    windowed `returns` (True, window)++copyOnWrite :: Property+copyOnWrite =+  Property.example $ do+    bitmap <- fromValues [1, 2, 3, 70000]+    evalIO $ Bitmap.setCopyOnWrite bitmap True+    copied <- owned $ Bitmap.copy bitmap+    Bitmap.containsShared bitmap `returns` True+    evalIO $ Bitmap.add copied 4+    toList bitmap >>= (=== [1, 2, 3, 70000])+    toList copied >>= (=== [1, 2, 3, 4, 70000])+    Bitmap.unshareAll bitmap `returns` True+    Bitmap.containsShared bitmap `returns` False+    evalIO $ Bitmap.setCopyOnWrite copied False+    Bitmap.containsShared copied `returns` False++serialized :: (Bitmap -> IO CSize) -> (Bitmap -> Ptr CChar -> IO a) -> Bitmap -> Test (Ptr CChar, CSize, a)+serialized measure write bitmap = do+  size <- evalIO $ measure bitmap+  buffer <- Property.array $ fromIntegral size+  written <- evalIO $ write bitmap buffer+  pure (buffer, size, written)++nativeRoundTrip :: Property+nativeRoundTrip =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    (buffer, size, written) <- serialized Bitmap.sizeInBytes Bitmap.serialize bitmap+    written === size+    owned (Bitmap.deserializeBounded (castPtr buffer) written) >>= toList >>= (=== Set.toAscList (Set.fromList values))+    owned (fromMaybe (Bitmap nullPtr) <$> Bitmap.deserializeValidated (castPtr buffer) written) >>= toList >>= (=== Set.toAscList (Set.fromList values))+    owned (Bitmap.deserialize $ castPtr buffer) >>= toList >>= (=== Set.toAscList (Set.fromList values))++portableRoundTrip :: Property+portableRoundTrip =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    (buffer, size, written) <- serialized Bitmap.portableSizeInBytes Bitmap.portableSerialize bitmap+    written === size+    Bitmap.portableDeserializeSize buffer written `returns` written+    owned (Bitmap.portableDeserializeBounded buffer written) >>= toList >>= (=== Set.toAscList (Set.fromList values))+    owned (fromMaybe (Bitmap nullPtr) <$> Bitmap.portableDeserializeValidated buffer written) >>= toList >>= (=== Set.toAscList (Set.fromList values))+    owned (Bitmap.portableDeserialize buffer) >>= toList >>= (=== Set.toAscList (Set.fromList values))++portableFrozenRoundTrip :: Property+portableFrozenRoundTrip =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    (buffer, _, _) <- serialized Bitmap.portableSizeInBytes Bitmap.portableSerialize bitmap+    owned (Bitmap.portableDeserializeFrozen buffer) >>= toList >>= (=== Set.toAscList (Set.fromList values))++frozenRoundTrip :: Property+frozenRoundTrip =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    size <- evalIO $ Bitmap.frozenSizeInBytes bitmap+    buffer <- Property.alignedBytes (fromIntegral size) 32+    evalIO $ Bitmap.frozenSerialize bitmap buffer+    owned (viewed $ Bitmap.frozenView buffer size) >>= toList >>= (=== Set.toAscList (Set.fromList values))+    adopt (viewed $ Bitmap.frozenView (buffer `plusPtr` 1) size) >>= (=== Bitmap nullPtr)+  where+    viewed = fmap $ Bitmap . unConstPtr++truncatedPortable :: Property+truncatedPortable =+  Property.example $ do+    bitmap <- fromValues [1, 2, 3, 100000]+    (buffer, _, written) <- serialized Bitmap.portableSizeInBytes Bitmap.portableSerialize bitmap+    Bitmap.portableDeserializeSize buffer (written - 1) `returns` 0+    adopt (Bitmap.portableDeserializeBounded buffer $ written - 1) >>= (=== Bitmap nullPtr)+    Bitmap.portableDeserializeValidated buffer (written - 1) `returns` Nothing++corruptedPortable :: Property+corruptedPortable =+  Property.example $ do+    bitmap <- fromValues [1, 2, 3]+    (buffer, _, written) <- serialized Bitmap.portableSizeInBytes Bitmap.portableSerialize bitmap+    written === 22+    let payload = castPtr buffer `plusPtr` 16 :: Ptr Word8+    peekArray 6 payload `returns` [1, 0, 2, 0, 3, 0]+    evalIO $ pokeArray payload [2, 0, 1, 0, 3, 0]+    Bitmap.portableDeserializeValidated buffer written `returns` Nothing+    loaded <- owned $ Bitmap.portableDeserializeBounded buffer written+    reasonPtr <- Property.array 1+    evalIO . poke reasonPtr $ ConstPtr nullPtr+    Bitmap.internalValidate loaded reasonPtr `returns` False+    Bitmap.withPortableFrozen buffer written elements `returns` Nothing+    ConstPtr reason <- evalIO $ peek reasonPtr+    assert $ reason /= nullPtr+    message <- evalIO $ peekCString reason+    annotate message+    assert . not $ null message++knownStatistics :: Property+knownStatistics =+  Property.example $ do+    bitmap <- owned $ Bitmap.fromRange 131072 (131072 + 65536) 2+    evalIO $ do+      forM_ [0, 2 .. 18] $ Bitmap.add bitmap+      Bitmap.addRange bitmap 65536 $ 65536 + 10000+    Bitmap.runOptimize bitmap `returns` True+    statistics <- evalIO . alloca $ \ptr -> Bitmap.statistics bitmap ptr >> peek ptr+    Bitmap.getStatistics bitmap `returns` statistics+    statistics+      === Statistics+        { containers = 3+        , arrayContainers = 1+        , runContainers = 1+        , bitsetContainers = 1+        , valuesArrayContainers = 10+        , valuesRunContainers = 10000+        , valuesBitsetContainers = 32768+        , bytesArrayContainers = statistics.bytesArrayContainers+        , bytesRunContainers = statistics.bytesRunContainers+        , bytesBitsetContainers = 8192+        , maxValue = 131072 + 65534+        , minValue = 0+        , cardinality = 10 + 10000 + 32768+        }+    assert $ statistics.bytesArrayContainers >= 20+    assert $ statistics.bytesRunContainers >= 4++statisticsStorable :: Property+statisticsStorable =+  Property.example $ do+    let statistics = Statistics 1 2 3 4 5 6 7 8 9 10 11 12 13+    ptr <- Property.array 1+    evalIO $ poke ptr statistics+    peek ptr `returns` statistics++callerMemory :: Property+callerMemory =+  Property.example $ do+    memory <- Property.alignedBytes Bitmap.size Bitmap.alignment+    let bitmap = Bitmap memory+    _ <- allocate (Bitmap.initCleared bitmap) . const $ Bitmap.clear bitmap+    evalIO . forM_ [5, 70000, 3] $ Bitmap.add bitmap+    toList bitmap >>= (=== [3, 5, 70000])+    evalIO $ Bitmap.clear bitmap+    Bitmap.initWithCapacity bitmap 4 `returns` True+    evalIO $ Bitmap.add bitmap 9+    toList bitmap >>= (=== [9])++bulkContext :: Property+bulkContext =+  Property.example $ do+    bitmap <- owned Bitmap.create+    memory <- Property.alignedBytes BulkContext.size BulkContext.alignment+    let+      context = BulkContext memory+      values = [1, 2, 3, 70000, 70001, 4000000]+      reset = evalIO $ fillBytes memory 0 BulkContext.size+    reset+    evalIO . forM_ values $ Bitmap.addBulk bitmap context+    reset+    forM (values <> [4, 70002]) (Bitmap.containsBulk bitmap context) `returns` (map (const True) values <> [False, False])+    toList bitmap >>= (=== values)++iterateAll :: Property+iterateAll =+  Property.example $ do+    bitmap <- fromValues [9, 1, 70000, 5]+    (seen, callback) <- collect Bitmap.wrapCallback $ const True+    Bitmap.iterate bitmap callback nullPtr `returns` True+    reverse <$> readIORef seen `returns` [1, 5, 9, 70000]++iterateStop :: Property+iterateStop =+  Property.example $ do+    bitmap <- fromValues [1 .. 10]+    (seen, callback) <- collect Bitmap.wrapCallback (< 3)+    Bitmap.iterate bitmap callback nullPtr `returns` False+    reverse <$> readIORef seen `returns` [1, 2, 3]++iterate64HighBits :: Property+iterate64HighBits =+  Property.example $ do+    bitmap <- fromValues [2, 70000]+    (seen, callback) <- collect Bitmap.wrapCallback64 $ const True+    Bitmap.iterate64 bitmap callback (7 * 2 ^ (32 :: Int)) nullPtr `returns` True+    reverse <$> readIORef seen `returns` [7 * 2 ^ (32 :: Int) + 2, 7 * 2 ^ (32 :: Int) + 70000]++bitsets :: Property+bitsets =+  Property.example $ do+    bitmap <- fromValues [0, 63, 64, 1000]+    (_, bitset) <- allocate Bitset.create Bitset.free+    assert $ bitset /= Bitset nullPtr+    Bitmap.toBitset bitmap bitset `returns` True+    Bitset.count bitset `returns` 4+    forM [0, 1, 63, 64, 1000, 1001] (Bitset.get bitset) `returns` [True, False, True, True, True, False]+    output <- Property.array 2+    evalIO $ Bitmap.wordRangeToBitset bitmap 0 2 output+    peekArray 2 output `returns` [2 ^ (63 :: Int) + 1, 1 :: Word64]++finalizer :: Property+finalizer =+  Property.example $ do+    total <- evalIO $ do+      Bitmap ptr <- withArrayLen [1, 2, 3] $ \count values -> Bitmap.ofPtr (fromIntegral count) values+      foreignPtr <- newForeignPtr Bitmap.finalizerFree ptr+      cardinality <- withForeignPtr foreignPtr $ Bitmap.getCardinality . Bitmap+      finalizeForeignPtr foreignPtr+      pure cardinality+    total === 3++scoped :: Property+scoped =+  Property.example $ do+    Bitmap.with (withArrayLen [1, 5, 9] $ \count values -> Bitmap.ofPtr (fromIntegral count) values) elements `returns` [1, 5, 9]+    outcome <- evalIO . try $ Bitmap.with (pure $ Bitmap nullPtr) elements+    outcome === Left (NullHandle "Data.Bitmap.Roaring.FFI.Bitmap.with")++portableFrozenScope :: Property+portableFrozenScope =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    (buffer, _, written) <- serialized Bitmap.portableSizeInBytes Bitmap.portableSerialize bitmap+    Bitmap.withPortableFrozen buffer written elements `returns` Just (Set.toAscList $ Set.fromList values)+    Bitmap.withPortableFrozen buffer (written - 1) elements `returns` Nothing++portableFrozenOffsets :: Property+portableFrozenOffsets =+  Property.example $ do+    buffer <- Property.array 20+    let+      viewedAt :: [Word8] -> IO (Maybe [Word32])+      viewedAt containerOffset = do+        pokeArray buffer $ [0x3A, 0x30, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0] <> containerOffset <> [1, 0, 2, 0]+        Bitmap.withPortableFrozen (castPtr buffer) 20 elements+    viewedAt [16, 0, 0, 0] `returns` Just [1, 2]+    Bitmap.portableDeserializeSize (castPtr buffer) 20 `returns` 20+    viewedAt [0, 0, 0, 0x40] `returns` Nothing+    viewedAt [17, 0, 0, 0] `returns` Nothing++frozenViewScope :: Property+frozenViewScope =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    size <- evalIO $ Bitmap.frozenSizeInBytes bitmap+    buffer <- Property.alignedBytes (fromIntegral size) 32+    evalIO $ Bitmap.frozenSerialize bitmap buffer+    Bitmap.withFrozenView buffer size elements `returns` Just (Set.toAscList $ Set.fromList values)+    Bitmap.withFrozenView (buffer `plusPtr` 1) size elements `returns` Nothing++callbackHelpers :: Property+callbackHelpers =+  Property.example $ do+    bitmap <- fromValues [1 .. 10]+    visiting (Bitmap.iterateWith bitmap) (const $ pure True) `returns` (Right True, [1 .. 10])+    visiting (Bitmap.iterateWith bitmap) (pure . (< 3)) `returns` (Right False, [1, 2, 3])+    visiting (Bitmap.iterateWith bitmap) stopAtThree `returns` (Left $ userError "stop", [1, 2, 3])+    visiting (Bitmap.iterateWith bitmap) lazilyStopAtThree `returns` (Left $ userError "lazy", [1, 2, 3])+    visiting (Bitmap.iterate64With bitmap highBits) (const $ pure True) `returns` (Right True, map (+ highBits) [1 .. 10])+  where+    stopAtThree value = if value == 3 then ioError $ userError "stop" else pure True+    lazilyStopAtThree value = pure $ value /= 3 || throw (userError "lazy")+    highBits = 7 * 2 ^ (32 :: Int)++selfAliasing :: Property+selfAliasing =+  Property.generated $ do+    values <- forAll genValues+    let+      expected = Set.toAscList $ Set.fromList values+      applied :: (Bitmap -> Bitmap -> IO ()) -> Test [Word32]+      applied operation = do+        bitmap <- fromValues values+        evalIO $ operation bitmap bitmap+        toList bitmap+    applied Bitmap.orInplace >>= (=== expected)+    applied lazyOr >>= (=== expected)+    applied Bitmap.xorInplace >>= (=== [])+    applied Bitmap.andnotInplace >>= (=== [])+    applied Bitmap.lazyXorInplace >>= (=== [])+    applied Bitmap.andInplace >>= (=== expected)+    bitmap <- fromValues values+    Bitmap.overwrite bitmap bitmap `returns` True+    toList bitmap >>= (=== expected)+  where+    lazyOr target other = Bitmap.lazyOrInplace target other True++scopedBulkContext :: Property+scopedBulkContext =+  Property.example $ do+    bitmap <- owned Bitmap.create+    evalIO . BulkContext.with $ \context -> forM_ values $ Bitmap.addBulk bitmap context+    BulkContext.with (\context -> forM (values <> [4]) $ Bitmap.containsBulk bitmap context) `returns` (map (const True) values <> [False])+    toList bitmap >>= (=== values)+  where+    values = [1, 2, 3, 70000, 4000000]
+ test/Test/Bitmap/Iterator.hs view
@@ -0,0 +1,181 @@+{-# LANGUAGE OverloadedRecordDot #-}++module Test.Bitmap.Iterator+  ( tests+  , readAll+  , modelRanges+  ) where++import Control.Monad (forM)+import Data.Bitmap.Roaring.FFI.Bitmap (Bitmap)+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator (Iterator (..), Range (..))+import Data.Bitmap.Roaring.FFI.Bitmap.Iterator qualified as Iterator+import Data.Set qualified as Set+import Data.Word (Word32)+import Foreign.Marshal.Array (allocaArray, peekArray)+import Foreign.Ptr (Ptr, nullPtr)+import Foreign.Storable (Storable, peek, poke)+import Hedgehog (Property, evalIO, forAll)+import Hedgehog.Gen qualified as Gen+import Hedgehog.Range qualified as Range+import Test.Bitmap (fromValues, genValues)+import Test.Property (Test, returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Bitmap.Iterator"+    [ testProperty "walks forward and backward" walks+    , testProperty "reads values in chunks" chunks+    , testProperty "reads maximal ranges" ranges+    , testProperty "moves to the first value at or above a probe" moves+    , testProperty "skips forward and backward" skips+    , testProperty "heap iterators copy independently" heap+    , testProperty "with and withLast scope an iterator" scoped+    , testProperty "range poke and peek agree" rangeStorable+    ]++placed :: (Bitmap -> Iterator -> IO ()) -> Bitmap -> Test Iterator+placed position bitmap = do+  iterator <- Iterator <$> Property.alignedBytes Iterator.size Iterator.alignment+  evalIO $ position bitmap iterator+  pure iterator++created :: IO Iterator -> Test Iterator+created = Property.owned (Iterator nullPtr) Iterator.free++walk :: (Iterator -> IO Bool) -> Iterator -> IO [Word32]+walk step iterator = do+  present <- Iterator.hasValue iterator+  if present then do+    value <- Iterator.currentValue iterator+    _ <- step iterator+    (value :) <$> walk step iterator+  else+    pure []++current :: Iterator -> IO (Maybe Word32)+current iterator = do+  present <- Iterator.hasValue iterator+  if present then Just <$> Iterator.currentValue iterator else pure Nothing++walks :: Property+walks =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    let expected = Set.toAscList $ Set.fromList values+    forward <- placed Iterator.init bitmap+    walk Iterator.advance forward `returns` expected+    backward <- placed Iterator.initLast bitmap+    walk Iterator.previous backward `returns` reverse expected++chunks :: Property+chunks =+  Property.generated $ do+    values <- forAll genValues+    chunk <- forAll . Gen.int $ Range.linear 1 50+    bitmap <- fromValues values+    let expected = Set.toAscList $ Set.fromList values+    forward <- placed Iterator.init bitmap+    readAll Iterator.read chunk forward `returns` expected+    backward <- placed Iterator.initLast bitmap+    readAll Iterator.readBackward chunk backward `returns` reverse expected++readAll :: (Storable value, Integral count) => (iterator -> Ptr value -> count -> IO count) -> Int -> iterator -> IO [value]+readAll readNext chunk iterator =+  allocaArray chunk $ \buffer -> do+    let loop = do+          count <- fromIntegral <$> readNext iterator buffer (fromIntegral chunk)+          found <- peekArray count buffer+          if count < chunk then pure found else (found <>) <$> loop+    loop++ranges :: Property+ranges =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    let expected = modelRanges . Set.toAscList $ Set.fromList values+    forward <- placed Iterator.init bitmap+    readRangesWith Iterator.readRanges forward `returns` expected+    backward <- placed Iterator.initLast bitmap+    readRangesWith Iterator.readPrevRanges backward `returns` reverse expected+  where+    readRangesWith readNext iterator =+      allocaArray 3 $ \buffer -> do+        let loop = do+              count <- fromIntegral <$> readNext iterator buffer 3+              found <- map (\range -> (range.min, range.max)) <$> peekArray count buffer+              if count < 3 then pure found else (found <>) <$> loop+        loop++modelRanges :: (Eq a, Num a) => [a] -> [(a, a)]+modelRanges = foldr extend []+  where+    extend value ((low, high) : rest)+      | value + 1 == low = (value, high) : rest+    extend value grouped = (value, value) : grouped++moves :: Property+moves =+  Property.generated $ do+    values <- forAll genValues+    probes <- forAll . Gen.list (Range.linear 0 50) $ Gen.word32 Range.constantBounded+    bitmap <- fromValues values+    iterator <- Iterator <$> Property.alignedBytes Iterator.size Iterator.alignment+    let+      model = Set.fromList values+      probe value = do+        Iterator.init bitmap iterator+        _ <- Iterator.moveEqualOrLarger iterator value+        current iterator+    forM probes probe `returns` map (`Set.lookupGE` model) probes++skips :: Property+skips =+  Property.generated $ do+    values <- forAll genValues+    count <- forAll . Gen.int $ Range.linear 0 5000+    bitmap <- fromValues values+    let+      model = Set.fromList values+      total = Set.size model+      skippable = fromIntegral $ min count total+    forward <- placed Iterator.init bitmap+    Iterator.skip forward (fromIntegral count) `returns` skippable+    current forward `returns` if count < total then Just (Set.elemAt count model) else Nothing+    backward <- placed Iterator.initLast bitmap+    Iterator.skipBackward backward (fromIntegral count) `returns` skippable+    current backward `returns` if count < total then Just (Set.elemAt (total - 1 - count) model) else Nothing++heap :: Property+heap =+  Property.example $ do+    bitmap <- fromValues [3, 70000, 9]+    original <- created $ Iterator.create bitmap+    Iterator.advance original `returns` True+    copied <- created $ Iterator.copy original+    Iterator.advance original `returns` True+    Iterator.currentValue original `returns` 70000+    walk Iterator.advance copied `returns` [9, 70000]++rangeStorable :: Property+rangeStorable =+  Property.example $ do+    let range = Range 5 99+    ptr <- Property.array 1+    evalIO $ poke ptr range+    peek ptr `returns` range++scoped :: Property+scoped =+  Property.generated $ do+    values <- forAll genValues+    bitmap <- fromValues values+    let expected = Set.toAscList $ Set.fromList values+    Iterator.with bitmap (walk Iterator.advance) `returns` expected+    Iterator.withLast bitmap (walk Iterator.previous) `returns` reverse expected
+ test/Test/Bitmap64.hs view
@@ -0,0 +1,417 @@+{-# LANGUAGE OverloadedRecordDot #-}++module Test.Bitmap64+  ( tests+  , genValues64+  , fromValues64+  ) where++import Control.Exception (try)+import Control.Monad (forM, forM_)+import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Bitmap64 (Bitmap64 (..))+import Data.Bitmap.Roaring.FFI.Bitmap64 qualified as Bitmap64+import Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext (BulkContext (..))+import Data.Bitmap.Roaring.FFI.Bitmap64.BulkContext qualified as BulkContext+import Data.Bitmap.Roaring.FFI.Bitmap64.Statistics (Statistics (..))+import Data.Bitmap.Roaring.FFI.NullHandle (NullHandle (..))+import Data.IORef (readIORef)+import Data.Maybe (fromMaybe)+import Data.Set qualified as Set+import Data.Word (Word64)+import Foreign.C.Types (CChar, CSize)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Marshal.Array (allocaArray, peekArray, withArrayLen)+import Foreign.Marshal.Utils (fillBytes)+import Foreign.Ptr (Ptr, nullPtr, plusPtr)+import Foreign.Storable (peek, poke)+import Hedgehog (Gen, Property, assert, evalIO, forAll, (===))+import Hedgehog.Gen qualified as Gen+import Hedgehog.Range qualified as Range+import Test.Bitmap (collect, fromValues, modelRank, symmetricDifference, visiting)+import Test.Property (Test, returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Bitmap64"+    [ testProperty "bool results in both polarities" boolResults+    , testGroup+        "model"+        [ testProperty "contents and queries" contentsAndQueries+        , testProperty "set algebra" setAlgebra+        , testProperty "in-place set algebra" inplaceSetAlgebra+        , testProperty "ranges" ranges+        , testProperty "offsets" offsets+        , testProperty "move from a 32-bit bitmap" moveFromRoaring32+        ]+    , testGroup+        "serialization"+        [ testProperty "portable round-trip" portableRoundTrip+        , testProperty "portable frozen round-trip" portableFrozenRoundTrip+        , testProperty "frozen view round-trip" frozenRoundTrip+        , testProperty "bounded portable load rejects a truncated buffer" truncatedPortable+        ]+    , testGroup+        "layout"+        [ testProperty "statistics of known containers" knownStatistics+        , testProperty "statistics poke and peek agree" statisticsStorable+        , testProperty "bulk context in caller memory" bulkContext+        ]+    , testProperty "iterate visits every value" iterateAll+    , testGroup+        "helpers"+        [ testProperty "with frees the bitmap and rejects NULL" scoped+        , testProperty "withPortableFrozen views valid bytes in place" portableFrozenScope+        , testProperty "withFrozenView views aligned bytes in place" frozenViewScope+        , testProperty "iterateWith visits, stops and rethrows" callbackHelpers+        , testProperty "in-place operations with the same bitmap twice" selfAliasing+        , testProperty "BulkContext.with" scopedBulkContext+        ]+    ]++genValues64 :: Gen [Word64]+genValues64 = concat <$> Gen.list (Range.linear 0 60) (Gen.choice [pure <$> genValue64, genRun])+  where+    genRun = do+      start <- genValue64+      len <- Gen.int $ Range.linear 0 3000+      pure . take len $ enumFrom start++genValue64 :: Gen Word64+genValue64 =+  Gen.choice+    [ Gen.word64 $ Range.constant 0 200000+    , Gen.word64 Range.constantBounded+    , (+) . (* 2 ^ (32 :: Int)) <$> Gen.word64 (Range.constant 1 3) <*> Gen.word64 (Range.constant 0 200000)+    , Gen.word64 $ Range.constant (maxBound - 200000) maxBound+    ]++adopt64 :: IO Bitmap64 -> Test Bitmap64+adopt64 = Property.adopt (Bitmap64 nullPtr) Bitmap64.free++owned64 :: IO Bitmap64 -> Test Bitmap64+owned64 = Property.owned (Bitmap64 nullPtr) Bitmap64.free++fromValues64 :: [Word64] -> Test Bitmap64+fromValues64 values = owned64 . withArrayLen values $ \count ptr -> Bitmap64.ofPtr (fromIntegral count) ptr++elements64 :: Bitmap64 -> IO [Word64]+elements64 bitmap = do+  count <- fromIntegral <$> Bitmap64.getCardinality bitmap+  allocaArray count $ \ptr -> do+    Bitmap64.toUint64Array bitmap ptr+    peekArray count ptr++toList64 :: Bitmap64 -> Test [Word64]+toList64 = evalIO . elements64++boolResults :: Property+boolResults =+  Property.example $ do+    bitmap <- owned64 Bitmap64.create+    Bitmap64.addChecked bitmap (2 ^ (40 :: Int)) `returns` True+    Bitmap64.addChecked bitmap (2 ^ (40 :: Int)) `returns` False+    Bitmap64.removeChecked bitmap (2 ^ (40 :: Int)) `returns` True+    Bitmap64.removeChecked bitmap (2 ^ (40 :: Int)) `returns` False+    Bitmap64.isEmpty bitmap `returns` True++contentsAndQueries :: Property+contentsAndQueries =+  Property.generated $ do+    values <- forAll genValues64+    probes <- forAll $ Gen.list (Range.linear 0 50) genValue64+    bitmap <- fromValues64 values+    let model = Set.fromList values+    toList64 bitmap >>= (=== Set.toAscList model)+    Bitmap64.getCardinality bitmap `returns` fromIntegral (Set.size model)+    Bitmap64.isEmpty bitmap `returns` Set.null model+    Bitmap64.minimum bitmap `returns` maybe maxBound fst (Set.minView model)+    Bitmap64.maximum bitmap `returns` maybe 0 fst (Set.maxView model)+    Bitmap64.internalValidate bitmap nullPtr `returns` True+    forM (values <> probes) (probe bitmap) `returns` map (modelProbe model) (values <> probes)+    forM [0 .. fromIntegral (Set.size model)] (select bitmap) `returns` (map Just (Set.toAscList model) <> [Nothing])+  where+    probe bitmap value = (,,) <$> Bitmap64.contains bitmap value <*> Bitmap64.rank bitmap value <*> index bitmap value+    index bitmap value =+      alloca $ \out -> do+        found <- Bitmap64.getIndex bitmap value out+        if found then Just <$> peek out else pure Nothing+    modelProbe model value =+      ( Set.member value model+      , modelRank model value+      , fromIntegral <$> Set.lookupIndex value model+      )+    select bitmap rank =+      alloca $ \out -> do+        found <- Bitmap64.select bitmap rank out+        if found then Just <$> peek out else pure Nothing++setAlgebra :: Property+setAlgebra =+  Property.generated $ do+    leftValues <- forAll genValues64+    rightValues <- forAll genValues64+    left <- fromValues64 leftValues+    right <- fromValues64 rightValues+    let+      leftModel = Set.fromList leftValues+      rightModel = Set.fromList rightValues+      intersection = Set.intersection leftModel rightModel+      union = Set.union leftModel rightModel+      difference = Set.difference leftModel rightModel+      symmetric = symmetricDifference leftModel rightModel+    owned64 (Bitmap64.and left right) >>= toList64 >>= (=== Set.toAscList intersection)+    owned64 (Bitmap64.or left right) >>= toList64 >>= (=== Set.toAscList union)+    owned64 (Bitmap64.andnot left right) >>= toList64 >>= (=== Set.toAscList difference)+    owned64 (Bitmap64.xor left right) >>= toList64 >>= (=== Set.toAscList symmetric)+    Bitmap64.andCardinality left right `returns` fromIntegral (Set.size intersection)+    Bitmap64.orCardinality left right `returns` fromIntegral (Set.size union)+    Bitmap64.andnotCardinality left right `returns` fromIntegral (Set.size difference)+    Bitmap64.xorCardinality left right `returns` fromIntegral (Set.size symmetric)+    Bitmap64.intersect left right `returns` not (Set.disjoint leftModel rightModel)+    Bitmap64.isSubset left right `returns` Set.isSubsetOf leftModel rightModel+    Bitmap64.isStrictSubset left right `returns` Set.isProperSubsetOf leftModel rightModel+    Bitmap64.equals left right `returns` (leftModel == rightModel)++inplaceSetAlgebra :: Property+inplaceSetAlgebra =+  Property.generated $ do+    leftValues <- forAll genValues64+    rightValues <- forAll genValues64+    right <- fromValues64 rightValues+    let+      leftModel = Set.fromList leftValues+      rightModel = Set.fromList rightValues+      applied :: (Bitmap64 -> Bitmap64 -> IO ()) -> Test [Word64]+      applied operation = do+        left <- fromValues64 leftValues+        evalIO $ operation left right+        toList64 left+    applied Bitmap64.andInplace >>= (=== Set.toAscList (Set.intersection leftModel rightModel))+    applied Bitmap64.orInplace >>= (=== Set.toAscList (Set.union leftModel rightModel))+    applied Bitmap64.andnotInplace >>= (=== Set.toAscList (Set.difference leftModel rightModel))+    applied Bitmap64.xorInplace >>= (=== Set.toAscList (symmetricDifference leftModel rightModel))+    applied Bitmap64.overwrite >>= (=== Set.toAscList rightModel)++ranges :: Property+ranges =+  Property.generated $ do+    values <- forAll genValues64+    start <- forAll genValue64+    len <- forAll . Gen.word64 $ Range.linear 0 100000+    bitmap <- fromValues64 values+    let+      model = Set.fromList values+      end = if maxBound - start < len then maxBound else start + len+      spanned = Set.fromList . take (fromIntegral $ end - start) $ enumFrom start+      inRange = Set.intersection model spanned+      modified :: (Bitmap64 -> Word64 -> Word64 -> IO ()) -> Test [Word64]+      modified operation = do+        copy <- fromValues64 values+        evalIO $ operation copy start end+        toList64 copy+    Bitmap64.rangeCardinality bitmap start end `returns` fromIntegral (Set.size inRange)+    Bitmap64.containsRange bitmap start end `returns` (inRange == spanned)+    Bitmap64.intersectWithRange bitmap start end `returns` not (Set.null inRange)+    owned64 (Bitmap64.flip bitmap start end) >>= toList64 >>= (=== Set.toAscList (symmetricDifference model spanned))+    modified Bitmap64.flipInplace >>= (=== Set.toAscList (symmetricDifference model spanned))+    modified Bitmap64.addRange >>= (=== Set.toAscList (Set.union model spanned))+    modified Bitmap64.removeRange >>= (=== Set.toAscList (Set.difference model spanned))++offsets :: Property+offsets =+  Property.generated $ do+    values <- forAll genValues64+    shift <- forAll genValue64+    bitmap <- fromValues64 values+    let+      ascending = Set.toAscList $ Set.fromList values+      down = [value - shift | value <- ascending, value >= shift]+    owned64 (Bitmap64.addOffset bitmap shift) >>= toList64 >>= (=== [value + shift | value <- ascending, value <= maxBound - shift])+    owned64 (Bitmap64.subOffset bitmap shift) >>= toList64 >>= (=== down)+    owned64 (Bitmap64.addOffsetSigned bitmap False shift) >>= toList64 >>= (=== down)++moveFromRoaring32 :: Property+moveFromRoaring32 =+  Property.generated $ do+    values <- forAll . Gen.list (Range.linear 0 200) $ Gen.word32 Range.constantBounded+    bitmap <- fromValues values+    owned64 (Bitmap64.moveFromRoaring32 bitmap) >>= toList64 >>= (=== map fromIntegral (Set.toAscList $ Set.fromList values))+    Bitmap.isEmpty bitmap `returns` True++serialized :: (Bitmap64 -> IO CSize) -> (Bitmap64 -> Ptr CChar -> IO CSize) -> Bitmap64 -> Test (Ptr CChar, CSize, CSize)+serialized measure write bitmap = do+  size <- evalIO $ measure bitmap+  buffer <- Property.alignedBytes (fromIntegral size) 64+  written <- evalIO $ write bitmap buffer+  pure (buffer, size, written)++portableRoundTrip :: Property+portableRoundTrip =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    (buffer, size, written) <- serialized Bitmap64.portableSizeInBytes Bitmap64.portableSerialize bitmap+    written === size+    Bitmap64.portableDeserializeSize buffer written `returns` written+    owned64 (Bitmap64.portableDeserializeBounded buffer written) >>= toList64 >>= (=== Set.toAscList (Set.fromList values))+    owned64 (fromMaybe (Bitmap64 nullPtr) <$> Bitmap64.portableDeserializeValidated buffer written) >>= toList64 >>= (=== Set.toAscList (Set.fromList values))++portableFrozenRoundTrip :: Property+portableFrozenRoundTrip =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    (buffer, _, written) <- serialized Bitmap64.portableSizeInBytes Bitmap64.portableSerialize bitmap+    owned64 (Bitmap64.portableDeserializeFrozen buffer written) >>= toList64 >>= (=== Set.toAscList (Set.fromList values))++frozenRoundTrip :: Property+frozenRoundTrip =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    _ <- evalIO $ Bitmap64.shrinkToFit bitmap+    (buffer, size, written) <- serialized Bitmap64.frozenSizeInBytes Bitmap64.frozenSerialize bitmap+    written === size+    owned64 (Bitmap64.frozenView buffer written) >>= toList64 >>= (=== Set.toAscList (Set.fromList values))+    adopt64 (Bitmap64.frozenView (buffer `plusPtr` 1) written) >>= (=== Bitmap64 nullPtr)++truncatedPortable :: Property+truncatedPortable =+  Property.example $ do+    bitmap <- fromValues64 [1, 2, 3, 2 ^ (40 :: Int)]+    (buffer, _, written) <- serialized Bitmap64.portableSizeInBytes Bitmap64.portableSerialize bitmap+    Bitmap64.portableDeserializeSize buffer (written - 1) `returns` 0+    adopt64 (Bitmap64.portableDeserializeBounded buffer $ written - 1) >>= (=== Bitmap64 nullPtr)+    Bitmap64.portableDeserializeValidated buffer (written - 1) `returns` Nothing++knownStatistics :: Property+knownStatistics =+  Property.example $ do+    bitmap <- owned64 $ Bitmap64.fromRange 131072 (131072 + 65536) 2+    evalIO $ do+      forM_ [0, 2 .. 18] $ Bitmap64.add bitmap+      Bitmap64.addRange bitmap 65536 $ 65536 + 10000+    Bitmap64.runOptimize bitmap `returns` True+    statistics <- evalIO . alloca $ \ptr -> Bitmap64.statistics bitmap ptr >> peek ptr+    Bitmap64.getStatistics bitmap `returns` statistics+    statistics+      === Statistics+        { containers = 3+        , arrayContainers = 1+        , runContainers = 1+        , bitsetContainers = 1+        , valuesArrayContainers = 10+        , valuesRunContainers = 10000+        , valuesBitsetContainers = 32768+        , bytesArrayContainers = statistics.bytesArrayContainers+        , bytesRunContainers = statistics.bytesRunContainers+        , bytesBitsetContainers = 8192+        , maxValue = 131072 + 65534+        , minValue = 0+        , cardinality = 10 + 10000 + 32768+        }+    assert $ statistics.bytesArrayContainers >= 20+    assert $ statistics.bytesRunContainers >= 4++statisticsStorable :: Property+statisticsStorable =+  Property.example $ do+    let statistics = Statistics 1 2 3 4 5 6 7 8 9 10 11 12 13+    ptr <- Property.array 1+    evalIO $ poke ptr statistics+    peek ptr `returns` statistics++bulkContext :: Property+bulkContext =+  Property.example $ do+    bitmap <- owned64 Bitmap64.create+    memory <- Property.alignedBytes BulkContext.size BulkContext.alignment+    let+      context = BulkContext memory+      values = [1, 2, 3, 2 ^ (40 :: Int), 2 ^ (40 :: Int) + 1, maxBound]+      reset = evalIO $ fillBytes memory 0 BulkContext.size+    reset+    evalIO . forM_ values $ Bitmap64.addBulk bitmap context+    reset+    forM (values <> [4, 2 ^ (40 :: Int) + 2]) (Bitmap64.containsBulk bitmap context) `returns` (map (const True) values <> [False, False])+    reset+    evalIO $ do+      Bitmap64.removeBulk bitmap context 2+      Bitmap64.removeBulk bitmap context 3+    toList64 bitmap >>= (=== [1, 2 ^ (40 :: Int), 2 ^ (40 :: Int) + 1, maxBound])++iterateAll :: Property+iterateAll =+  Property.example $ do+    bitmap <- fromValues64 [9, 1, 2 ^ (40 :: Int), 5]+    (seen, callback) <- collect Bitmap.wrapCallback64 $ const True+    Bitmap64.iterate bitmap callback nullPtr `returns` True+    reverse <$> readIORef seen `returns` [1, 5, 9, 2 ^ (40 :: Int)]++scoped :: Property+scoped =+  Property.example $ do+    Bitmap64.with (withArrayLen [1, 5, 2 ^ (40 :: Int)] $ \count values -> Bitmap64.ofPtr (fromIntegral count) values) elements64 `returns` [1, 5, 2 ^ (40 :: Int)]+    outcome <- evalIO . try $ Bitmap64.with (pure $ Bitmap64 nullPtr) elements64+    outcome === Left (NullHandle "Data.Bitmap.Roaring.FFI.Bitmap64.with")++portableFrozenScope :: Property+portableFrozenScope =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    (buffer, _, written) <- serialized Bitmap64.portableSizeInBytes Bitmap64.portableSerialize bitmap+    Bitmap64.withPortableFrozen buffer written elements64 `returns` Just (Set.toAscList $ Set.fromList values)+    Bitmap64.withPortableFrozen buffer (written - 1) elements64 `returns` Nothing++frozenViewScope :: Property+frozenViewScope =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    _ <- evalIO $ Bitmap64.shrinkToFit bitmap+    (buffer, _, written) <- serialized Bitmap64.frozenSizeInBytes Bitmap64.frozenSerialize bitmap+    Bitmap64.withFrozenView buffer written elements64 `returns` Just (Set.toAscList $ Set.fromList values)+    Bitmap64.withFrozenView (buffer `plusPtr` 1) written elements64 `returns` Nothing++callbackHelpers :: Property+callbackHelpers =+  Property.example $ do+    bitmap <- fromValues64 [1 .. 10]+    visiting (Bitmap64.iterateWith bitmap) (const $ pure True) `returns` (Right True, [1 .. 10])+    visiting (Bitmap64.iterateWith bitmap) (pure . (< 3)) `returns` (Right False, [1, 2, 3])+    visiting (Bitmap64.iterateWith bitmap) stopAtThree `returns` (Left $ userError "stop", [1, 2, 3])+  where+    stopAtThree value = if value == 3 then ioError $ userError "stop" else pure True++selfAliasing :: Property+selfAliasing =+  Property.generated $ do+    values <- forAll genValues64+    let+      expected = Set.toAscList $ Set.fromList values+      applied :: (Bitmap64 -> Bitmap64 -> IO ()) -> Test [Word64]+      applied operation = do+        bitmap <- fromValues64 values+        evalIO $ operation bitmap bitmap+        toList64 bitmap+    applied Bitmap64.orInplace >>= (=== expected)+    applied Bitmap64.xorInplace >>= (=== [])+    applied Bitmap64.andnotInplace >>= (=== [])+    applied Bitmap64.andInplace >>= (=== expected)+    applied Bitmap64.overwrite >>= (=== expected)++scopedBulkContext :: Property+scopedBulkContext =+  Property.example $ do+    bitmap <- owned64 Bitmap64.create+    evalIO . BulkContext.with $ \context -> forM_ values $ Bitmap64.addBulk bitmap context+    BulkContext.with (\context -> forM (values <> [4]) $ Bitmap64.containsBulk bitmap context) `returns` (map (const True) values <> [False])+    toList64 bitmap >>= (=== values)+  where+    values = [1, 2, 3, 2 ^ (40 :: Int), maxBound]
+ test/Test/Bitmap64/Iterator.hs view
@@ -0,0 +1,136 @@+{-# LANGUAGE OverloadedRecordDot #-}++module Test.Bitmap64.Iterator (tests) where++import Control.Monad (forM)+import Data.Bitmap.Roaring.FFI.Bitmap64.Iterator (Iterator (..), Range (..))+import Data.Bitmap.Roaring.FFI.Bitmap64.Iterator qualified as Iterator+import Data.Set qualified as Set+import Data.Word (Word64)+import Foreign.Marshal.Array (allocaArray, peekArray)+import Foreign.Ptr (nullPtr)+import Foreign.Storable (peek, poke)+import Hedgehog (Property, evalIO, forAll)+import Hedgehog.Gen qualified as Gen+import Hedgehog.Range qualified as Range+import Test.Bitmap.Iterator (modelRanges, readAll)+import Test.Bitmap64 (fromValues64, genValues64)+import Test.Property (Test, returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Bitmap64.Iterator"+    [ testProperty "walks forward and backward" walks+    , testProperty "reads values in chunks" chunks+    , testProperty "reads maximal ranges" ranges+    , testProperty "moves to the first value at or above a probe" moves+    , testProperty "copies and reinitialises" copies+    , testProperty "with and withLast scope an iterator" scoped+    , testProperty "range poke and peek agree" rangeStorable+    ]++created :: IO Iterator -> Test Iterator+created = Property.owned (Iterator nullPtr) Iterator.free++walk :: (Iterator -> IO Bool) -> Iterator -> IO [Word64]+walk step iterator = do+  present <- Iterator.hasValue iterator+  if present then do+    value <- Iterator.value iterator+    _ <- step iterator+    (value :) <$> walk step iterator+  else+    pure []++walks :: Property+walks =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    let expected = Set.toAscList $ Set.fromList values+    forward <- created $ Iterator.create bitmap+    walk Iterator.advance forward `returns` expected+    backward <- created $ Iterator.createLast bitmap+    walk Iterator.previous backward `returns` reverse expected++chunks :: Property+chunks =+  Property.generated $ do+    values <- forAll genValues64+    chunk <- forAll . Gen.int $ Range.linear 1 50+    bitmap <- fromValues64 values+    let expected = Set.toAscList $ Set.fromList values+    forward <- created $ Iterator.create bitmap+    readAll Iterator.read chunk forward `returns` expected+    backward <- created $ Iterator.createLast bitmap+    readAll Iterator.readBackward chunk backward `returns` reverse expected++ranges :: Property+ranges =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    let expected = modelRanges . Set.toAscList $ Set.fromList values+    forward <- created $ Iterator.create bitmap+    readRangesWith Iterator.readRanges forward `returns` expected+    backward <- created $ Iterator.createLast bitmap+    readRangesWith Iterator.readPrevRanges backward `returns` reverse expected+  where+    readRangesWith readNext iterator =+      allocaArray 3 $ \buffer -> do+        let loop = do+              count <- fromIntegral <$> readNext iterator buffer 3+              found <- map (\range -> (range.min, range.max)) <$> peekArray count buffer+              if count < 3 then pure found else (found <>) <$> loop+        loop++moves :: Property+moves =+  Property.generated $ do+    values <- forAll genValues64+    probes <- forAll . Gen.list (Range.linear 0 50) $ Gen.word64 Range.constantBounded+    bitmap <- fromValues64 values+    iterator <- created $ Iterator.create bitmap+    let+      model = Set.fromList values+      probe value = do+        Iterator.reinit bitmap iterator+        present <- Iterator.moveEqualOrLarger iterator value+        if present then Just <$> Iterator.value iterator else pure Nothing+    forM probes probe `returns` map (`Set.lookupGE` model) probes++copies :: Property+copies =+  Property.example $ do+    bitmap <- fromValues64 [3, 2 ^ (40 :: Int), 9]+    original <- created $ Iterator.create bitmap+    Iterator.advance original `returns` True+    copied <- created $ Iterator.copy original+    Iterator.advance original `returns` True+    Iterator.value original `returns` 2 ^ (40 :: Int)+    walk Iterator.advance copied `returns` [9, 2 ^ (40 :: Int)]+    evalIO $ Iterator.reinit bitmap copied+    Iterator.value copied `returns` 3+    evalIO $ Iterator.reinitLast bitmap copied+    walk Iterator.previous copied `returns` [2 ^ (40 :: Int), 9, 3]++rangeStorable :: Property+rangeStorable =+  Property.example $ do+    let range = Range 5 (2 ^ (40 :: Int))+    ptr <- Property.array 1+    evalIO $ poke ptr range+    peek ptr `returns` range++scoped :: Property+scoped =+  Property.generated $ do+    values <- forAll genValues64+    bitmap <- fromValues64 values+    let expected = Set.toAscList $ Set.fromList values+    Iterator.with bitmap (walk Iterator.advance) `returns` expected+    Iterator.withLast bitmap (walk Iterator.previous) `returns` reverse expected
+ test/Test/Bitset.hs view
@@ -0,0 +1,202 @@+module Test.Bitset (tests) where++import Control.Exception (try)+import Control.Monad (forM, forM_)+import Data.Bitmap.Roaring.FFI.Bitset (Bitset (..))+import Data.Bitmap.Roaring.FFI.Bitset qualified as Bitset+import Data.Bitmap.Roaring.FFI.NullHandle (NullHandle (..))+import Data.IORef (readIORef)+import Data.Set qualified as Set+import Data.Word (Word64)+import Foreign.C.Types (CSize)+import Foreign.Marshal.Alloc (alloca)+import Foreign.Marshal.Array (allocaArray, peekArray)+import Foreign.Ptr (nullPtr)+import Foreign.Storable (peek, poke)+import Hedgehog (Gen, Property, assert, evalIO, forAll, (===))+import Hedgehog.Gen qualified as Gen+import Hedgehog.Range qualified as Range+import Test.Bitmap (collect, visiting)+import Test.Property (Test, returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Bitset"+    [ testProperty "set bits and queries" queries+    , testProperty "set algebra" setAlgebra+    , testProperty "shifts" shifts+    , testProperty "buffer fields" fields+    , testProperty "forEach stops when the callback says so" forEachStop+    , testProperty "resize, trim and fill" sizing+    , testProperty "with frees the bitset and rejects NULL" scoped+    , testProperty "forEachWith visits, stops and rethrows" callbackHelpers+    , testProperty "in-place operations with the same bitset twice" selfAliasing+    ]++genBits :: CSize -> Gen [CSize]+genBits limit = Gen.list (Range.linear 0 200) . fmap fromIntegral . Gen.word64 . Range.constant 0 $ fromIntegral limit++created :: IO Bitset -> Test Bitset+created = Property.owned (Bitset nullPtr) Bitset.free++fromBits :: [CSize] -> Test Bitset+fromBits bits = do+  bitset <- created Bitset.create+  evalIO . forM_ bits $ Bitset.set bitset+  pure bitset++toList :: Bitset -> IO [CSize]+toList bitset =+  allocaArray 5 $ \buffer ->+    alloca $ \start -> do+      poke start 0+      let loop = do+            count <- fromIntegral <$> Bitset.nextSetBits bitset buffer 5 start+            found <- peekArray count buffer+            if count == 0 then+              pure []+            else do+              peek start >>= poke start . (+ 1)+              (found <>) <$> loop+      loop++queries :: Property+queries =+  Property.generated $ do+    bits <- forAll $ genBits 5000+    probes <- forAll $ genBits 6000+    bitset <- fromBits bits+    let+      model = Set.fromList bits+      next probe =+        alloca $ \index -> do+          poke index probe+          present <- Bitset.nextSetBit bitset index+          if present then Just <$> peek index else pure Nothing+    toList bitset `returns` Set.toAscList model+    Bitset.count bitset `returns` fromIntegral (Set.size model)+    Bitset.empty bitset `returns` Set.null model+    Bitset.minimum bitset `returns` maybe maxBound fst (Set.minView model)+    Bitset.maximum bitset `returns` maybe 0 fst (Set.maxView model)+    forM probes (Bitset.get bitset) `returns` map (`Set.member` model) probes+    forM probes next `returns` map (`Set.lookupGE` model) probes++setAlgebra :: Property+setAlgebra =+  Property.generated $ do+    leftBits <- forAll $ genBits 3000+    rightBits <- forAll $ genBits 3000+    left <- fromBits leftBits+    right <- fromBits rightBits+    let+      leftModel = Set.fromList leftBits+      rightModel = Set.fromList rightBits+      union = Set.union leftModel rightModel+      intersection = Set.intersection leftModel rightModel+      difference = Set.difference leftModel rightModel+      symmetric = union `Set.difference` intersection+      applied operation = do+        copy <- fromBits leftBits+        _ <- evalIO $ operation copy right+        evalIO $ toList copy+    Bitset.unionCount left right `returns` fromIntegral (Set.size union)+    Bitset.intersectionCount left right `returns` fromIntegral (Set.size intersection)+    Bitset.differenceCount left right `returns` fromIntegral (Set.size difference)+    Bitset.symmetricDifferenceCount left right `returns` fromIntegral (Set.size symmetric)+    Bitset.disjoint left right `returns` Set.disjoint leftModel rightModel+    Bitset.intersect left right `returns` not (Set.disjoint leftModel rightModel)+    Bitset.containsAll left right `returns` Set.isSubsetOf rightModel leftModel+    applied Bitset.inplaceUnion >>= (=== Set.toAscList union)+    applied Bitset.inplaceIntersection >>= (=== Set.toAscList intersection)+    applied Bitset.inplaceDifference >>= (=== Set.toAscList difference)+    applied Bitset.inplaceSymmetricDifference >>= (=== Set.toAscList symmetric)+    copied <- created $ Bitset.copy left+    toList copied `returns` Set.toAscList leftModel++shifts :: Property+shifts =+  Property.generated $ do+    bits <- forAll $ genBits 3000+    shift <- forAll . fmap fromIntegral . Gen.word64 $ Range.linear 0 200+    let ascending = Set.toAscList $ Set.fromList bits+    left <- fromBits bits+    evalIO $ Bitset.shiftLeft left shift+    toList left `returns` map (+ shift) ascending+    right <- fromBits bits+    evalIO $ Bitset.shiftRight right shift+    toList right `returns` [bit - shift | bit <- ascending, bit >= shift]++fields :: Property+fields =+  Property.example $ do+    bitset <- fromBits [0, 63, 64, 200]+    used <- evalIO $ Bitset.sizeInWords bitset+    Bitset.arraySize bitset `returns` used+    Bitset.sizeInBits bitset `returns` (used * 64)+    Bitset.sizeInBytes bitset `returns` (used * 8)+    capacity <- evalIO $ Bitset.capacity bitset+    assert $ capacity >= used+    take 4 <$> (Bitset.array bitset >>= peekArray (fromIntegral used)) `returns` [2 ^ (63 :: Int) + 1, 1, 0, 2 ^ (200 - 192 :: Int) :: Word64]++forEachStop :: Property+forEachStop =+  Property.example $ do+    bitset <- fromBits [1, 5, 9, 700]+    (seen, callback) <- collect Bitset.wrapCallback (< 9)+    Bitset.forEach bitset callback nullPtr `returns` False+    reverse <$> readIORef seen `returns` [1, 5, 9]++sizing :: Property+sizing =+  Property.example $ do+    bitset <- created $ Bitset.createWithCapacity 128+    Bitset.sizeInWords bitset `returns` 2+    Bitset.count bitset `returns` 0+    evalIO $ Bitset.fill bitset+    Bitset.count bitset `returns` 128+    evalIO $ Bitset.setToValue bitset 3 False+    Bitset.get bitset 3 `returns` False+    Bitset.resize bitset 4 True `returns` True+    Bitset.sizeInWords bitset `returns` 4+    Bitset.count bitset `returns` 127+    Bitset.grow bitset 8 `returns` True+    Bitset.sizeInWords bitset `returns` 8+    evalIO $ Bitset.clear bitset+    Bitset.empty bitset `returns` True+    Bitset.trim bitset `returns` True++scoped :: Property+scoped =+  Property.example $ do+    Bitset.with (Bitset.createWithCapacity 64) (\bitset -> Bitset.set bitset 5 >> Bitset.count bitset) `returns` 1+    outcome <- evalIO . try $ Bitset.with (pure $ Bitset nullPtr) Bitset.count+    outcome === Left (NullHandle "Data.Bitmap.Roaring.FFI.Bitset.with")++callbackHelpers :: Property+callbackHelpers =+  Property.example $ do+    bitset <- fromBits [1 .. 10]+    visiting (Bitset.forEachWith bitset) (const $ pure True) `returns` (Right True, [1 .. 10])+    visiting (Bitset.forEachWith bitset) (pure . (< 3)) `returns` (Right False, [1, 2, 3])+    visiting (Bitset.forEachWith bitset) stopAtThree `returns` (Left $ userError "stop", [1, 2, 3])+  where+    stopAtThree bit = if bit == 3 then ioError $ userError "stop" else pure True++selfAliasing :: Property+selfAliasing =+  Property.generated $ do+    bits <- forAll $ genBits 3000+    let+      expected = Set.toAscList $ Set.fromList bits+      applied operation = do+        bitset <- fromBits bits+        result <- evalIO $ operation bitset bitset+        (,) result <$> evalIO (toList bitset)+    applied Bitset.inplaceUnion >>= (=== (True, expected))+    applied Bitset.inplaceIntersection >>= (=== ((), expected))+    applied Bitset.inplaceDifference >>= (=== ((), []))+    applied Bitset.inplaceSymmetricDifference >>= (=== (True, []))
+ test/Test/Memory.hs view
@@ -0,0 +1,73 @@+module Test.Memory+  ( tests+  , countingHooks+  ) where++import Data.Bitmap.Roaring.FFI.Bitmap qualified as Bitmap+import Data.Bitmap.Roaring.FFI.Memory (Hooks (..))+import Data.Bitmap.Roaring.FFI.Memory qualified as Memory+import Foreign.C.Types (CSize (..))+import Foreign.Marshal.Array (withArrayLen)+import Foreign.Ptr (FunPtr, castPtrToFunPtr, nullPtr, plusPtr)+import Foreign.Storable (peek, poke)+import Hedgehog (Property, assert, evalIO, (===))+import Test.Property (returns)+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++foreign import ccall "&counting_malloc"+  countingMalloc :: FunPtr Memory.Malloc++foreign import ccall "&counting_realloc"+  countingRealloc :: FunPtr Memory.Realloc++foreign import ccall "&counting_calloc"+  countingCalloc :: FunPtr Memory.Calloc++foreign import ccall "&counting_aligned_malloc"+  countingAlignedMalloc :: FunPtr Memory.AlignedMalloc++foreign import ccall "stdlib.h &free"+  libcFree :: FunPtr Memory.Free++foreign import ccall unsafe "counting_allocations"+  allocations :: IO CSize++countingHooks :: Hooks+countingHooks =+  Hooks+    { malloc = countingMalloc+    , realloc = countingRealloc+    , calloc = countingCalloc+    , free = libcFree+    , alignedMalloc = countingAlignedMalloc+    , alignedFree = libcFree+    }++tests :: TestTree+tests =+  testGroup+    "Memory"+    [ testProperty "hooks poke and peek agree" hooksStorable+    , testProperty "allocations go through the installed hooks" counted+    ]++hooksStorable :: Property+hooksStorable =+  Property.example $ do+    let+      pointer offset = castPtrToFunPtr $ nullPtr `plusPtr` offset+      hooks = Hooks (pointer 16) (pointer 32) (pointer 48) (pointer 64) (pointer 80) (pointer 96)+    ptr <- Property.array 1+    evalIO $ poke ptr hooks+    peek ptr `returns` hooks++counted :: Property+counted =+  Property.example $ do+    before <- evalIO allocations+    total <- evalIO . Bitmap.with (withArrayLen [0, 2 .. 200000] $ \count values -> Bitmap.ofPtr (fromIntegral count) values) $ Bitmap.getCardinality+    total === 100001+    after <- evalIO allocations+    assert $ after > before
+ test/Test/Property.hs view
@@ -0,0 +1,58 @@+module Test.Property+  ( Test+  , generated+  , example+  , returns+  , adopt+  , owned+  , array+  , alignedBytes+  ) where++import Control.Monad (unless, when)+import Control.Monad.Morph (hoist)+import Control.Monad.Trans.Resource (ResourceT, allocate, runResourceT)+import Data.Word (Word8)+import Foreign.Marshal.Alloc (free, mallocBytes)+import Foreign.Marshal.Array (mallocArray)+import Foreign.Ptr (Ptr, alignPtr, castPtr)+import Foreign.Storable (Storable)+import GHC.Stack (HasCallStack, withFrozenCallStack)+import Hedgehog (Property, PropertyT, annotate, evalIO, failure, property, withTests, (===))++type Test = PropertyT (ResourceT IO)++generated :: Test () -> Property+generated = property . hoist runResourceT++example :: Test () -> Property+example = withTests 1 . generated++infix 1 `returns`++returns :: (HasCallStack, Eq a, Show a) => IO a -> a -> Test ()+returns action expected =+  withFrozenCallStack $ do+    actual <- evalIO action+    actual === expected++adopt :: (Eq handle) => handle -> (handle -> IO ()) -> IO handle -> Test handle+adopt missing release create = snd <$> allocate create releasePresent+  where+    releasePresent handle = unless (handle == missing) $ release handle++owned :: (Eq handle) => handle -> (handle -> IO ()) -> IO handle -> Test handle+owned missing release create = do+  handle <- adopt missing release create+  when (handle == missing) $ do+    annotate "C returned a NULL handle"+    failure+  pure handle++array :: (Storable a) => Int -> Test (Ptr a)+array count = snd <$> allocate (mallocArray $ max 1 count) free++alignedBytes :: Int -> Int -> Test (Ptr a)+alignedBytes size alignment = do+  (_, bytes) <- allocate (mallocBytes $ size + alignment) free+  pure . castPtr $ alignPtr (bytes :: Ptr Word8) alignment
+ test/Test/Version.hs view
@@ -0,0 +1,20 @@+module Test.Version (tests) where++import Data.Bitmap.Roaring.FFI.Version qualified as Version+import Hedgehog (Property, (===))+import Test.Property qualified as Property+import Test.Tasty (TestTree, testGroup)+import Test.Tasty.Hedgehog (testProperty)++tests :: TestTree+tests =+  testGroup+    "Version"+    [ testProperty "matches the pinned CRoaring release" pinned+    ]++pinned :: Property+pinned =+  Property.example $ do+    Version.version === "5.5.5"+    (Version.major, Version.minor, Version.revision) === (5, 5, 5)
+ test/cbits/counting-allocator.c view
@@ -0,0 +1,27 @@+#include <stdatomic.h>+#include <stdlib.h>++static atomic_size_t allocations;++void *counting_malloc(size_t size) {+    atomic_fetch_add(&allocations, 1);+    return malloc(size);+}++void *counting_realloc(void *block, size_t size) {+    atomic_fetch_add(&allocations, 1);+    return realloc(block, size);+}++void *counting_calloc(size_t count, size_t size) {+    atomic_fetch_add(&allocations, 1);+    return calloc(count, size);+}++void *counting_aligned_malloc(size_t alignment, size_t size) {+    void *block = NULL;+    atomic_fetch_add(&allocations, 1);+    return posix_memalign(&block, alignment, size) == 0 ? block : NULL;+}++size_t counting_allocations(void) { return atomic_load(&allocations); }