packages feed

souffle-haskell 0.2.3 → 1.0.0

raw patch · 60 files changed

+19539/−114 lines, 60 filesdep +containersdep +extradep +megaparsecnew-component:exe:import-souffle-headersPVP ok

version bump matches the API change (PVP)

Dependencies added: containers, extra, megaparsec

API changes (from Hackage documentation)

- Language.Souffle.TH: embedProgram :: String -> Q [Dec]
+ Language.Souffle.Interpreted: [cfgFactDir] :: Config -> Maybe FilePath
+ Language.Souffle.Interpreted: [cfgOutputDir] :: Config -> Maybe FilePath
+ Language.Souffle.Interpreted: souffleStdErr :: forall prog. Program prog => Handle prog -> SouffleM (Maybe Text)
+ Language.Souffle.Interpreted: souffleStdOut :: forall prog. Program prog => Handle prog -> SouffleM (Maybe Text)
- Language.Souffle.Class: writeFiles :: MonadSouffleFileIO m => Handler m prog -> m ()
+ Language.Souffle.Class: writeFiles :: MonadSouffleFileIO m => Handler m prog -> FilePath -> m ()
- Language.Souffle.Internal: printAll :: ForeignPtr Souffle -> IO ()
+ Language.Souffle.Internal: printAll :: ForeignPtr Souffle -> FilePath -> IO ()
- Language.Souffle.Internal.Bindings: printAll :: Ptr Souffle -> IO ()
+ Language.Souffle.Internal.Bindings: printAll :: Ptr Souffle -> CString -> IO ()
- Language.Souffle.Interpreted: Config :: FilePath -> Maybe FilePath -> Config
+ Language.Souffle.Interpreted: Config :: FilePath -> Maybe FilePath -> Maybe FilePath -> Maybe FilePath -> Config

Files

CHANGELOG.md view
@@ -4,7 +4,24 @@ All notable changes to this project (as seen by library users) will be documented in this file. The CHANGELOG is available on [Github](https://github.com/luc-tielen/souffle-haskell.git/CHANGELOG.md). -## [0.2.2] - 2020-05-21+## [1.0.0] - 2020-07-09+### Changed++- Libraries using souffle-haskell are now "self-contained": if a project+  depends on such a library, it will not require to also have Souffle installed.+- souffle-haskell now supports Soufflé version 2.0.0.+- `writeFiles` now takes an extra `FilePath` argument for writing facts to a+  certain directory.++### Deleted++- Language.Souffle.TH module is deleted because it is no longer needed anymore+  due to a change in the generated Souffle code. The generated code can now be+  correctly integrated by adding the files to `cxx-sources`+  in package.yaml / cabal file.+++## [0.2.3] - 2020-05-21 ### Changed  - Optimize performance when marshalling and unmarshalling facts.
README.md view
@@ -45,7 +45,7 @@  ```haskell -- Enable some necessary extensions:-{-# LANGUAGE TemplateHaskell, ScopedTypeVariables, DataKinds, TypeFamilies, DeriveGeneric #-}+{-# LANGUAGE ScopedTypeVariables, DataKinds, TypeFamilies, DeriveGeneric #-}  module Main ( main ) where @@ -53,19 +53,13 @@ import Control.Monad.IO.Class import GHC.Generics import Data.Vector-import qualified Language.Souffle.TH as Souffle import qualified Language.Souffle.Compiled as Souffle --- We only use template haskell for directly embedding the .cpp file into this file.--- If we do not do this, it will link incorrectly due to the way the--- C++ code is generated.-Souffle.embedProgram "/path/to/path.cpp" - -- We define a data type representing our datalog program. data Path = Path --- Facts are represent in Haskell as simple product types,+-- Facts are represented in Haskell as simple product types, -- Numbers map to Int32, symbols to Strings / Text.  data Edge = Edge String String@@ -81,7 +75,7 @@   type ProgramFacts Path = [Edge, Reachable]   programName = const "path" --- By making a data type an instance of Edge, we give Haskell the+-- By making a data type an instance of Fact, we give Haskell the -- necessary information to bind to the datalog fact. instance Souffle.Fact Edge where   factName = const "edge"@@ -138,14 +132,16 @@ Without Nix, you will have to follow the manual install instructions on the [Souffle website](https://souffle-lang.github.io/install). -In your package.yaml / *.cabal file, make sure to add the following options+In your package.yaml or .cabal file, make sure to add the following options (assuming package.yaml here):  ```yaml # ... -cpp-options:+cxx-options:   - -D__EMBEDDED_SOUFFLE__+cxx-sources:+  - /path/to/FILE.cpp  # be sure to change this according to what you need!  # ... ```@@ -153,7 +149,18 @@ This will instruct the Souffle compiler to compile the C++ in such a way that it can be linked with other languages (including Haskell!). +For an example, take a look at the configuration for the+[test suite](https://github.com/luc-tielen/souffle-haskell/blob/master/package.yaml#L68-L80) of this project. +If you run into C++ compilation issues when using stack, this might be because+the `-std=c++17` flag is not being used correctly when compiling souffle-haskell.+To fix this, you can add the following to your `stack.yaml`:++```yaml+ghc-options:+  souffle-haskell: -optcxx-std=c++17+```+ ## Supported modes  Souffle programs can be run in 2 ways. They can either run in **interpreted** mode@@ -177,7 +184,6 @@ 2. You need to call `Souffle.cleanup` after you no longer need the Souffle    functionality. This will clean up the generated CSV fact files located in    a temporary directory.-3. You don't need to import `Language.Souffle.TH` to embed a Datalog program.   #### Interpreter configuration@@ -210,10 +216,7 @@ The main differences with interpreted mode are the following:  1. Compile the Datalog code with `souffle -g`.-2. You need to import `Language.Souffle.TH` to embed a Datalog program-   using `Language.Souffle.TH.embedProgram`, as shown in the-   [motivating example](#motivating-example).-3. Remove `Souffle.cleanup` if it is present in your code, compiled mode+2. Remove `Souffle.cleanup` if it is present in your code, compiled mode    leaves no CSV artifacts.  The [motivating example](#motivating-example) is a complete example for the compiled mode.
cbits/souffle.cpp view
@@ -32,17 +32,18 @@         prog->run();     } -    void souffle_load_all(souffle_t* program, const char* directory) {+    void souffle_load_all(souffle_t* program, const char* input_directory) {         auto prog = reinterpret_cast<souffle::SouffleProgram*>(program);         assert(prog);-        assert(directory);-        prog->loadAll(directory);+        assert(input_directory);+        prog->loadAll(input_directory);     } -    void souffle_print_all(souffle_t* program) {+    void souffle_print_all(souffle_t* program, const char* output_directory) {         auto prog = reinterpret_cast<souffle::SouffleProgram*>(program);         assert(prog);-        prog->printAll();+        assert(output_directory);+        prog->printAll(output_directory);     }      relation_t* souffle_relation(souffle_t* program, const char* relation_name) {
cbits/souffle.h view
@@ -61,7 +61,7 @@      * You need to check if both pointers are non-NULL before passing it to this      * function. Not doing so results in undefined behavior.      */-    void souffle_load_all(souffle_t* program, const char* directory);+    void souffle_load_all(souffle_t* program, const char* input_directory);      /*      * Write out all facts of the program to CSV files@@ -70,7 +70,7 @@      * You need to check if the pointer is non-NULL before passing it to this      * function. Not doing so results in undefined behavior.      */-    void souffle_print_all(souffle_t* program);+    void souffle_print_all(souffle_t* program, const char* output_directory);      /*      * Lookup a relation in the Souffle program.
+ cbits/souffle/BTree.h view
@@ -0,0 +1,2334 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file BTree.h+ *+ * An implementation of a generic B-tree data structure including+ * interfaces for utilizing instances as set or multiset containers.+ *+ ***********************************************************************/++#pragma once++#include "utility/CacheUtil.h"+#include "utility/ContainerUtil.h"+#include "utility/ParallelUtil.h"+#include <algorithm>+#include <cassert>+#include <cstddef>+#include <cstdint>+#include <iostream>+#include <iterator>+#include <string>+#include <tuple>+#include <type_traits>+#include <typeinfo>+#include <vector>++namespace souffle {++namespace detail {++// ---------- comparators --------------++/**+ * A generic comparator implementation as it is used by+ * a b-tree based on types that can be less-than and+ * equality comparable.+ */+template <typename T>+struct comparator {+    /**+     * Compares the values of a and b and returns+     * -1 if a<b, 1 if a>b and 0 otherwise+     */+    int operator()(const T& a, const T& b) const {+        return (a > b) - (a < b);+    }+    bool less(const T& a, const T& b) const {+        return a < b;+    }+    bool equal(const T& a, const T& b) const {+        return a == b;+    }+};++// ---------- search strategies --------------++/**+ * A common base class for search strategies in b-trees.+ */+struct search_strategy {};++/**+ * A linear search strategy for looking up keys in b-tree nodes.+ */+struct linear_search : public search_strategy {+    /**+     * Required user-defined default constructor.+     */+    linear_search() = default;++    /**+     * Obtains an iterator referencing an element equivalent to the+     * given key in the given range. If no such element is present,+     * a reference to the first element not less than the given key+     * is returned.+     */+    template <typename Key, typename Iter, typename Comp>+    inline Iter operator()(const Key& k, Iter a, Iter b, Comp& comp) const {+        return lower_bound(k, a, b, comp);+    }++    /**+     * Obtains a reference to the first element in the given range that+     * is not less than the given key.+     */+    template <typename Key, typename Iter, typename Comp>+    inline Iter lower_bound(const Key& k, Iter a, Iter b, Comp& comp) const {+        auto c = a;+        while (c < b) {+            auto r = comp(*c, k);+            if (r >= 0) {+                return c;+            }+            ++c;+        }+        return b;+    }++    /**+     * Obtains a reference to the first element in the given range that+     * such that the given key is less than the referenced element.+     */+    template <typename Key, typename Iter, typename Comp>+    inline Iter upper_bound(const Key& k, Iter a, Iter b, Comp& comp) const {+        auto c = a;+        while (c < b) {+            if (comp(*c, k) > 0) {+                return c;+            }+            ++c;+        }+        return b;+    }+};++/**+ * A binary search strategy for looking up keys in b-tree nodes.+ */+struct binary_search : public search_strategy {+    /**+     * Required user-defined default constructor.+     */+    binary_search() = default;++    /**+     * Obtains an iterator pointing to some element within the given+     * range that is equal to the given key, if available. If multiple+     * elements are equal to the given key, an undefined instance will+     * be obtained (no guaranteed lower or upper boundary).  If no such+     * element is present, a reference to the first element not less than+     * the given key will be returned.+     */+    template <typename Key, typename Iter, typename Comp>+    Iter operator()(const Key& k, Iter a, Iter b, Comp& comp) const {+        Iter c;+        auto count = b - a;+        while (count > 0) {+            auto step = count >> 1;+            c = a + step;+            auto r = comp(*c, k);+            if (r == 0) {+                return c;+            }+            if (r < 0) {+                a = ++c;+                count -= step + 1;+            } else {+                count = step;+            }+        }+        return a;+    }++    /**+     * Obtains a reference to the first element in the given range that+     * is not less than the given key.+     */+    template <typename Key, typename Iter, typename Comp>+    Iter lower_bound(const Key& k, Iter a, Iter b, Comp& comp) const {+        Iter c;+        auto count = b - a;+        while (count > 0) {+            auto step = count >> 1;+            c = a + step;+            if (comp(*c, k) < 0) {+                a = ++c;+                count -= step + 1;+            } else {+                count = step;+            }+        }+        return a;+    }++    /**+     * Obtains a reference to the first element in the given range that+     * such that the given key is less than the referenced element.+     */+    template <typename Key, typename Iter, typename Comp>+    Iter upper_bound(const Key& k, Iter a, Iter b, Comp& comp) const {+        Iter c;+        auto count = b - a;+        while (count > 0) {+            auto step = count >> 1;+            c = a + step;+            if (comp(k, *c) >= 0) {+                a = ++c;+                count -= step + 1;+            } else {+                count = step;+            }+        }+        return a;+    }+};++// ---------- search strategies selection --------------++/**+ * A template-meta class to select search strategies for b-trees+ * depending on the key type.+ */+template <typename S>+struct strategy_selection {+    using type = S;+};++struct linear : public strategy_selection<linear_search> {};+struct binary : public strategy_selection<binary_search> {};++// by default every key utilizes binary search+template <typename Key>+struct default_strategy : public binary {};++template <>+struct default_strategy<int> : public linear {};++template <typename... Ts>+struct default_strategy<std::tuple<Ts...>> : public linear {};++/**+ * The default non-updater+ */+template <typename T>+struct updater {+    void update(T& /* old_t */, const T& /* new_t */) {}+};++/**+ * The actual implementation of a b-tree data structure.+ *+ * @tparam Key             .. the element type to be stored in this tree+ * @tparam Comparator     .. a class defining an order on the stored elements+ * @tparam Allocator     .. utilized for allocating memory for required nodes+ * @tparam blockSize    .. determines the number of bytes/block utilized by leaf nodes+ * @tparam SearchStrategy .. enables switching between linear, binary or any other search strategy+ * @tparam isSet        .. true = set, false = multiset+ */+template <typename Key, typename Comparator,+        typename Allocator,  // is ignored so far - TODO: add support+        unsigned blockSize, typename SearchStrategy, bool isSet, typename WeakComparator = Comparator,+        typename Updater = detail::updater<Key>>+class btree {+public:+    class iterator;+    using const_iterator = iterator;++    using key_type = Key;+    using element_type = Key;+    using chunk = range<iterator>;++protected:+    /* ------------- static utilities ----------------- */++    const static SearchStrategy search;++    /* ---------- comparison utilities ---------------- */++    mutable Comparator comp;++    bool less(const Key& a, const Key& b) const {+        return comp.less(a, b);+    }++    bool equal(const Key& a, const Key& b) const {+        return comp.equal(a, b);+    }++    mutable WeakComparator weak_comp;++    bool weak_less(const Key& a, const Key& b) const {+        return weak_comp.less(a, b);+    }++    bool weak_equal(const Key& a, const Key& b) const {+        return weak_comp.equal(a, b);+    }++    /* -------------- updater utilities ------------- */++    mutable Updater upd;+    void update(Key& old_k, const Key& new_k) {+        upd.update(old_k, new_k);+    }++    /* -------------- the node type ----------------- */++    using size_type = std::size_t;+    using field_index_type = uint8_t;+    using lock_type = OptimisticReadWriteLock;++    struct node;++    /**+     * The base type of all node types containing essential+     * book-keeping information.+     */+    struct base {+#ifdef IS_PARALLEL++        // the parent node+        node* volatile parent;++        // a lock for synchronizing parallel operations on this node+        lock_type lock;++        // the number of keys in this node+        volatile size_type numElements;++        // the position in the parent node+        volatile field_index_type position;+#else+        // the parent node+        node* parent;++        // the number of keys in this node+        size_type numElements;++        // the position in the parent node+        field_index_type position;+#endif++        // a flag indicating whether this is a inner node or not+        const bool inner;++        /**+         * A simple constructor for nodes+         */+        base(bool inner) : parent(nullptr), numElements(0), position(0), inner(inner) {}++        bool isLeaf() const {+            return !inner;+        }++        bool isInner() const {+            return inner;+        }++        node* getParent() const {+            return parent;+        }++        field_index_type getPositionInParent() const {+            return position;+        }++        size_type getNumElements() const {+            return numElements;+        }+    };++    struct inner_node;++    /**+     * The actual, generic node implementation covering the operations+     * for both, inner and leaf nodes.+     */+    struct node : public base {+        /**+         * The number of keys/node desired by the user.+         */+        static constexpr size_t desiredNumKeys =+                ((blockSize > sizeof(base)) ? blockSize - sizeof(base) : 0) / sizeof(Key);++        /**+         * The actual number of keys/node corrected by functional requirements.+         */+        static constexpr size_t maxKeys = (desiredNumKeys > 3) ? desiredNumKeys : 3;++        // the keys stored in this node+        Key keys[maxKeys];++        // a simple constructor+        node(bool inner) : base(inner) {}++        /**+         * A deep-copy operation creating a clone of this node.+         */+        node* clone() const {+            // create a clone of this node+            node* res = (this->isInner()) ? static_cast<node*>(new inner_node())+                                          : static_cast<node*>(new leaf_node());++            // copy basic fields+            res->position = this->position;+            res->numElements = this->numElements;++            for (size_type i = 0; i < this->numElements; ++i) {+                res->keys[i] = this->keys[i];+            }++            // if this is a leaf we are done+            if (this->isLeaf()) {+                return res;+            }++            // copy child nodes recursively+            auto* ires = (inner_node*)res;+            for (size_type i = 0; i <= this->numElements; ++i) {+                ires->children[i] = this->getChild(i)->clone();+                ires->children[i]->parent = res;+            }++            // that's it+            return res;+        }++        /**+         * A utility function providing a reference to this node as+         * an inner node.+         */+        inner_node& asInnerNode() {+            assert(this->inner && "Invalid cast!");+            return *static_cast<inner_node*>(this);+        }++        /**+         * A utility function providing a reference to this node as+         * a const inner node.+         */+        const inner_node& asInnerNode() const {+            assert(this->inner && "Invalid cast!");+            return *static_cast<const inner_node*>(this);+        }++        /**+         * Computes the number of nested levels of the tree rooted+         * by this node.+         */+        size_type getDepth() const {+            if (this->isLeaf()) {+                return 1;+            }+            return getChild(0)->getDepth() + 1;+        }++        /**+         * Counts the number of nodes contained in the sub-tree rooted+         * by this node.+         */+        size_type countNodes() const {+            if (this->isLeaf()) {+                return 1;+            }+            size_type sum = 1;+            for (unsigned i = 0; i <= this->numElements; ++i) {+                sum += getChild(i)->countNodes();+            }+            return sum;+        }++        /**+         * Counts the number of entries contained in the sub-tree rooted+         * by this node.+         */+        size_type countEntries() const {+            if (this->isLeaf()) {+                return this->numElements;+            }+            size_type sum = this->numElements;+            for (unsigned i = 0; i <= this->numElements; ++i) {+                sum += getChild(i)->countEntries();+            }+            return sum;+        }++        /**+         * Determines the amount of memory used by the sub-tree rooted+         * by this node.+         */+        size_type getMemoryUsage() const {+            if (this->isLeaf()) {+                return sizeof(leaf_node);+            }+            size_type res = sizeof(inner_node);+            for (unsigned i = 0; i <= this->numElements; ++i) {+                res += getChild(i)->getMemoryUsage();+            }+            return res;+        }++        /**+         * Obtains a pointer to the array of child-pointers+         * of this node -- if it is an inner node.+         */+        node** getChildren() {+            return asInnerNode().children;+        }++        /**+         * Obtains a pointer to the array of const child-pointers+         * of this node -- if it is an inner node.+         */+        node* const* getChildren() const {+            return asInnerNode().children;+        }++        /**+         * Obtains a reference to the child of the given index.+         */+        node* getChild(size_type s) const {+            return asInnerNode().children[s];+        }++        /**+         * Checks whether this node is empty -- can happen due to biased insertion.+         */+        bool isEmpty() const {+            return this->numElements == 0;+        }++        /**+         * Checks whether this node is full.+         */+        bool isFull() const {+            return this->numElements == maxKeys;+        }++        /**+         * Obtains the point at which full nodes should be split.+         * Conventional b-trees always split in half. However, in cases+         * where in-order insertions are frequent, a split assigning+         * larger portions to the right fragment provide higher performance+         * and a better node-filling rate.+         */+        int getSplitPoint(int /*unused*/) {+            return std::min(3 * maxKeys / 4, maxKeys - 2);+        }++        /**+         * Splits this node.+         *+         * @param root .. a pointer to the root-pointer of the enclosing b-tree+         *                 (might have to be updated if the root-node needs to be split)+         * @param idx  .. the position of the insert causing the split+         */+#ifdef IS_PARALLEL+        void split(node** root, lock_type& root_lock, int idx, std::vector<node*>& locked_nodes) {+            assert(this->lock.is_write_locked());+            assert(!this->parent || this->parent->lock.is_write_locked());+            assert((this->parent != nullptr) || root_lock.is_write_locked());+            assert(this->isLeaf() || souffle::contains(locked_nodes, this));+            assert(!this->parent || souffle::contains(locked_nodes, const_cast<node*>(this->parent)));+#else+        void split(node** root, lock_type& root_lock, int idx) {+#endif+            assert(this->numElements == maxKeys);++            // get middle element+            int split_point = getSplitPoint(idx);++            // create a new sibling node+            node* sibling = (this->inner) ? static_cast<node*>(new inner_node())+                                          : static_cast<node*>(new leaf_node());++#ifdef IS_PARALLEL+            // lock sibling+            sibling->lock.start_write();+            locked_nodes.push_back(sibling);+#endif++            // move data over to the new node+            for (unsigned i = split_point + 1, j = 0; i < maxKeys; ++i, ++j) {+                sibling->keys[j] = keys[i];+            }++            // move child pointers+            if (this->inner) {+                // move pointers to sibling+                auto* other = static_cast<inner_node*>(sibling);+                for (unsigned i = split_point + 1, j = 0; i <= maxKeys; ++i, ++j) {+                    other->children[j] = getChildren()[i];+                    other->children[j]->parent = other;+                    other->children[j]->position = j;+                }+            }++            // update number of elements+            this->numElements = split_point;+            sibling->numElements = maxKeys - split_point - 1;++            // update parent+#ifdef IS_PARALLEL+            grow_parent(root, root_lock, sibling, locked_nodes);+#else+            grow_parent(root, root_lock, sibling);+#endif+        }++        /**+         * Moves keys from this node to one of its siblings or splits+         * this node to make some space for the insertion of an element at+         * position idx.+         *+         * Returns the number of elements moved to the left side, 0 in case+         * of a split. The number of moved elements will be <= the given idx.+         *+         * @param root .. the root node of the b-tree being part of+         * @param idx  .. the position of the insert triggering this operation+         */+        // TODO: remove root_lock ... no longer needed+#ifdef IS_PARALLEL+        int rebalance_or_split(node** root, lock_type& root_lock, int idx, std::vector<node*>& locked_nodes) {+            assert(this->lock.is_write_locked());+            assert(!this->parent || this->parent->lock.is_write_locked());+            assert((this->parent != nullptr) || root_lock.is_write_locked());+            assert(this->isLeaf() || souffle::contains(locked_nodes, this));+            assert(!this->parent || souffle::contains(locked_nodes, const_cast<node*>(this->parent)));+#else+        int rebalance_or_split(node** root, lock_type& root_lock, int idx) {+#endif++            // this node is full ... and needs some space+            assert(this->numElements == maxKeys);++            // get snap-shot of parent+            auto parent = this->parent;+            auto pos = this->position;++            // Option A) re-balance data+            if (parent && pos > 0) {+                node* left = parent->getChild(pos - 1);++#ifdef IS_PARALLEL+                // lock access to left sibling+                if (!left->lock.try_start_write()) {+                    // left node is currently updated => skip balancing and split+                    split(root, root_lock, idx, locked_nodes);+                    return 0;+                }+#endif++                // compute number of elements to be movable to left+                //    space available in left vs. insertion index+                size_type num = std::min<int>(maxKeys - left->numElements, idx);++                // if there are elements to move ..+                if (num > 0) {+                    Key* splitter = &(parent->keys[this->position - 1]);++                    // .. move keys to left node+                    left->keys[left->numElements] = *splitter;+                    for (size_type i = 0; i < num - 1; ++i) {+                        left->keys[left->numElements + 1 + i] = keys[i];+                    }+                    *splitter = keys[num - 1];++                    // shift keys in this node to the left+                    for (size_type i = 0; i < this->numElements - num; ++i) {+                        keys[i] = keys[i + num];+                    }++                    // .. and children if necessary+                    if (this->isInner()) {+                        auto* ileft = static_cast<inner_node*>(left);+                        auto* iright = static_cast<inner_node*>(this);++                        // move children+                        for (size_type i = 0; i < num; ++i) {+                            ileft->children[left->numElements + i + 1] = iright->children[i];+                        }++                        // update moved children+                        for (size_type i = 0; i < num; ++i) {+                            iright->children[i]->parent = ileft;+                            iright->children[i]->position = left->numElements + i + 1;+                        }++                        // shift child-pointer to the left+                        for (size_type i = 0; i < this->numElements - num + 1; ++i) {+                            iright->children[i] = iright->children[i + num];+                        }++                        // update position of children+                        for (size_type i = 0; i < this->numElements - num + 1; ++i) {+                            iright->children[i]->position = i;+                        }+                    }++                    // update node sizes+                    left->numElements += num;+                    this->numElements -= num;++#ifdef IS_PARALLEL+                    left->lock.end_write();+#endif++                    // done+                    return num;+                }++#ifdef IS_PARALLEL+                left->lock.abort_write();+#endif+            }++            // Option B) split node+#ifdef IS_PARALLEL+            split(root, root_lock, idx, locked_nodes);+#else+            split(root, root_lock, idx);+#endif+            return 0;  // = no re-balancing+        }++    private:+        /**+         * Inserts a new sibling into the parent of this node utilizing+         * the last key of this node as a separation key. (for internal+         * use only)+         *+         * @param root .. a pointer to the root-pointer of the containing tree+         * @param sibling .. the new right-sibling to be add to the parent node+         */+#ifdef IS_PARALLEL+        void grow_parent(node** root, lock_type& root_lock, node* sibling, std::vector<node*>& locked_nodes) {+            assert(this->lock.is_write_locked());+            assert(!this->parent || this->parent->lock.is_write_locked());+            assert((this->parent != nullptr) || root_lock.is_write_locked());+            assert(this->isLeaf() || souffle::contains(locked_nodes, this));+            assert(!this->parent || souffle::contains(locked_nodes, const_cast<node*>(this->parent)));+#else+        void grow_parent(node** root, lock_type& root_lock, node* sibling) {+#endif++            if (this->parent == nullptr) {+                assert(*root == this);++                // create a new root node+                auto* new_root = new inner_node();+                new_root->numElements = 1;+                new_root->keys[0] = keys[this->numElements];++                new_root->children[0] = this;+                new_root->children[1] = sibling;++                // link this and the sibling node to new root+                this->parent = new_root;+                sibling->parent = new_root;+                sibling->position = 1;++                // switch root node+                *root = new_root;++            } else {+                // insert new element in parent element+                auto parent = this->parent;+                auto pos = this->position;++#ifdef IS_PARALLEL+                parent->insert_inner(+                        root, root_lock, pos, this, keys[this->numElements], sibling, locked_nodes);+#else+                parent->insert_inner(root, root_lock, pos, this, keys[this->numElements], sibling);+#endif+            }+        }++        /**+         * Inserts a new element into an inner node (for internal use only).+         *+         * @param root .. a pointer to the root-pointer of the containing tree+         * @param pos  .. the position to insert the new key+         * @param key  .. the key to insert+         * @param newNode .. the new right-child of the inserted key+         */+#ifdef IS_PARALLEL+        void insert_inner(node** root, lock_type& root_lock, unsigned pos, node* predecessor, const Key& key,+                node* newNode, std::vector<node*>& locked_nodes) {+            assert(this->lock.is_write_locked());+            assert(souffle::contains(locked_nodes, this));+#else+        void insert_inner(node** root, lock_type& root_lock, unsigned pos, node* predecessor, const Key& key,+                node* newNode) {+#endif++            // check capacity+            if (this->numElements >= maxKeys) {+#ifdef IS_PARALLEL+                assert(!this->parent || this->parent->lock.is_write_locked());+                assert((this->parent) || root_lock.is_write_locked());+                assert(!this->parent || souffle::contains(locked_nodes, const_cast<node*>(this->parent)));+#endif++                // split this node+#ifdef IS_PARALLEL+                pos -= rebalance_or_split(root, root_lock, pos, locked_nodes);+#else+                pos -= rebalance_or_split(root, root_lock, pos);+#endif++                // complete insertion within new sibling if necessary+                if (pos > this->numElements) {+                    // correct position+                    pos = pos - this->numElements - 1;++                    // get new sibling+                    auto other = this->parent->getChild(this->position + 1);++#ifdef IS_PARALLEL+                    // make sure other side is write locked+                    assert(other->lock.is_write_locked());+                    assert(souffle::contains(locked_nodes, other));++                    // search for new position (since other may have been altered in the meanwhile)+                    size_type i = 0;+                    for (; i <= other->numElements; ++i) {+                        if (other->getChild(i) == predecessor) {+                            break;+                        }+                    }++                    pos = (i > other->numElements) ? 0 : i;+                    other->insert_inner(root, root_lock, pos, predecessor, key, newNode, locked_nodes);+#else+                    other->insert_inner(root, root_lock, pos, predecessor, key, newNode);+#endif+                    return;+                }+            }++            // move bigger keys one forward+            for (int i = this->numElements - 1; i >= (int)pos; --i) {+                keys[i + 1] = keys[i];+                getChildren()[i + 2] = getChildren()[i + 1];+                ++getChildren()[i + 2]->position;+            }++            // ensure proper position+            assert(getChild(pos) == predecessor);++            // insert new element+            keys[pos] = key;+            getChildren()[pos + 1] = newNode;+            newNode->parent = this;+            newNode->position = pos + 1;+            ++this->numElements;+        }++    public:+        /**+         * Prints a textual representation of this tree to the given output stream.+         * This feature is mainly intended for debugging and tuning purposes.+         *+         * @see btree::printTree+         */+        void printTree(std::ostream& out, const std::string& prefix) const {+            // print the header+            out << prefix << "@" << this << "[" << ((int)(this->position)) << "] - "+                << (this->inner ? "i" : "") << "node : " << this->numElements << "/" << maxKeys << " [";++            // print the keys+            for (unsigned i = 0; i < this->numElements; i++) {+                out << keys[i];+                if (i != this->numElements - 1) {+                    out << ",";+                }+            }+            out << "]";++            // print references to children+            if (this->inner) {+                out << " - [";+                for (unsigned i = 0; i <= this->numElements; i++) {+                    out << getChildren()[i];+                    if (i != this->numElements) {+                        out << ",";+                    }+                }+                out << "]";+            }++#ifdef IS_PARALLEL+            // print the lock state+            if (this->lock.is_write_locked()) {+                std::cout << " locked";+            }+#endif++            out << "\n";++            // print the children recursively+            if (this->inner) {+                for (unsigned i = 0; i < this->numElements + 1; ++i) {+                    static_cast<const inner_node*>(this)->children[i]->printTree(out, prefix + "    ");+                }+            }+        }++        /**+         * A function decomposing the sub-tree rooted by this node into approximately equally+         * sized chunks. To minimize computational overhead, no strict load balance nor limit+         * on the number of actual chunks is given.+         *+         * @see btree::getChunks()+         *+         * @param res   .. the list of chunks to be extended+         * @param num   .. the number of chunks to be produced+         * @param begin .. the iterator to start the first chunk with+         * @param end   .. the iterator to end the last chunk with+         * @return the handed in list of chunks extended by generated chunks+         */+        std::vector<chunk>& collectChunks(+                std::vector<chunk>& res, size_type num, const iterator& begin, const iterator& end) const {+            assert(num > 0);++            // special case: this node is empty+            if (isEmpty()) {+                if (begin != end) {+                    res.push_back(chunk(begin, end));+                }+                return res;+            }++            // special case: a single chunk is requested+            if (num == 1) {+                res.push_back(chunk(begin, end));+                return res;+            }++            // cut-off+            if (this->isLeaf() || num < (this->numElements + 1)) {+                auto step = this->numElements / num;+                if (step == 0) {+                    step = 1;+                }++                size_type i = 0;++                // the first chunk starts at the begin+                res.push_back(chunk(begin, iterator(this, step - 1)));++                // split up the main part+                for (i = step - 1; i < this->numElements - step; i += step) {+                    res.push_back(chunk(iterator(this, i), iterator(this, i + step)));+                }++                // the last chunk runs to the end+                res.push_back(chunk(iterator(this, i), end));++                // done+                return res;+            }++            // else: collect chunks of sub-set elements++            auto part = num / (this->numElements + 1);+            assert(part > 0);+            getChild(0)->collectChunks(res, part, begin, iterator(this, 0));+            for (size_type i = 1; i < this->numElements; i++) {+                getChild(i)->collectChunks(res, part, iterator(this, i - 1), iterator(this, i));+            }+            getChild(this->numElements)+                    ->collectChunks(res, num - (part * this->numElements),+                            iterator(this, this->numElements - 1), end);++            // done+            return res;+        }++        /**+         * A function to verify the consistency of this node.+         *+         * @param root ... a reference to the root of the enclosing tree.+         * @return true if valid, false otherwise+         */+        template <typename Comp>+        bool check(Comp& comp, const node* root) const {+            bool valid = true;++            // check fill-state+            if (this->numElements > maxKeys) {+                std::cout << "Node with " << this->numElements << "/" << maxKeys << " encountered!\n";+                valid = false;+            }++            // check root state+            if (root == this) {+                if (this->parent != nullptr) {+                    std::cout << "Root not properly linked!\n";+                    valid = false;+                }+            } else {+                // check parent relation+                if (!this->parent) {+                    std::cout << "Invalid null-parent!\n";+                    valid = false;+                } else {+                    if (this->parent->getChildren()[this->position] != this) {+                        std::cout << "Parent reference invalid!\n";+                        std::cout << "   Node:     " << this << "\n";+                        std::cout << "   Parent:   " << this->parent << "\n";+                        std::cout << "   Position: " << ((int)this->position) << "\n";+                        valid = false;+                    }++                    // check parent key+                    if (valid && this->position != 0 &&+                            !(comp(this->parent->keys[this->position - 1], keys[0]) < ((isSet) ? 0 : 1))) {+                        std::cout << "Left parent key not lower bound!\n";+                        std::cout << "   Node:     " << this << "\n";+                        std::cout << "   Parent:   " << this->parent << "\n";+                        std::cout << "   Position: " << ((int)this->position) << "\n";+                        std::cout << "   Key:   " << (this->parent->keys[this->position]) << "\n";+                        std::cout << "   Lower: " << (keys[0]) << "\n";+                        valid = false;+                    }++                    // check parent key+                    if (valid && this->position != this->parent->numElements &&+                            !(comp(keys[this->numElements - 1], this->parent->keys[this->position]) <+                                    ((isSet) ? 0 : 1))) {+                        std::cout << "Right parent key not lower bound!\n";+                        std::cout << "   Node:     " << this << "\n";+                        std::cout << "   Parent:   " << this->parent << "\n";+                        std::cout << "   Position: " << ((int)this->position) << "\n";+                        std::cout << "   Key:   " << (this->parent->keys[this->position]) << "\n";+                        std::cout << "   Upper: " << (keys[0]) << "\n";+                        valid = false;+                    }+                }+            }++            // check element order+            if (this->numElements > 0) {+                for (unsigned i = 0; i < this->numElements - 1; i++) {+                    if (valid && !(comp(keys[i], keys[i + 1]) < ((isSet) ? 0 : 1))) {+                        std::cout << "Element order invalid!\n";+                        std::cout << " @" << this << " key " << i << " is " << keys[i] << " vs "+                                  << keys[i + 1] << "\n";+                        valid = false;+                    }+                }+            }++            // check state of sub-nodes+            if (this->inner) {+                for (unsigned i = 0; i <= this->numElements; i++) {+                    valid &= getChildren()[i]->check(comp, root);+                }+            }++            return valid;+        }+    };  // namespace detail++    /**+     * The data type representing inner nodes of the b-tree. It extends+     * the generic implementation of a node by the storage locations+     * of child pointers.+     */+    struct inner_node : public node {+        // references to child nodes owned by this node+        node* children[node::maxKeys + 1];++        // a simple default constructor initializing member fields+        inner_node() : node(true) {}++        // clear up child nodes recursively+        ~inner_node() {+            for (unsigned i = 0; i <= this->numElements; ++i) {+                if (children[i] != nullptr) {+                    if (children[i]->isLeaf()) {+                        delete static_cast<leaf_node*>(children[i]);+                    } else {+                        delete static_cast<inner_node*>(children[i]);+                    }+                }+            }+        }+    };++    /**+     * The data type representing leaf nodes of the b-tree. It does not+     * add any capabilities to the generic node type.+     */+    struct leaf_node : public node {+        // a simple default constructor initializing member fields+        leaf_node() : node(false) {}+    };++    // ------------------- iterators ------------------------++public:+    /**+     * The iterator type to be utilized for scanning through btree instances.+     */+    class iterator : public std::iterator<std::forward_iterator_tag, Key> {+        // a pointer to the node currently referred to+        node const* cur;++        // the index of the element currently addressed within the referenced node+        field_index_type pos = 0;++    public:+        // default constructor -- creating an end-iterator+        iterator() : cur(nullptr) {}++        // creates an iterator referencing a specific element within a given node+        iterator(node const* cur, field_index_type pos) : cur(cur), pos(pos) {}++        // a copy constructor+        iterator(const iterator& other) : cur(other.cur), pos(other.pos) {}++        // an assignment operator+        iterator& operator=(const iterator& other) {+            cur = other.cur;+            pos = other.pos;+            return *this;+        }++        // the equality operator as required by the iterator concept+        bool operator==(const iterator& other) const {+            return cur == other.cur && pos == other.pos;+        }++        // the not-equality operator as required by the iterator concept+        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        // the deref operator as required by the iterator concept+        const Key& operator*() const {+            return cur->keys[pos];+        }++        // the increment operator as required by the iterator concept+        iterator& operator++() {+            // the quick mode -- if in a leaf and there are elements left+            if (cur->isLeaf() && ++pos < cur->getNumElements()) {+                return *this;+            }++            // otherwise it is a bit more tricky++            // A) currently in an inner node => go to the left-most child+            if (cur->isInner()) {+                cur = cur->getChildren()[pos + 1];+                while (!cur->isLeaf()) {+                    cur = cur->getChildren()[0];+                }+                pos = 0;++                // nodes may be empty due to biased insertion+                if (!cur->isEmpty()) {+                    return *this;+                }+            }++            // B) we are at the right-most element of a leaf => go to next inner node+            assert(cur->isLeaf());+            assert(pos == cur->getNumElements());++            while (cur != nullptr && pos == cur->getNumElements()) {+                pos = cur->getPositionInParent();+                cur = cur->getParent();+            }+            return *this;+        }++        // prints a textual representation of this iterator to the given stream (mainly for debugging)+        void print(std::ostream& out = std::cout) const {+            out << cur << "[" << (int)pos << "]";+        }+    };++    /**+     * A collection of operation hints speeding up some of the involved operations+     * by exploiting temporal locality.+     */+    template <unsigned size = 1>+    struct btree_operation_hints {+        using node_cache = LRUCache<node*, size>;++        // the node where the last insertion terminated+        node_cache last_insert;++        // the node where the last find-operation terminated+        node_cache last_find_end;++        // the node where the last lower-bound operation terminated+        node_cache last_lower_bound_end;++        // the node where the last upper-bound operation terminated+        node_cache last_upper_bound_end;++        // default constructor+        btree_operation_hints() = default;++        // resets all hints (to be triggered e.g. when deleting nodes)+        void clear() {+            last_insert.clear(nullptr);+            last_find_end.clear(nullptr);+            last_lower_bound_end.clear(nullptr);+            last_upper_bound_end.clear(nullptr);+        }+    };++    using operation_hints = btree_operation_hints<1>;++protected:+#ifdef IS_PARALLEL+    // a pointer to the root node of this tree+    node* volatile root;++    // a lock to synchronize update operations on the root pointer+    lock_type root_lock;+#else+    // a pointer to the root node of this tree+    node* root;++    // required to not duplicate too much code+    lock_type root_lock;+#endif++    // a pointer to the left-most node of this tree (initial note for iteration)+    leaf_node* leftmost;++    /* -------------- operator hint statistics ----------------- */++    // an aggregation of statistical values of the hint utilization+    struct hint_statistics {+        // the counter for insertion operations+        CacheAccessCounter inserts;++        // the counter for contains operations+        CacheAccessCounter contains;++        // the counter for lower_bound operations+        CacheAccessCounter lower_bound;++        // the counter for upper_bound operations+        CacheAccessCounter upper_bound;+    };++    // the hint statistic of this b-tree instance+    mutable hint_statistics hint_stats;++public:+    // the maximum number of keys stored per node+    static constexpr size_t max_keys_per_node = node::maxKeys;++    // -- ctors / dtors --++    // the default constructor creating an empty tree+    btree(Comparator comp = Comparator(), WeakComparator weak_comp = WeakComparator())+            : comp(std::move(comp)), weak_comp(std::move(weak_comp)), root(nullptr), leftmost(nullptr) {}++    // a constructor creating a tree from the given iterator range+    template <typename Iter>+    btree(const Iter& a, const Iter& b) : root(nullptr), leftmost(nullptr) {+        insert(a, b);+    }++    // a move constructor+    btree(btree&& other)+            : comp(other.comp), weak_comp(other.weak_comp), root(other.root), leftmost(other.leftmost) {+        other.root = nullptr;+        other.leftmost = nullptr;+    }++    // a copy constructor+    btree(const btree& set) : comp(set.comp), weak_comp(set.weak_comp), root(nullptr), leftmost(nullptr) {+        // use assignment operator for a deep copy+        *this = set;+    }++protected:+    /**+     * An internal constructor enabling the specific creation of a tree+     * based on internal parameters.+     */+    btree(size_type /* size */, node* root, leaf_node* leftmost) : root(root), leftmost(leftmost) {}++public:+    // the destructor freeing all contained nodes+    ~btree() {+        clear();+    }++    // -- mutators and observers --++    // emptiness check+    bool empty() const {+        return root == nullptr;+    }++    // determines the number of elements in this tree+    size_type size() const {+        return (root) ? root->countEntries() : 0;+    }++    /**+     * Inserts the given key into this tree.+     */+    bool insert(const Key& k) {+        operation_hints hints;+        return insert(k, hints);+    }++    /**+     * Inserts the given key into this tree.+     */+    bool insert(const Key& k, operation_hints& hints) {+#ifdef IS_PARALLEL++        // special handling for inserting first element+        while (root == nullptr) {+            // try obtaining root-lock+            if (!root_lock.try_start_write()) {+                // somebody else was faster => re-check+                continue;+            }++            // check loop condition again+            if (root != nullptr) {+                // somebody else was faster => normal insert+                root_lock.end_write();+                break;+            }++            // create new node+            leftmost = new leaf_node();+            leftmost->numElements = 1;+            leftmost->keys[0] = k;+            root = leftmost;++            // operation complete => we can release the root lock+            root_lock.end_write();++            hints.last_insert.access(leftmost);++            return true;+        }++        // insert using iterative implementation++        node* cur = nullptr;++        // test last insert hints+        lock_type::Lease cur_lease;++        auto checkHint = [&](node* last_insert) {+            // ignore null pointer+            if (!last_insert) return false;+            // get a read lease on indicated node+            auto hint_lease = last_insert->lock.start_read();+            // check whether it covers the key+            if (!weak_covers(last_insert, k)) return false;+            // and if there was no concurrent modification+            if (!last_insert->lock.validate(hint_lease)) return false;+            // use hinted location+            cur = last_insert;+            // and keep lease+            cur_lease = hint_lease;+            // we found a hit+            return true;+        };++        if (hints.last_insert.any(checkHint)) {+            // register this as a hit+            hint_stats.inserts.addHit();+        } else {+            // register this as a miss+            hint_stats.inserts.addMiss();+        }++        // if there is no valid hint ..+        if (!cur) {+            do {+                // get root - access lock+                auto root_lease = root_lock.start_read();++                // start with root+                cur = root;++                // get lease of the next node to be accessed+                cur_lease = cur->lock.start_read();++                // check validity of root pointer+                if (root_lock.end_read(root_lease)) {+                    break;+                }++            } while (true);+        }++        while (true) {+            // handle inner nodes+            if (cur->inner) {+                auto a = &(cur->keys[0]);+                auto b = &(cur->keys[cur->numElements]);++                auto pos = search.lower_bound(k, a, b, weak_comp);+                auto idx = pos - a;++                // early exit for sets+                if (isSet && pos != b && weak_equal(*pos, k)) {+                    // validate results+                    if (!cur->lock.validate(cur_lease)) {+                        // start over again+                        return insert(k, hints);+                    }++                    // update provenance information+                    if (typeid(Comparator) != typeid(WeakComparator) && less(k, *pos)) {+                        if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                            // start again+                            return insert(k, hints);+                        }+                        update(*pos, k);+                        cur->lock.end_write();+                        return true;+                    }++                    // we found the element => no check of lock necessary+                    return false;+                }++                // get next pointer+                auto next = cur->getChild(idx);++                // get lease on next level+                auto next_lease = next->lock.start_read();++                // check whether there was a write+                if (!cur->lock.end_read(cur_lease)) {+                    // start over+                    return insert(k, hints);+                }++                // go to next+                cur = next;++                // move on lease+                cur_lease = next_lease;++                continue;+            }++            // the rest is for leaf nodes+            assert(!cur->inner);++            // -- insert node in leaf node --++            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = search.upper_bound(k, a, b, weak_comp);+            auto idx = pos - a;++            // early exit for sets+            if (isSet && pos != a && weak_equal(*(pos - 1), k)) {+                // validate result+                if (!cur->lock.validate(cur_lease)) {+                    // start over again+                    return insert(k, hints);+                }++                // update provenance information+                if (typeid(Comparator) != typeid(WeakComparator) && less(k, *(pos - 1))) {+                    if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                        // start again+                        return insert(k, hints);+                    }+                    update(*(pos - 1), k);+                    cur->lock.end_write();+                    return true;+                }++                // we found the element => done+                return false;+            }++            // upgrade to write-permission+            if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                // something has changed => restart+                hints.last_insert.access(cur);+                return insert(k, hints);+            }++            if (cur->numElements >= node::maxKeys) {+                // -- lock parents --+                auto priv = cur;+                auto parent = priv->parent;+                std::vector<node*> parents;+                do {+                    if (parent) {+                        parent->lock.start_write();+                        while (true) {+                            // check whether parent is correct+                            if (parent == priv->parent) {+                                break;+                            }+                            // switch parent+                            parent->lock.abort_write();+                            parent = priv->parent;+                            parent->lock.start_write();+                        }+                    } else {+                        // lock root lock => since cur is root+                        root_lock.start_write();+                    }++                    // record locked node+                    parents.push_back(parent);++                    // stop at "sphere of influence"+                    if (!parent || !parent->isFull()) {+                        break;+                    }++                    // go one step higher+                    priv = parent;+                    parent = parent->parent;++                } while (true);++                // split this node+                auto old_root = root;+                idx -= cur->rebalance_or_split(const_cast<node**>(&root), root_lock, idx, parents);++                // release parent lock+                for (auto it = parents.rbegin(); it != parents.rend(); ++it) {+                    auto parent = *it;++                    // release this lock+                    if (parent) {+                        parent->lock.end_write();+                    } else {+                        if (old_root != root) {+                            root_lock.end_write();+                        } else {+                            root_lock.abort_write();+                        }+                    }+                }++                // insert element in right fragment+                if (((size_type)idx) > cur->numElements) {+                    // release current lock+                    cur->lock.end_write();++                    // insert in sibling+                    return insert(k, hints);+                }+            }++            // ok - no split necessary+            assert(cur->numElements < node::maxKeys && "Split required!");++            // move keys+            for (int j = cur->numElements; j > idx; --j) {+                cur->keys[j] = cur->keys[j - 1];+            }++            // insert new element+            cur->keys[idx] = k;+            cur->numElements++;++            // release lock on current node+            cur->lock.end_write();++            // remember last insertion position+            hints.last_insert.access(cur);+            return true;+        }++#else+        // special handling for inserting first element+        if (empty()) {+            // create new node+            leftmost = new leaf_node();+            leftmost->numElements = 1;+            leftmost->keys[0] = k;+            root = leftmost;++            hints.last_insert.access(leftmost);++            return true;+        }++        // insert using iterative implementation+        node* cur = root;++        auto checkHints = [&](node* last_insert) {+            if (!last_insert) return false;+            if (!weak_covers(last_insert, k)) return false;+            cur = last_insert;+            return true;+        };++        // test last insert+        if (hints.last_insert.any(checkHints)) {+            hint_stats.inserts.addHit();+        } else {+            hint_stats.inserts.addMiss();+        }++        while (true) {+            // handle inner nodes+            if (cur->inner) {+                auto a = &(cur->keys[0]);+                auto b = &(cur->keys[cur->numElements]);++                auto pos = search.lower_bound(k, a, b, weak_comp);+                auto idx = pos - a;++                // early exit for sets+                if (isSet && pos != b && weak_equal(*pos, k)) {+                    // update provenance information+                    if (typeid(Comparator) != typeid(WeakComparator) && less(k, *pos)) {+                        update(*pos, k);+                        return true;+                    }++                    return false;+                }++                cur = cur->getChild(idx);+                continue;+            }++            // the rest is for leaf nodes+            assert(!cur->inner);++            // -- insert node in leaf node --++            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = search.upper_bound(k, a, b, weak_comp);+            auto idx = pos - a;++            // early exit for sets+            if (isSet && pos != a && weak_equal(*(pos - 1), k)) {+                // update provenance information+                if (typeid(Comparator) != typeid(WeakComparator) && less(k, *(pos - 1))) {+                    update(*(pos - 1), k);+                    return true;+                }++                return false;+            }++            if (cur->numElements >= node::maxKeys) {+                // split this node+                idx -= cur->rebalance_or_split(&root, root_lock, idx);++                // insert element in right fragment+                if (((size_type)idx) > cur->numElements) {+                    idx -= cur->numElements + 1;+                    cur = cur->parent->getChild(cur->position + 1);+                }+            }++            // ok - no split necessary+            assert(cur->numElements < node::maxKeys && "Split required!");++            // move keys+            for (int j = cur->numElements; j > idx; --j) {+                cur->keys[j] = cur->keys[j - 1];+            }++            // insert new element+            cur->keys[idx] = k;+            cur->numElements++;++            // remember last insertion position+            hints.last_insert.access(cur);++            return true;+        }+#endif+    }++    /**+     * Inserts the given range of elements into this tree.+     */+    template <typename Iter>+    void insert(const Iter& a, const Iter& b) {+        // TODO: improve this beyond a naive insert+        operation_hints hints;+        // a naive insert so far .. seems to work fine+        for (auto it = a; it != b; ++it) {+            // use insert with hint+            insert(*it, hints);+        }+    }++    // Obtains an iterator referencing the first element of the tree.+    iterator begin() const {+        return iterator(leftmost, 0);+    }++    // Obtains an iterator referencing the position after the last element of the tree.+    iterator end() const {+        return iterator();+    }++    /**+     * Partitions the full range of this set into up to a given number of chunks.+     * The chunks will cover approximately the same number of elements. Also, the+     * number of chunks will only approximate the desired number of chunks.+     *+     * @param num .. the number of chunks requested+     * @return a list of chunks partitioning this tree+     */+    std::vector<chunk> partition(size_type num) const {+        return getChunks(num);+    }++    std::vector<chunk> getChunks(size_type num) const {+        std::vector<chunk> res;+        if (empty()) {+            return res;+        }+        return root->collectChunks(res, num, begin(), end());+    }++    /**+     * Determines whether the given element is a member of this tree.+     */+    bool contains(const Key& k) const {+        operation_hints hints;+        return contains(k, hints);+    }++    /**+     * Determines whether the given element is a member of this tree.+     */+    bool contains(const Key& k, operation_hints& hints) const {+        return find(k, hints) != end();+    }++    /**+     * Locates the given key within this tree and returns an iterator+     * referencing its position. If not found, an end-iterator will be returned.+     */+    iterator find(const Key& k) const {+        operation_hints hints;+        return find(k, hints);+    }++    /**+     * Locates the given key within this tree and returns an iterator+     * referencing its position. If not found, an end-iterator will be returned.+     */+    iterator find(const Key& k, operation_hints& hints) const {+        if (empty()) {+            return end();+        }++        node* cur = root;++        auto checkHints = [&](node* last_find_end) {+            if (!last_find_end) return false;+            if (!covers(last_find_end, k)) return false;+            cur = last_find_end;+            return true;+        };++        // test last location searched (temporal locality)+        if (hints.last_find_end.any(checkHints)) {+            // register it as a hit+            hint_stats.contains.addHit();+        } else {+            // register it as a miss+            hint_stats.contains.addMiss();+        }++        // an iterative implementation (since 2/7 faster than recursive)++        while (true) {+            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = search(k, a, b, comp);++            if (pos < b && equal(*pos, k)) {+                hints.last_find_end.access(cur);+                return iterator(cur, pos - a);+            }++            if (!cur->inner) {+                hints.last_find_end.access(cur);+                return end();+            }++            // continue search in child node+            cur = cur->getChild(pos - a);+        }+    }++    /**+     * Obtains a lower boundary for the given key -- hence an iterator referencing+     * the smallest value that is not less the given key. If there is no such element,+     * an end-iterator will be returned.+     */+    iterator lower_bound(const Key& k) const {+        operation_hints hints;+        return lower_bound(k, hints);+    }++    /**+     * Obtains a lower boundary for the given key -- hence an iterator referencing+     * the smallest value that is not less the given key. If there is no such element,+     * an end-iterator will be returned.+     */+    iterator lower_bound(const Key& k, operation_hints& hints) const {+        if (empty()) {+            return end();+        }++        node* cur = root;++        auto checkHints = [&](node* last_lower_bound_end) {+            if (!last_lower_bound_end) return false;+            if (!covers(last_lower_bound_end, k)) return false;+            cur = last_lower_bound_end;+            return true;+        };++        // test last searched node+        if (hints.last_lower_bound_end.any(checkHints)) {+            hint_stats.lower_bound.addHit();+        } else {+            hint_stats.lower_bound.addMiss();+        }++        iterator res = end();+        while (true) {+            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = search.lower_bound(k, a, b, comp);+            auto idx = pos - a;++            if (!cur->inner) {+                hints.last_lower_bound_end.access(cur);+                return (pos != b) ? iterator(cur, idx) : res;+            }++            if (isSet && pos != b && equal(*pos, k)) {+                return iterator(cur, idx);+            }++            if (pos != b) {+                res = iterator(cur, idx);+            }++            cur = cur->getChild(idx);+        }+    }++    /**+     * Obtains an upper boundary for the given key -- hence an iterator referencing+     * the first element that the given key is less than the referenced value. If+     * there is no such element, an end-iterator will be returned.+     */+    iterator upper_bound(const Key& k) const {+        operation_hints hints;+        return upper_bound(k, hints);+    }++    /**+     * Obtains an upper boundary for the given key -- hence an iterator referencing+     * the first element that the given key is less than the referenced value. If+     * there is no such element, an end-iterator will be returned.+     */+    iterator upper_bound(const Key& k, operation_hints& hints) const {+        if (empty()) {+            return end();+        }++        node* cur = root;++        auto checkHints = [&](node* last_upper_bound_end) {+            if (!last_upper_bound_end) return false;+            if (!coversUpperBound(last_upper_bound_end, k)) return false;+            cur = last_upper_bound_end;+            return true;+        };++        // test last search node+        if (hints.last_upper_bound_end.any(checkHints)) {+            hint_stats.upper_bound.addHit();+        } else {+            hint_stats.upper_bound.addMiss();+        }++        iterator res = end();+        while (true) {+            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = search.upper_bound(k, a, b, comp);+            auto idx = pos - a;++            if (!cur->inner) {+                hints.last_upper_bound_end.access(cur);+                return (pos != b) ? iterator(cur, idx) : res;+            }++            if (pos != b) {+                res = iterator(cur, idx);+            }++            cur = cur->getChild(idx);+        }+    }++    /**+     * Clears this tree.+     */+    void clear() {+        if (root != nullptr) {+            if (root->isLeaf()) {+                delete static_cast<leaf_node*>(root);+            } else {+                delete static_cast<inner_node*>(root);+            }+        }+        root = nullptr;+        leftmost = nullptr;+    }++    /**+     * Swaps the content of this tree with the given tree. This+     * is a much more efficient operation than creating a copy and+     * realizing the swap utilizing assignment operations.+     */+    void swap(btree& other) {+        // swap the content+        std::swap(root, other.root);+        std::swap(leftmost, other.leftmost);+    }++    // Implementation of the assignment operation for trees.+    btree& operator=(const btree& other) {+        // check identity+        if (this == &other) {+            return *this;+        }++        // create a deep-copy of the content of the other tree+        // shortcut for empty sets+        if (other.empty()) {+            return *this;+        }++        // clone content (deep copy)+        root = other.root->clone();++        // update leftmost reference+        auto tmp = root;+        while (!tmp->isLeaf()) {+            tmp = tmp->getChild(0);+        }+        leftmost = static_cast<leaf_node*>(tmp);++        // done+        return *this;+    }++    // Implementation of an equality operation for trees.+    bool operator==(const btree& other) const {+        // check identity+        if (this == &other) {+            return true;+        }++        // check size+        if (size() != other.size()) {+            return false;+        }+        if (size() < other.size()) {+            return other == *this;+        }++        // check content+        for (const auto& key : other) {+            if (!contains(key)) {+                return false;+            }+        }+        return true;+    }++    // Implementation of an inequality operation for trees.+    bool operator!=(const btree& other) const {+        return !(*this == other);+    }++    // -- for debugging --++    // Determines the number of levels contained in this tree.+    size_type getDepth() const {+        return (empty()) ? 0 : root->getDepth();+    }++    // Determines the number of nodes contained in this tree.+    size_type getNumNodes() const {+        return (empty()) ? 0 : root->countNodes();+    }++    // Determines the amount of memory used by this data structure+    size_type getMemoryUsage() const {+        return sizeof(*this) + (empty() ? 0 : root->getMemoryUsage());+    }++    /*+     * Prints a textual representation of this tree to the given+     * output stream (mostly for debugging and tuning).+     */+    void printTree(std::ostream& out = std::cout) const {+        out << "B-Tree with " << size() << " elements:\n";+        if (empty()) {+            out << " - empty - \n";+        } else {+            root->printTree(out, "");+        }+    }++    /**+     * Prints a textual summary of statistical properties of this+     * tree to the given output stream (for debugging and tuning).+     */+    void printStats(std::ostream& out = std::cout) const {+        auto nodes = getNumNodes();+        out << " ---------------------------------\n";+        out << "  Elements: " << size() << "\n";+        out << "  Depth:    " << (empty() ? 0 : root->getDepth()) << "\n";+        out << "  Nodes:    " << nodes << "\n";+        out << " ---------------------------------\n";+        out << "  Size of inner node: " << sizeof(inner_node) << "\n";+        out << "  Size of leaf node:  " << sizeof(leaf_node) << "\n";+        out << "  Size of Key:        " << sizeof(Key) << "\n";+        out << "  max keys / node:  " << node::maxKeys << "\n";+        out << "  avg keys / node:  " << (size() / (double)nodes) << "\n";+        out << "  avg filling rate: " << ((size() / (double)nodes) / node::maxKeys) << "\n";+        out << " ---------------------------------\n";+        out << "  insert-hint (hits/misses/total): " << hint_stats.inserts.getHits() << "/"+            << hint_stats.inserts.getMisses() << "/" << hint_stats.inserts.getAccesses() << "\n";+        out << "  contains-hint(hits/misses/total):" << hint_stats.contains.getHits() << "/"+            << hint_stats.contains.getMisses() << "/" << hint_stats.contains.getAccesses() << "\n";+        out << "  lower-bound-hint (hits/misses/total):" << hint_stats.lower_bound.getHits() << "/"+            << hint_stats.lower_bound.getMisses() << "/" << hint_stats.lower_bound.getAccesses() << "\n";+        out << "  upper-bound-hint (hits/misses/total):" << hint_stats.upper_bound.getHits() << "/"+            << hint_stats.upper_bound.getMisses() << "/" << hint_stats.upper_bound.getAccesses() << "\n";+        out << " ---------------------------------\n";+    }++    /**+     * Checks the consistency of this tree.+     */+    bool check() {+        auto ok = empty() || root->check(comp, root);+        if (!ok) {+            printTree();+        }+        return ok;+    }++    /**+     * A static member enabling the bulk-load of ordered data into an empty+     * tree. This function is much more efficient in creating a index over+     * an ordered set of elements than an iterative insertion of values.+     *+     * @tparam Iter .. the type of iterator specifying the range+     *                     it must be a random-access iterator+     */+    template <typename R, typename Iter>+    static typename std::enable_if<std::is_same<typename std::iterator_traits<Iter>::iterator_category,+                                           std::random_access_iterator_tag>::value,+            R>::type+    load(const Iter& a, const Iter& b) {+        // quick exit - empty range+        if (a == b) {+            return R();+        }++        // resolve tree recursively+        auto root = buildSubTree(a, b - 1);++        // find leftmost node+        node* leftmost = root;+        while (!leftmost->isLeaf()) {+            leftmost = leftmost->getChild(0);+        }++        // build result+        return R(b - a, root, static_cast<leaf_node*>(leftmost));+    }++protected:+    /**+     * Determines whether the range covered by the given node is also+     * covering the given key value.+     */+    bool covers(const node* node, const Key& k) const {+        if (isSet) {+            // in sets we can include the ends as covered elements+            return !node->isEmpty() && !less(k, node->keys[0]) && !less(node->keys[node->numElements - 1], k);+        }+        // in multi-sets the ends may not be completely covered+        return !node->isEmpty() && less(node->keys[0], k) && less(k, node->keys[node->numElements - 1]);+    }++    /**+     * Determines whether the range covered by the given node is also+     * covering the given key value.+     */+    bool weak_covers(const node* node, const Key& k) const {+        if (isSet) {+            // in sets we can include the ends as covered elements+            return !node->isEmpty() && !weak_less(k, node->keys[0]) &&+                   !weak_less(node->keys[node->numElements - 1], k);+        }+        // in multi-sets the ends may not be completely covered+        return !node->isEmpty() && weak_less(node->keys[0], k) &&+               weak_less(k, node->keys[node->numElements - 1]);+    }++private:+    /**+     * Determines whether the range covered by this node covers+     * the upper bound of the given key.+     */+    bool coversUpperBound(const node* node, const Key& k) const {+        // ignore edges+        return !node->isEmpty() && !less(k, node->keys[0]) && less(k, node->keys[node->numElements - 1]);+    }++    // Utility function for the load operation above.+    template <typename Iter>+    static node* buildSubTree(const Iter& a, const Iter& b) {+        const int N = node::maxKeys;++        // divide range in N+1 sub-ranges+        int length = (b - a) + 1;++        // terminal case: length is less then maxKeys+        if (length <= N) {+            // create a leaf node+            node* res = new leaf_node();+            res->numElements = length;++            for (int i = 0; i < length; ++i) {+                res->keys[i] = a[i];+            }++            return res;+        }++        // recursive case - compute step size+        int numKeys = N;+        int step = ((length - numKeys) / (numKeys + 1));++        while (numKeys > 1 && (step < N / 2)) {+            numKeys--;+            step = ((length - numKeys) / (numKeys + 1));+        }++        // create inner node+        node* res = new inner_node();+        res->numElements = numKeys;++        Iter c = a;+        for (int i = 0; i < numKeys; i++) {+            // get dividing key+            res->keys[i] = c[step];++            // get sub-tree+            auto child = buildSubTree(c, c + (step - 1));+            child->parent = res;+            child->position = i;+            res->getChildren()[i] = child;++            c = c + (step + 1);+        }++        // and the remaining part+        auto child = buildSubTree(c, b);+        child->parent = res;+        child->position = numKeys;+        res->getChildren()[numKeys] = child;++        // done+        return res;+    }+};  // namespace souffle++// Instantiation of static member search.+template <typename Key, typename Comparator, typename Allocator, unsigned blockSize, typename SearchStrategy,+        bool isSet, typename WeakComparator, typename Updater>+const SearchStrategy+        btree<Key, Comparator, Allocator, blockSize, SearchStrategy, isSet, WeakComparator, Updater>::search;++}  // end namespace detail++/**+ * A b-tree based set implementation.+ *+ * @tparam Key             .. the element type to be stored in this set+ * @tparam Comparator     .. a class defining an order on the stored elements+ * @tparam Allocator     .. utilized for allocating memory for required nodes+ * @tparam blockSize    .. determines the number of bytes/block utilized by leaf nodes+ * @tparam SearchStrategy .. enables switching between linear, binary or any other search strategy+ */+template <typename Key, typename Comparator = detail::comparator<Key>,+        typename Allocator = std::allocator<Key>,  // is ignored so far+        unsigned blockSize = 256,+        typename SearchStrategy = typename souffle::detail::default_strategy<Key>::type,+        typename WeakComparator = Comparator, typename Updater = souffle::detail::updater<Key>>+class btree_set : public souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy, true,+                          WeakComparator, Updater> {+    using super = souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy, true,+            WeakComparator, Updater>;++    friend class souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy, true,+            WeakComparator, Updater>;++public:+    /**+     * A default constructor creating an empty set.+     */+    btree_set(const Comparator& comp = Comparator(), const WeakComparator& weak_comp = WeakComparator())+            : super(comp, weak_comp) {}++    /**+     * A constructor creating a set based on the given range.+     */+    template <typename Iter>+    btree_set(const Iter& a, const Iter& b) {+        this->insert(a, b);+    }++    // A copy constructor.+    btree_set(const btree_set& other) : super(other) {}++    // A move constructor.+    btree_set(btree_set&& other) : super(std::move(other)) {}++private:+    // A constructor required by the bulk-load facility.+    template <typename s, typename n, typename l>+    btree_set(s size, n* root, l* leftmost) : super(size, root, leftmost) {}++public:+    // Support for the assignment operator.+    btree_set& operator=(const btree_set& other) {+        super::operator=(other);+        return *this;+    }++    // Support for the bulk-load operator.+    template <typename Iter>+    static btree_set load(const Iter& a, const Iter& b) {+        return super::template load<btree_set>(a, b);+    }+};++/**+ * A b-tree based multi-set implementation.+ *+ * @tparam Key             .. the element type to be stored in this set+ * @tparam Comparator     .. a class defining an order on the stored elements+ * @tparam Allocator     .. utilized for allocating memory for required nodes+ * @tparam blockSize    .. determines the number of bytes/block utilized by leaf nodes+ * @tparam SearchStrategy .. enables switching between linear, binary or any other search strategy+ */+template <typename Key, typename Comparator = detail::comparator<Key>,+        typename Allocator = std::allocator<Key>,  // is ignored so far+        unsigned blockSize = 256,+        typename SearchStrategy = typename souffle::detail::default_strategy<Key>::type,+        typename WeakComparator = Comparator, typename Updater = souffle::detail::updater<Key>>+class btree_multiset : public souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy,+                               false, WeakComparator, Updater> {+    using super = souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy, false,+            WeakComparator, Updater>;++    friend class souffle::detail::btree<Key, Comparator, Allocator, blockSize, SearchStrategy, false,+            WeakComparator, Updater>;++public:+    /**+     * A default constructor creating an empty set.+     */+    btree_multiset(const Comparator& comp = Comparator(), const WeakComparator& weak_comp = WeakComparator())+            : super(comp, weak_comp) {}++    /**+     * A constructor creating a set based on the given range.+     */+    template <typename Iter>+    btree_multiset(const Iter& a, const Iter& b) {+        this->insert(a, b);+    }++    // A copy constructor.+    btree_multiset(const btree_multiset& other) : super(other) {}++    // A move constructor.+    btree_multiset(btree_multiset&& other) : super(std::move(other)) {}++private:+    // A constructor required by the bulk-load facility.+    template <typename s, typename n, typename l>+    btree_multiset(s size, n* root, l* leftmost) : super(size, root, leftmost) {}++public:+    // Support for the assignment operator.+    btree_multiset& operator=(const btree_multiset& other) {+        super::operator=(other);+        return *this;+    }++    // Support for the bulk-load operator.+    template <typename Iter>+    static btree_multiset load(const Iter& a, const Iter& b) {+        return super::template load<btree_multiset>(a, b);+    }+};++}  // end of namespace souffle
+ cbits/souffle/Brie.h view
@@ -0,0 +1,3166 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file Brie.h+ *+ * This header file contains the implementation for a generic, fixed+ * length integer trie.+ *+ * Tries trie is utilized to store n-ary tuples of integers. Each level+ * is implemented via a sparse array (also covered by this header file),+ * referencing the following nested level. The leaf level is realized+ * by a sparse bit-map to minimize the memory footprint.+ *+ * Multiple insert operations can be be conducted concurrently on trie+ * structures. So can read-only operations. However, inserts and read+ * operations may not be conducted at the same time.+ *+ ***********************************************************************/++#pragma once++#include "CompiledTuple.h"+#include "RamTypes.h"+#include "utility/CacheUtil.h"+#include "utility/ContainerUtil.h"+#include "utility/StreamUtil.h"+#include <algorithm>+#include <atomic>+#include <bitset>+#include <cassert>+#include <cstdint>+#include <cstring>+#include <iostream>+#include <iterator>+#include <limits>+#include <utility>+#include <vector>++#ifdef _WIN32+/**+ * When compiling for windows, redefine the gcc builtins which are used to+ * their equivalents on the windows platform.+ */+#define __sync_synchronize MemoryBarrier+#define __sync_bool_compare_and_swap(ptr, oldval, newval) \+    (InterlockedCompareExchangePointer((void* volatile*)ptr, (void*)newval, (void*)oldval) == (void*)oldval)+#endif  // _WIN32++namespace souffle {++namespace detail {++/**+ * A templated functor to obtain default values for+ * unspecified elements of sparse array instances.+ */+template <typename T>+struct default_factory {+    T operator()() const {+        return T();  // just use the default constructor+    }+};++/**+ * A functor representing the identity function.+ */+template <typename T>+struct identity {+    T operator()(T v) const {+        return v;+    }+};++/**+ * A operation to be utilized by the sparse map when merging+ * elements associated to different values.+ */+template <typename T>+struct default_merge {+    /**+     * Merges two values a and b when merging spase maps.+     */+    T operator()(T a, T b) const {+        default_factory<T> def;+        // if a is the default => us b, else stick to a+        return (a != def()) ? a : b;+    }+};++}  // end namespace detail++/**+ * A sparse array simulates an array associating to every element+ * of uint32_t an element of a generic type T. Any non-defined element+ * will be default-initialized utilizing the detail::default_factory+ * functor.+ *+ * Internally the array is organized as a balanced tree. The leaf+ * level of the tree corresponds to the elements of the represented+ * array. Inner nodes utilize individual bits of the indices to reference+ * sub-trees. For efficiency reasons, only the minimal sub-tree required+ * to cover all non-null / non-default values stored in the array is+ * maintained. Furthermore, several levels of nodes are aggreated in a+ * B-tree like fashion to inprove cache utilization and reduce the number+ * of steps required for lookup and insert operations.+ *+ * @tparam T the type of the stored elements+ * @tparam BITS the number of bits consumed per node-level+ *              e.g. if it is set to 3, the resulting tree will be of a degree of+ *              2^3=8, and thus 8 child-pointers will be stored in each inner node+ *              and as many values will be stored in each leaf node.+ * @tparam merge_op the functor to be utilized when merging the content of two+ *              instances of this type.+ * @tparam copy_op a functor to be applied to each stored value when copying an+ *              instance of this array. For instance, this is utilized by the+ *              trie implementation to create a clone of each sub-tree instead+ *              of preserving the original pointer.+ */+template <typename T, unsigned BITS = 6, typename merge_op = detail::default_merge<T>,+        typename copy_op = detail::identity<T>>+class SparseArray {+    using key_type = uint64_t;++    // some internal constants+    static constexpr int BIT_PER_STEP = BITS;+    static constexpr int NUM_CELLS = 1 << BIT_PER_STEP;+    static constexpr key_type INDEX_MASK = NUM_CELLS - 1;++public:+    // the type utilized for indexing contained elements+    using index_type = key_type;++    // the type of value stored in this array+    using value_type = T;++    // the atomic view on stored values+    using atomic_value_type = std::atomic<value_type>;++private:+    struct Node;++    /**+     * The value stored in a single cell of a inner+     * or leaf node.+     */+    union Cell {+        // an atomic view on the pointer referencing a nested level+        std::atomic<Node*> aptr;++        // a pointer to the nested level (unsynchronized operations)+        Node* ptr;++        // an atomic view on the value stored in this cell (leaf node)+        atomic_value_type avalue;++        // the value stored in this cell (unsynchronized access, leaf node)+        value_type value;+    };++    /**+     * The node type of the internally maintained tree.+     */+    struct Node {+        // a pointer to the parent node (for efficient iteration)+        const Node* parent;+        // the pointers to the child nodes (inner nodes) or the stored values (leaf nodes)+        Cell cell[NUM_CELLS];+    };++    /**+     * A struct describing all the information required by the container+     * class to manage the wrapped up tree.+     */+    struct RootInfo {+        // the root node of the tree+        Node* root;+        // the number of levels of the tree+        uint32_t levels;+        // the absolute offset of the theoretical first element in the tree+        index_type offset;++        // the first leaf node in the tree+        Node* first;+        // the absolute offset of the first element in the first leaf node+        index_type firstOffset;+    };++    union {+        RootInfo unsynced;         // for sequential operations+        volatile RootInfo synced;  // for synchronized operations+    };++public:+    /**+     * A default constructor creating an empty sparse array.+     */+    SparseArray() : unsynced(RootInfo{nullptr, 0, 0, nullptr, std::numeric_limits<index_type>::max()}) {}++    /**+     * A copy constructor for sparse arrays. It creates a deep+     * copy of the data structure maintained by the handed in+     * array instance.+     */+    SparseArray(const SparseArray& other)+            : unsynced(RootInfo{clone(other.unsynced.root, other.unsynced.levels), other.unsynced.levels,+                      other.unsynced.offset, nullptr, other.unsynced.firstOffset}) {+        if (unsynced.root) {+            unsynced.root->parent = nullptr;+            unsynced.first = findFirst(unsynced.root, unsynced.levels);+        }+    }++    /**+     * A r-value based copy constructor for sparse arrays. It+     * takes over ownership of the structure maintained by the+     * handed in array.+     */+    SparseArray(SparseArray&& other)+            : unsynced(RootInfo{other.unsynced.root, other.unsynced.levels, other.unsynced.offset,+                      other.unsynced.first, other.unsynced.firstOffset}) {+        other.unsynced.root = nullptr;+        other.unsynced.levels = 0;+        other.unsynced.first = nullptr;+    }++    /**+     * A destructor for sparse arrays clearing up the internally+     * maintained data structure.+     */+    ~SparseArray() {+        clean();+    }++    /**+     * An assignment creating a deep copy of the handed in+     * array structure (utilizing the copy functor provided+     * as a template parameter).+     */+    SparseArray& operator=(const SparseArray& other) {+        if (this == &other) return *this;++        // clean this one+        clean();++        // copy content+        unsynced.levels = other.unsynced.levels;+        unsynced.root = clone(other.unsynced.root, unsynced.levels);+        if (unsynced.root) {+            unsynced.root->parent = nullptr;+        }+        unsynced.offset = other.unsynced.offset;+        unsynced.first = (unsynced.root) ? findFirst(unsynced.root, unsynced.levels) : nullptr;+        unsynced.firstOffset = other.unsynced.firstOffset;++        // done+        return *this;+    }++    /**+     * An assignment operation taking over ownership+     * from a r-value reference to a sparse array.+     */+    SparseArray& operator=(SparseArray&& other) {+        // clean this one+        clean();++        // harvest content+        unsynced.root = other.unsynced.root;+        unsynced.levels = other.unsynced.levels;+        unsynced.offset = other.unsynced.offset;+        unsynced.first = other.unsynced.first;+        unsynced.firstOffset = other.unsynced.firstOffset;++        // reset other+        other.unsynced.root = nullptr;+        other.unsynced.levels = 0;+        other.unsynced.first = nullptr;++        // done+        return *this;+    }++    /**+     * Tests whether this sparse array is empty, thus it only+     * contains default-values, or not.+     */+    bool empty() const {+        return unsynced.root == nullptr;+    }++    /**+     * Computes the number of non-empty elements within this+     * sparse array.+     */+    std::size_t size() const {+        // quick one for the empty map+        if (empty()) return 0;++        // count elements -- since maintaining is making inserts more expensive+        std::size_t res = 0;+        for (auto it = begin(); it != end(); ++it) {+            ++res;+        }+        return res;+    }++private:+    /**+     * Computes the memory usage of the given sub-tree.+     */+    static std::size_t getMemoryUsage(const Node* node, int level) {+        // support null-nodes+        if (!node) return 0;++        // add size of current node+        std::size_t res = sizeof(Node);++        // sum up memory usage of child nodes+        if (level > 0) {+            for (int i = 0; i < NUM_CELLS; i++) {+                res += getMemoryUsage(node->cell[i].ptr, level - 1);+            }+        }++        // done+        return res;+    }++public:+    /**+     * Computes the total memory usage of this data structure.+     */+    std::size_t getMemoryUsage() const {+        // the memory of the wrapper class+        std::size_t res = sizeof(*this);++        // add nodes+        if (unsynced.root) {+            res += getMemoryUsage(unsynced.root, unsynced.levels);+        }++        // done+        return res;+    }++    /**+     * Resets the content of this array to default values for each contained+     * element.+     */+    void clear() {+        clean();+        unsynced.root = nullptr;+        unsynced.levels = 0;+        unsynced.first = nullptr;+        unsynced.firstOffset = std::numeric_limits<index_type>::max();+    }++    /**+     * A struct to be utilized as a local, temporal context by client code+     * to speed up the execution of various operations (optional parameter).+     */+    struct op_context {+        index_type lastIndex{0};+        Node* lastNode{nullptr};+        op_context() = default;+    };++private:+    // ---------------------------------------------------------------------+    //              Optimistic Locking of Root-Level Infos+    // ---------------------------------------------------------------------++    /**+     * A struct to cover a snapshot of the root node state.+     */+    struct RootInfoSnapshot {+        // the current pointer to a root node+        Node* root;+        // the current number of levels+        uint32_t levels;+        // the current offset of the first theoretical element+        index_type offset;+        // a version number for the optimistic locking+        uintptr_t version;+    };++    /**+     * Obtains the current version of the root.+     */+    uint64_t getRootVersion() const {+        // here it is assumed that the load of a 64-bit word is atomic+        return (uint64_t)synced.root;+    }++    /**+     * Obtains a snapshot of the current root information.+     */+    RootInfoSnapshot getRootInfo() const {+        RootInfoSnapshot res{};+        do {+            // first take the mod counter+            do {+                // if res.mod % 2 == 1 .. there is an update in progress+                res.version = getRootVersion();+            } while (res.version % 2);++            // then the rest+            res.root = synced.root;+            res.levels = synced.levels;+            res.offset = synced.offset;++            // check consistency of obtained data (optimistic locking)+        } while (res.version != getRootVersion());++        // got a consistent snapshot+        return res;+    }++    /**+     * Updates the current root information based on the handed in modified+     * snapshot instance if the version number of the snapshot still corresponds+     * to the current version. Otherwise a concurrent update took place and the+     * operation is aborted.+     *+     * @param info the updated information to be assigned to the active root-info data+     * @return true if successfully updated, false if aborted+     */+    bool tryUpdateRootInfo(const RootInfoSnapshot& info) {+        // check mod counter+        uintptr_t version = info.version;++        // update root to invalid pointer (ending with 1)+        if (!__sync_bool_compare_and_swap(&synced.root, (Node*)version, (Node*)(version + 1))) {+            return false;+        }++        // conduct update+        synced.levels = info.levels;+        synced.offset = info.offset;++        // update root (and thus the version to enable future retrievals)+        __sync_synchronize();+        synced.root = info.root;++        // done+        return true;+    }++    /**+     * A struct summarizing the state of the first node reference.+     */+    struct FirstInfoSnapshot {+        // the pointer to the first node+        Node* node;+        // the offset of the first node+        index_type offset;+        // the version number of the first node (for the optimistic locking)+        uintptr_t version;+    };++    /**+     * Obtains the current version number of the first node information.+     */+    uint64_t getFirstVersion() const {+        // here it is assumed that the load of a 64-bit word is atomic+        return (uint64_t)synced.first;+    }++    /**+     * Obtains a snapshot of the current first-node information.+     */+    FirstInfoSnapshot getFirstInfo() const {+        FirstInfoSnapshot res{};+        do {+            // first take the version+            do {+                res.version = getFirstVersion();+            } while (res.version % 2);++            // collect the values+            res.node = synced.first;+            res.offset = synced.firstOffset;++        } while (res.version != getFirstVersion());++        // we got a consistent snapshot+        return res;+    }++    /**+     * Updates the information stored regarding the first node in a+     * concurrent setting utilizing a optimistic locking approach.+     * This is identical to the approach utilized for the root info.+     */+    bool tryUpdateFirstInfo(const FirstInfoSnapshot& info) {+        // check mod counter+        uintptr_t version = info.version;++        // temporary update first pointer to point to uneven value (lock-out)+        if (!__sync_bool_compare_and_swap(&synced.first, (Node*)version, (Node*)(version + 1))) {+            return false;+        }++        // conduct update+        synced.firstOffset = info.offset;++        // update node pointer (and thus the version number)+        __sync_synchronize();+        synced.first = info.node;  // must be last (and atomic)++        // done+        return true;+    }++public:+    /**+     * Obtains a mutable reference to the value addressed by the given index.+     *+     * @param i the index of the element to be addressed+     * @return a mutable reference to the corresponding element+     */+    value_type& get(index_type i) {+        op_context ctxt;+        return get(i, ctxt);+    }++    /**+     * Obtains a mutable reference to the value addressed by the given index.+     *+     * @param i the index of the element to be addressed+     * @param ctxt a operation context to exploit state-less temporal locality+     * @return a mutable reference to the corresponding element+     */+    value_type& get(index_type i, op_context& ctxt) {+        return getLeaf(i, ctxt).value;+    }++    /**+     * Obtains a mutable reference to the atomic value addressed by the given index.+     *+     * @param i the index of the element to be addressed+     * @return a mutable reference to the corresponding element+     */+    atomic_value_type& getAtomic(index_type i) {+        op_context ctxt;+        return getAtomic(i, ctxt);+    }++    /**+     * Obtains a mutable reference to the atomic value addressed by the given index.+     *+     * @param i the index of the element to be addressed+     * @param ctxt a operation context to exploit state-less temporal locality+     * @return a mutable reference to the corresponding element+     */+    atomic_value_type& getAtomic(index_type i, op_context& ctxt) {+        return getLeaf(i, ctxt).avalue;+    }++private:+    /**+     * An internal function capable of navigating to a given leaf node entry.+     * If the cell does not exist yet it will be created as a side-effect.+     *+     * @param i the index of the requested cell+     * @param ctxt a operation context to exploit state-less temporal locality+     * @return a reference to the requested cell+     */+    inline Cell& getLeaf(index_type i, op_context& ctxt) {+        // check context+        if (ctxt.lastNode && (ctxt.lastIndex == (i & ~INDEX_MASK))) {+            // return reference to referenced+            return ctxt.lastNode->cell[i & INDEX_MASK];+        }++        // get snapshot of root+        auto info = getRootInfo();++        // check for emptiness+        if (info.root == nullptr) {+            // build new root node+            info.root = newNode();++            // initialize the new node+            info.root->parent = nullptr;+            info.offset = i & ~(INDEX_MASK);++            // try updating root information atomically+            if (tryUpdateRootInfo(info)) {+                // success -- finish get call++                // update first+                auto firstInfo = getFirstInfo();+                while (info.offset < firstInfo.offset) {+                    firstInfo.node = info.root;+                    firstInfo.offset = info.offset;+                    if (!tryUpdateFirstInfo(firstInfo)) {+                        // there was some concurrent update => check again+                        firstInfo = getFirstInfo();+                    }+                }++                // return reference to proper cell+                return info.root->cell[i & INDEX_MASK];+            }++            // somebody else was faster => use standard insertion procedure+            delete info.root;++            // retrieve new root info+            info = getRootInfo();++            // make sure there is a root+            assert(info.root);+        }++        // for all other inserts+        //   - check boundary+        //   - navigate to node+        //   - insert value++        // check boundaries+        while (!inBoundaries(i, info.levels, info.offset)) {+            // boundaries need to be expanded by growing upwards+            raiseLevel(info);  // try raising level unless someone else did already+            // update root info+            info = getRootInfo();+        }++        // navigate to node+        Node* node = info.root;+        unsigned level = info.levels;+        while (level != 0) {+            // get X coordinate+            auto x = getIndex(i, level);++            // decrease level counter+            --level;++            // check next node+            std::atomic<Node*>& aNext = node->cell[x].aptr;+            Node* next = aNext;+            if (!next) {+                // create new sub-tree+                Node* newNext = newNode();+                newNext->parent = node;++                // try to update next+                if (!aNext.compare_exchange_strong(next, newNext)) {+                    // some other thread was faster => use updated next+                    delete newNext;+                } else {+                    // the locally created next is the new next+                    next = newNext;++                    // update first+                    if (level == 0) {+                        // compute offset of this node+                        auto off = i & ~INDEX_MASK;++                        // fast over-approximation of whether a update is necessary+                        if (off < unsynced.firstOffset) {+                            // update first reference if this one is the smallest+                            auto first_info = getFirstInfo();+                            while (off < first_info.offset) {+                                first_info.node = next;+                                first_info.offset = off;+                                if (!tryUpdateFirstInfo(first_info)) {+                                    // there was some concurrent update => check again+                                    first_info = getFirstInfo();+                                }+                            }+                        }+                    }+                }++                // now next should be defined+                assert(next);+            }++            // continue one level below+            node = next;+        }++        // update context+        ctxt.lastIndex = (i & ~INDEX_MASK);+        ctxt.lastNode = node;++        // return reference to cell+        return node->cell[i & INDEX_MASK];+    }++public:+    /**+     * Updates the value stored in cell i by the given value.+     */+    void update(index_type i, const value_type& val) {+        op_context ctxt;+        update(i, val, ctxt);+    }++    /**+     * Updates the value stored in cell i by the given value. A operation+     * context can be provided for exploiting temporal locality.+     */+    void update(index_type i, const value_type& val, op_context& ctxt) {+        get(i, ctxt) = val;+    }++    /**+     * Obtains the value associated to index i -- which might be+     * the default value of the covered type if the value hasn't been+     * defined previously.+     */+    value_type operator[](index_type i) const {+        return lookup(i);+    }++    /**+     * Obtains the value associated to index i -- which might be+     * the default value of the covered type if the value hasn't been+     * defined previously.+     */+    value_type lookup(index_type i) const {+        op_context ctxt;+        return lookup(i, ctxt);+    }++    /**+     * Obtains the value associated to index i -- which might be+     * the default value of the covered type if the value hasn't been+     * defined previously. A operation context can be provided for+     * exploiting temporal locality.+     */+    value_type lookup(index_type i, op_context& ctxt) const {+        // check whether it is empty+        if (!unsynced.root) return souffle::detail::default_factory<value_type>()();++        // check boundaries+        if (!inBoundaries(i)) return souffle::detail::default_factory<value_type>()();++        // check context+        if (ctxt.lastNode && ctxt.lastIndex == (i & ~INDEX_MASK)) {+            return ctxt.lastNode->cell[i & INDEX_MASK].value;+        }++        // navigate to value+        Node* node = unsynced.root;+        unsigned level = unsynced.levels;+        while (level != 0) {+            // get X coordinate+            auto x = getIndex(i, level);++            // decrease level counter+            --level;++            // check next node+            Node* next = node->cell[x].ptr;++            // check next step+            if (!next) return souffle::detail::default_factory<value_type>()();++            // continue one level below+            node = next;+        }++        // remember context+        ctxt.lastIndex = (i & ~INDEX_MASK);+        ctxt.lastNode = node;++        // return reference to cell+        return node->cell[i & INDEX_MASK].value;+    }++private:+    /**+     * A static operation utilized internally for merging sub-trees recursively.+     *+     * @param parent the parent node of the current merge operation+     * @param trg a reference to the pointer the cloned node should be stored to+     * @param src the node to be cloned+     * @param levels the height of the cloned node+     */+    static void merge(const Node* parent, Node*& trg, const Node* src, int levels) {+        // if other side is null => done+        if (src == nullptr) {+            return;+        }++        // if the trg sub-tree is empty, clone the corresponding branch+        if (trg == nullptr) {+            trg = clone(src, levels);+            if (trg != nullptr) {+                trg->parent = parent;+            }+            return;  // done+        }++        // otherwise merge recursively++        // the leaf-node step+        if (levels == 0) {+            merge_op merg;+            for (int i = 0; i < NUM_CELLS; ++i) {+                trg->cell[i].value = merg(trg->cell[i].value, src->cell[i].value);+            }+            return;+        }++        // the recursive step+        for (int i = 0; i < NUM_CELLS; ++i) {+            merge(trg, trg->cell[i].ptr, src->cell[i].ptr, levels - 1);+        }+    }++public:+    /**+     * Adds all the values stored in the given array to this array.+     */+    void addAll(const SparseArray& other) {+        // skip if other is empty+        if (other.empty()) {+            return;+        }++        // special case: emptiness+        if (empty()) {+            // use assignment operator+            *this = other;+            return;+        }++        // adjust levels+        while (unsynced.levels < other.unsynced.levels || !inBoundaries(other.unsynced.offset)) {+            raiseLevel();+        }++        // navigate to root node equivalent of the other node in this tree+        auto level = unsynced.levels;+        Node** node = &unsynced.root;+        while (level > other.unsynced.levels) {+            // get X coordinate+            auto x = getIndex(other.unsynced.offset, level);++            // decrease level counter+            --level;++            // check next node+            Node*& next = (*node)->cell[x].ptr;+            if (!next) {+                // create new sub-tree+                next = newNode();+                next->parent = *node;+            }++            // continue one level below+            node = &next;+        }++        // merge sub-branches from here+        merge((*node)->parent, *node, other.unsynced.root, level);++        // update first+        if (unsynced.firstOffset > other.unsynced.firstOffset) {+            unsynced.first = findFirst(*node, level);+            unsynced.firstOffset = other.unsynced.firstOffset;+        }+    }++    // ---------------------------------------------------------------------+    //                           Iterator+    // ---------------------------------------------------------------------++    /**+     * The iterator type to be utilized to iterate over the non-default elements of this array.+     */+    class iterator {+        using pair_type = std::pair<index_type, value_type>;++        // a pointer to the leaf node currently processed or null (end)+        const Node* node;++        // the value currently pointed to+        pair_type value;++    public:+        // default constructor -- creating an end-iterator+        iterator() : node(nullptr) {}++        iterator(const Node* node, pair_type value) : node(node), value(std::move(value)) {}++        iterator(const Node* first, index_type firstOffset) : node(first), value(firstOffset, 0) {+            // if the start is the end => we are done+            if (!first) return;++            // load the value+            if (first->cell[0].value == value_type()) {+                ++(*this);  // walk to first element+            } else {+                value.second = first->cell[0].value;+            }+        }++        // a copy constructor+        iterator(const iterator& other) = default;++        // an assignment operator+        iterator& operator=(const iterator& other) = default;++        // the equality operator as required by the iterator concept+        bool operator==(const iterator& other) const {+            // only equivalent if pointing to the end+            return (node == nullptr && other.node == nullptr) ||+                   (node == other.node && value.first == other.value.first);+        }++        // the not-equality operator as required by the iterator concept+        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        // the deref operator as required by the iterator concept+        const pair_type& operator*() const {+            return value;+        }++        // support for the pointer operator+        const pair_type* operator->() const {+            return &value;+        }++        // the increment operator as required by the iterator concept+        iterator& operator++() {+            // get current offset+            index_type x = value.first & INDEX_MASK;++            // go to next non-empty value in current node+            do {+                x++;+            } while (x < NUM_CELLS && node->cell[x].value == value_type());++            // check whether one has been found+            if (x < NUM_CELLS) {+                // update value and be done+                value.first = (value.first & ~INDEX_MASK) | x;+                value.second = node->cell[x].value;+                return *this;  // done+            }++            // go to parent+            node = node->parent;+            int level = 1;++            // get current index on this level+            x = getIndex(value.first, level);+            x++;++            while (level > 0 && node) {+                // search for next child+                while (x < NUM_CELLS) {+                    if (node->cell[x].ptr != nullptr) {+                        break;+                    }+                    x++;+                }++                // pick next step+                if (x < NUM_CELLS) {+                    // going down+                    node = node->cell[x].ptr;+                    value.first &= getLevelMask(level + 1);+                    value.first |= x << (BIT_PER_STEP * level);+                    level--;+                    x = 0;+                } else {+                    // going up+                    node = node->parent;+                    level++;++                    // get current index on this level+                    x = getIndex(value.first, level);+                    x++;  // go one step further+                }+            }++            // check whether it is the end of range+            if (node == nullptr) {+                return *this;+            }++            // search the first value in this node+            x = 0;+            while (node->cell[x].value == value_type()) {+                x++;+            }++            // update value+            value.first |= x;+            value.second = node->cell[x].value;++            // done+            return *this;+        }++        // True if this iterator is passed the last element.+        bool isEnd() const {+            return node == nullptr;+        }++        // enables this iterator core to be printed (for debugging)+        void print(std::ostream& out) const {+            out << "SparseArrayIter(" << node << " @ " << value << ")";+        }++        friend std::ostream& operator<<(std::ostream& out, const iterator& iter) {+            iter.print(out);+            return out;+        }+    };++    /**+     * Obtains an iterator referencing the first non-default element or end in+     * case there are no such elements.+     */+    iterator begin() const {+        return iterator(unsynced.first, unsynced.firstOffset);+    }++    /**+     * An iterator referencing the position after the last non-default element.+     */+    iterator end() const {+        return iterator();+    }++    /**+     * An operation to obtain an iterator referencing an element addressed by the+     * given index. If the corresponding element is a non-default value, a corresponding+     * iterator will be returned. Otherwise end() will be returned.+     */+    iterator find(index_type i) const {+        op_context ctxt;+        return find(i, ctxt);+    }++    /**+     * An operation to obtain an iterator referencing an element addressed by the+     * given index. If the corresponding element is a non-default value, a corresponding+     * iterator will be returned. Otherwise end() will be returned. A operation context+     * can be provided for exploiting temporal locality.+     */+    iterator find(index_type i, op_context& ctxt) const {+        // check whether it is empty+        if (!unsynced.root) return end();++        // check boundaries+        if (!inBoundaries(i)) return end();++        // check context+        if (ctxt.lastNode && ctxt.lastIndex == (i & ~INDEX_MASK)) {+            Node* node = ctxt.lastNode;++            // check whether there is a proper entry+            value_type value = node->cell[i & INDEX_MASK].value;+            if (value == value_type{}) {+                return end();+            }+            // return iterator pointing to value+            return iterator(node, std::make_pair(i, value));+        }++        // navigate to value+        Node* node = unsynced.root;+        unsigned level = unsynced.levels;+        while (level != 0) {+            // get X coordinate+            auto x = getIndex(i, level);++            // decrease level counter+            --level;++            // check next node+            Node* next = node->cell[x].ptr;++            // check next step+            if (!next) return end();++            // continue one level below+            node = next;+        }++        // register in context+        ctxt.lastNode = node;+        ctxt.lastIndex = (i & ~INDEX_MASK);++        // check whether there is a proper entry+        value_type value = node->cell[i & INDEX_MASK].value;+        if (value == value_type{}) {+            return end();+        }++        // return iterator pointing to cell+        return iterator(node, std::make_pair(i, value));+    }++    /**+     * An operation obtaining the smallest non-default element such that it's index is >=+     * the given index.+     */+    iterator lowerBound(index_type i) const {+        op_context ctxt;+        return lowerBound(i, ctxt);+    }++    /**+     * An operation obtaining the smallest non-default element such that it's index is >=+     * the given index. A operation context can be provided for exploiting temporal locality.+     */+    iterator lowerBound(index_type i, op_context&) const {+        // check whether it is empty+        if (!unsynced.root) return end();++        // check boundaries+        if (!inBoundaries(i)) {+            // if it is on the lower end, return minimum result+            if (i < unsynced.offset) {+                const auto& value = unsynced.first->cell[0].value;+                auto res = iterator(unsynced.first, std::make_pair(unsynced.offset, value));+                if (value == value_type()) {+                    ++res;+                }+                return res;+            }+            // otherwise it is on the high end, return end iterator+            return end();+        }++        // navigate to value+        Node* node = unsynced.root;+        unsigned level = unsynced.levels;+        while (true) {+            // get X coordinate+            auto x = getIndex(i, level);++            // check next node+            Node* next = node->cell[x].ptr;++            // check next step+            if (!next) {+                if (x == NUM_CELLS - 1) {+                    ++level;+                    node = const_cast<Node*>(node->parent);+                    if (!node) return end();+                }++                // continue search+                i = i & getLevelMask(level);++                // find next higher value+                i += 1ull << (BITS * level);++            } else {+                if (level == 0) {+                    // found boundary+                    return iterator(node, std::make_pair(i, node->cell[x].value));+                }++                // decrease level counter+                --level;++                // continue one level below+                node = next;+            }+        }+    }++    /**+     * An operation obtaining the smallest non-default element such that it's index is greater+     * the given index.+     */+    iterator upperBound(index_type i) const {+        op_context ctxt;+        return upperBound(i, ctxt);+    }++    /**+     * An operation obtaining the smallest non-default element such that it's index is greater+     * the given index. A operation context can be provided for exploiting temporal locality.+     */+    iterator upperBound(index_type i, op_context& ctxt) const {+        if (i == std::numeric_limits<index_type>::max()) {+            return end();+        }+        return lowerBound(i + 1, ctxt);+    }++private:+    /**+     * An internal debug utility printing the internal structure of this sparse array to the given output+     * stream.+     */+    void dump(bool detailed, std::ostream& out, const Node& node, int level, index_type offset,+            int indent = 0) const {+        auto x = getIndex(offset, level + 1);+        out << times("\t", indent) << x << ": Node " << &node << " on level " << level+            << " parent: " << node.parent << " -- range: " << offset << " - "+            << (offset + ~getLevelMask(level + 1)) << "\n";++        if (level == 0) {+            for (int i = 0; i < NUM_CELLS; i++) {+                if (detailed || node.cell[i].value != value_type()) {+                    out << times("\t", indent + 1) << i << ": [" << (offset + i) << "] " << node.cell[i].value+                        << "\n";+                }+            }+        } else {+            for (int i = 0; i < NUM_CELLS; i++) {+                if (node.cell[i].ptr) {+                    dump(detailed, out, *node.cell[i].ptr, level - 1,+                            offset + (i * (index_type(1) << (level * BIT_PER_STEP))), indent + 1);+                } else if (detailed) {+                    auto low = offset + (i * (1 << (level * BIT_PER_STEP)));+                    auto hig = low + ~getLevelMask(level);+                    out << times("\t", indent + 1) << i << ": empty range " << low << " - " << hig << "\n";+                }+            }+        }+        out << "\n";+    }++public:+    /**+     * A debug utility printing the internal structure of this sparse array to the given output stream.+     */+    void dump(bool detail = false, std::ostream& out = std::cout) const {+        if (!unsynced.root) {+            out << " - empty - \n";+            return;+        }+        out << "root:  " << unsynced.root << "\n";+        out << "offset: " << unsynced.offset << "\n";+        out << "first: " << unsynced.first << "\n";+        out << "fist offset: " << unsynced.firstOffset << "\n";+        dump(detail, out, *unsynced.root, unsynced.levels, unsynced.offset);+    }++private:+    // --------------------------------------------------------------------------+    //                                 Utilities+    // --------------------------------------------------------------------------++    /**+     * Creates new nodes and initializes them with 0.+     */+    static Node* newNode() {+        auto* res = new Node();+        std::memset(res->cell, 0, sizeof(Cell) * NUM_CELLS);+        return res;+    }++    /**+     * Destroys a node and all its sub-nodes recursively.+     */+    static void freeNodes(Node* node, int level) {+        if (!node) return;+        if (level != 0) {+            for (int i = 0; i < NUM_CELLS; i++) {+                freeNodes(node->cell[i].ptr, level - 1);+            }+        }+        delete node;+    }++    /**+     * Conducts a cleanup of the internal tree structure.+     */+    void clean() {+        freeNodes(unsynced.root, unsynced.levels);+        unsynced.root = nullptr;+        unsynced.levels = 0;+    }++    /**+     * Clones the given node and all its sub-nodes.+     */+    static Node* clone(const Node* node, int level) {+        // support null-pointers+        if (node == nullptr) {+            return nullptr;+        }++        // create a clone+        auto* res = new Node();++        // handle leaf level+        if (level == 0) {+            copy_op copy;+            for (int i = 0; i < NUM_CELLS; i++) {+                res->cell[i].value = copy(node->cell[i].value);+            }+            return res;+        }++        // for inner nodes clone each child+        for (int i = 0; i < NUM_CELLS; i++) {+            auto cur = clone(node->cell[i].ptr, level - 1);+            if (cur != nullptr) {+                cur->parent = res;+            }+            res->cell[i].ptr = cur;+        }++        // done+        return res;+    }++    /**+     * Obtains the left-most leaf-node of the tree rooted by the given node+     * with the given level.+     */+    static Node* findFirst(Node* node, int level) {+        while (level > 0) {+            bool found = false;+            for (int i = 0; i < NUM_CELLS; i++) {+                Node* cur = node->cell[i].ptr;+                if (cur) {+                    node = cur;+                    --level;+                    found = true;+                    break;+                }+            }+            assert(found && "No first node!");+        }++        return node;+    }++    /**+     * Raises the level of this tree by one level. It does so by introducing+     * a new root node and inserting the current root node as a child node.+     */+    void raiseLevel() {+        // something went wrong when we pass that line+        assert(unsynced.levels < (sizeof(index_type) * 8 / BITS) + 1);++        // create new root+        Node* node = newNode();+        node->parent = nullptr;++        // insert existing root as child+        auto x = getIndex(unsynced.offset, unsynced.levels + 1);+        node->cell[x].ptr = unsynced.root;++        // swap the root+        unsynced.root->parent = node;++        // update root+        unsynced.root = node;+        ++unsynced.levels;++        // update offset be removing additional bits+        unsynced.offset &= getLevelMask(unsynced.levels + 1);+    }++    /**+     * Attempts to raise the height of this tree based on the given root node+     * information and updates the root-info snapshot correspondingly.+     */+    void raiseLevel(RootInfoSnapshot& info) {+        // something went wrong when we pass that line+        assert(info.levels < (sizeof(index_type) * 8 / BITS) + 1);++        // create new root+        Node* newRoot = newNode();+        newRoot->parent = nullptr;++        // insert existing root as child+        auto x = getIndex(info.offset, info.levels + 1);+        newRoot->cell[x].ptr = info.root;++        // exchange the root in the info struct+        auto oldRoot = info.root;+        info.root = newRoot;++        // update level counter+        ++info.levels;++        // update offset+        info.offset &= getLevelMask(info.levels + 1);++        // try exchanging root info+        if (tryUpdateRootInfo(info)) {+            // success => final step, update parent of old root+            oldRoot->parent = info.root;+        } else {+            // throw away temporary new node+            delete newRoot;+        }+    }++    /**+     * Tests whether the given index is covered by the boundaries defined+     * by the hight and offset of the internally maintained tree.+     */+    bool inBoundaries(index_type a) const {+        return inBoundaries(a, unsynced.levels, unsynced.offset);+    }++    /**+     * Tests whether the given index is within the boundaries defined by the+     * given tree hight and offset.+     */+    static bool inBoundaries(index_type a, uint32_t levels, index_type offset) {+        auto mask = getLevelMask(levels + 1);+        return (a & mask) == offset;+    }++    /**+     * Obtains the index within the arrays of cells of a given index on a given+     * level of the internally maintained tree.+     */+    static index_type getIndex(RamDomain a, unsigned level) {+        return (a & (INDEX_MASK << (level * BIT_PER_STEP))) >> (level * BIT_PER_STEP);+    }++    /**+     * Computes the bit-mask to be applicable to obtain the offset of a node on a+     * given tree level.+     */+    static index_type getLevelMask(unsigned level) {+        if (level > (sizeof(index_type) * 8 / BITS)) return 0;+        return (~(index_type(0)) << (level * BIT_PER_STEP));+    }+};++/**+ * A sparse bit-map is a bit map virtually assigning a bit value to every value if the+ * uint32_t domain. However, only 1-bits are stored utilizing a nested sparse array+ * structure.+ *+ * @tparam BITS similar to the BITS parameter of the sparse array type+ */+template <unsigned BITS = 4>+class SparseBitMap {+    // the element type stored in the nested sparse array+    using value_t = uint64_t;++    // define the bit-level merge operation+    struct merge_op {+        value_t operator()(value_t a, value_t b) const {+            return a | b;  // merging bit masks => bitwise or operation+        }+    };++    // the type of the internal data store+    using data_store_t = SparseArray<value_t, BITS, merge_op>;+    using atomic_value_t = typename data_store_t::atomic_value_type;++    // some constants for manipulating stored values+    static constexpr short BITS_PER_ENTRY = sizeof(value_t) * 8;+    static constexpr short LEAF_INDEX_WIDTH = static_cast<short>(__builtin_ctz(BITS_PER_ENTRY));+    static constexpr uint64_t LEAF_INDEX_MASK = BITS_PER_ENTRY - 1;++public:+    // the type to address individual entries+    using index_type = typename data_store_t::index_type;++private:+    // it utilizes a sparse map to store its data+    data_store_t store;++public:+    // a simple default constructor+    SparseBitMap() = default;++    // a default copy constructor+    SparseBitMap(const SparseBitMap&) = default;++    // a default r-value copy constructor+    SparseBitMap(SparseBitMap&&) = default;++    // a default assignment operator+    SparseBitMap& operator=(const SparseBitMap&) = default;++    // a default r-value assignment operator+    SparseBitMap& operator=(SparseBitMap&&) = default;++    // checks whether this bit-map is empty -- thus it does not have any 1-entries+    bool empty() const {+        return store.empty();+    }++    // the type utilized for recording context information for exploiting temporal locality+    using op_context = typename data_store_t::op_context;++    /**+     * Sets the bit addressed by i to 1.+     */+    bool set(index_type i) {+        op_context ctxt;+        return set(i, ctxt);+    }++    /**+     * Sets the bit addressed by i to 1. A context for exploiting temporal locality+     * can be provided.+     */+    bool set(index_type i, op_context& ctxt) {+        atomic_value_t& val = store.getAtomic(i >> LEAF_INDEX_WIDTH, ctxt);+        value_t bit = (1ull << (i & LEAF_INDEX_MASK));++#ifdef __GNUC__+#if __GNUC__ >= 7+        // In GCC >= 7 the usage of fetch_or causes a bug that needs further investigation+        // For now, this two-instruction based implementation provides a fix that does+        // not sacrifice too much performance.++        while (true) {+            auto order = std::memory_order::memory_order_relaxed;++            // load current value+            value_t old = val.load(order);++            // if bit is already set => we are done+            if (old & bit) return false;++            // set the bit, if failed, repeat+            if (!val.compare_exchange_strong(old, old | bit, order, order)) continue;++            // it worked, new bit added+            return true;+        }++#endif+#endif++        value_t old = val.fetch_or(bit, std::memory_order::memory_order_relaxed);+        return (old & bit) == 0u;+    }++    /**+     * Determines the whether the bit addressed by i is set or not.+     */+    bool test(index_type i) const {+        op_context ctxt;+        return test(i, ctxt);+    }++    /**+     * Determines the whether the bit addressed by i is set or not. A context for+     * exploiting temporal locality can be provided.+     */+    bool test(index_type i, op_context& ctxt) const {+        value_t bit = (1ull << (i & LEAF_INDEX_MASK));+        return store.lookup(i >> LEAF_INDEX_WIDTH, ctxt) & bit;+    }++    /**+     * Determines the whether the bit addressed by i is set or not.+     */+    bool operator[](index_type i) const {+        return test(i);+    }++    /**+     * Resets all contained bits to 0.+     */+    void clear() {+        store.clear();+    }++    /**+     * Determines the number of bits set.+     */+    std::size_t size() const {+        // this is computed on demand to keep the set operation simple.+        std::size_t res = 0;+        for (const auto& cur : store) {+            res += __builtin_popcountll(cur.second);+        }+        return res;+    }++    /**+     * Computes the total memory usage of this data structure.+     */+    std::size_t getMemoryUsage() const {+        // compute the total memory usage+        return sizeof(*this) - sizeof(data_store_t) + store.getMemoryUsage();+    }++    /**+     * Sets all bits set in other to 1 within this bit map.+     */+    void addAll(const SparseBitMap& other) {+        // nothing to do if it is a self-assignment+        if (this == &other) return;++        // merge the sparse store+        store.addAll(other.store);+    }++    // ---------------------------------------------------------------------+    //                           Iterator+    // ---------------------------------------------------------------------++    /**+     * An iterator iterating over all indices set to 1.+     */+    class iterator : public std::iterator<std::forward_iterator_tag, index_type> {+        using nested_iterator = typename data_store_t::iterator;++        // the iterator through the underlying sparse data structure+        nested_iterator iter;++        // the currently consumed mask+        uint64_t mask = 0;++        // the value currently pointed to+        index_type value{};++    public:+        // default constructor -- creating an end-iterator+        iterator() = default;++        iterator(const nested_iterator& iter)+                : iter(iter), mask(toMask(iter->second)), value(iter->first << LEAF_INDEX_WIDTH) {+            moveToNextInMask();+        }++        iterator(const nested_iterator& iter, uint64_t m, index_type value)+                : iter(iter), mask(m), value(value) {}++        // a copy constructor+        iterator(const iterator& other) = default;++        // an assignment operator+        iterator& operator=(const iterator& other) = default;++        // the equality operator as required by the iterator concept+        bool operator==(const iterator& other) const {+            // only equivalent if pointing to the end+            return iter == other.iter && mask == other.mask;+        }++        // the not-equality operator as required by the iterator concept+        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        // the deref operator as required by the iterator concept+        const index_type& operator*() const {+            return value;+        }++        // support for the pointer operator+        const index_type* operator->() const {+            return &value;+        }++        // the increment operator as required by the iterator concept+        iterator& operator++() {+            // progress in current mask+            if (moveToNextInMask()) return *this;++            // go to next entry+            ++iter;++            // update value+            if (!iter.isEnd()) {+                value = iter->first << LEAF_INDEX_WIDTH;+                mask = toMask(iter->second);+                moveToNextInMask();+            }++            // done+            return *this;+        }++        bool isEnd() const {+            return iter.isEnd();+        }++        void print(std::ostream& out) const {+            out << "SparseBitMapIter(" << iter << " -> " << std::bitset<64>(mask) << " @ " << value << ")";+        }++        // enables this iterator core to be printed (for debugging)+        friend std::ostream& operator<<(std::ostream& out, const iterator& iter) {+            iter.print(out);+            return out;+        }++        static uint64_t toMask(const value_t& value) {+            static_assert(sizeof(value_t) == sizeof(uint64_t), "Fixed for 64-bit compiler.");+            return reinterpret_cast<const uint64_t&>(value);+        }++    private:+        bool moveToNextInMask() {+            // check if there is something left+            if (mask == 0) return false;++            // get position of leading 1+            auto pos = __builtin_ctzll(mask);++            // consume this bit+            mask &= ~(1llu << pos);++            // update value+            value &= ~LEAF_INDEX_MASK;+            value |= pos;++            // done+            return true;+        }+    };++    /**+     * Obtains an iterator pointing to the first index set to 1. If there+     * is no such bit, end() will be returned.+     */+    iterator begin() const {+        auto it = store.begin();+        if (it.isEnd()) return end();+        return iterator(it);+    }++    /**+     * Returns an iterator referencing the position after the last set bit.+     */+    iterator end() const {+        return iterator();+    }++    /**+     * Obtains an iterator referencing the position i if the corresponding+     * bit is set, end() otherwise.+     */+    iterator find(index_type i) const {+        op_context ctxt;+        return find(i, ctxt);+    }++    /**+     * Obtains an iterator referencing the position i if the corresponding+     * bit is set, end() otherwise. An operation context can be provided+     * to exploit temporal locality.+     */+    iterator find(index_type i, op_context& ctxt) const {+        // check prefix part+        auto it = store.find(i >> LEAF_INDEX_WIDTH, ctxt);+        if (it.isEnd()) return end();++        // check bit-set part+        uint64_t mask = iterator::toMask(it->second);+        if (!(mask & (1llu << (i & LEAF_INDEX_MASK)))) return end();++        // OK, it is there => create iterator+        mask &= ((1ull << (i & LEAF_INDEX_MASK)) - 1);  // remove all bits before pos i+        return iterator(it, mask, i);+    }++    /**+     * Locates an iterator to the first element in this sparse bit map not less+     * than the given index.+     */+    iterator lower_bound(index_type i) const {+        auto it = store.lowerBound(i >> LEAF_INDEX_WIDTH);+        if (it.isEnd()) return end();++        // check bit-set part+        uint64_t mask = iterator::toMask(it->second);++        // if there is no bit remaining in this mask, check next mask.+        if (!(mask & ((~uint64_t(0)) << (i & LEAF_INDEX_MASK)))) {+            index_type next = ((i >> LEAF_INDEX_WIDTH) + 1) << LEAF_INDEX_WIDTH;+            if (next < i) return end();+            return lower_bound(next);+        }++        // there are bits left, use least significant bit of those+        if (it->first == i >> LEAF_INDEX_WIDTH) {+            mask &= ((~uint64_t(0)) << (i & LEAF_INDEX_MASK));  // remove all bits before pos i+        }++        // compute value represented by least significant bit+        index_type pos = __builtin_ctzll(mask);++        // remove this bit as well+        mask = mask & ~(1ull << pos);++        // construct value of this located bit+        index_type val = (it->first << LEAF_INDEX_WIDTH) | pos;+        return iterator(it, mask, val);+    }++    /**+     * Locates an iterator to the first element in this sparse bit map than is greater+     * than the given index.+     */+    iterator upper_bound(index_type i) const {+        if (i == std::numeric_limits<index_type>::max()) {+            return end();+        }+        return lower_bound(i + 1);+    }++    /**+     * A debugging utility printing the internal structure of this map to the+     * given output stream.+     */+    void dump(bool detail = false, std::ostream& out = std::cout) const {+        store.dump(detail, out);+    }++    /**+     * Provides write-protected access to the internal store for running+     * analysis on the data structure.+     */+    const data_store_t& getStore() const {+        return store;+    }+};++// ---------------------------------------------------------------------+//                              TRIE+// ---------------------------------------------------------------------++namespace detail {++/**+ * A base class for the Trie implementation allowing various+ * specializations of the Trie template to inherit common functionality.+ *+ * @tparam Dim the number of dimensions / arity of the stored tuples+ * @tparam Derived the type derived from this base class+ */+template <unsigned Dim, typename Derived>+class TrieBase {+public:+    /**+     * The type of the stored entries / tuples.+     */+    using entry_type = typename souffle::Tuple<RamDomain, Dim>;++    // -- operation wrappers --++    /**+     * A generic function enabling the insertion of tuple values in a user-friendly way.+     */+    template <typename... Values>+    bool insert(Values... values) {+        return static_cast<Derived&>(*this).insert(entry_type{{RamDomain(values)...}});+    }++    /**+     * A generic function enabling the convenient conduction of a membership check.+     */+    template <typename... Values>+    bool contains(Values... values) const {+        return static_cast<const Derived&>(*this).contains(entry_type{{RamDomain(values)...}});+    }++    // ---------------------------------------------------------------------+    //                           Iterator+    // ---------------------------------------------------------------------++    /**+     * An iterator over the stored entries.+     *+     * Iterators for tries consist of a top-level iterator maintaining the+     * master copy of a materialized tuple and a recursively nested iterator+     * core -- one for each nested trie level.+     */+    template <template <unsigned D> class IterCore>+    class iterator : public std::iterator<std::forward_iterator_tag, entry_type> {+        template <unsigned Len, unsigned Pos, unsigned Dimensions>+        friend struct fix_binding;++        template <unsigned Pos, unsigned Dimensions>+        friend struct fix_lower_bound;++        template <unsigned Pos, unsigned Dimensions>+        friend struct fix_upper_bound;++        template <unsigned Pos, unsigned Dimensions>+        friend struct fix_first;++        // the iterator core of this level+        using iter_core_t = IterCore<0>;++        // the wrapped iterator+        iter_core_t iter_core;++        // the value currently pointed to+        entry_type value;++    public:+        // default constructor -- creating an end-iterator+        iterator() = default;++        // a copy constructor+        iterator(const iterator& other) = default;++        iterator(iterator&& other) = default;++        template <typename Param>+        explicit iterator(const Param& param) : iter_core(param, value) {}++        // an assignment operator+        iterator& operator=(const iterator& other) = default;++        // the equality operator as required by the iterator concept+        bool operator==(const iterator& other) const {+            // equivalent if pointing to the same value+            return iter_core == other.iter_core;+        }++        // the not-equality operator as required by the iterator concept+        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        // the deref operator as required by the iterator concept+        const entry_type& operator*() const {+            return value;+        }++        // support for the pointer operator+        const entry_type* operator->() const {+            return &value;+        }++        // the increment operator as required by the iterator concept+        iterator& operator++() {+            iter_core.inc(value);+            return *this;+        }++        // enables this iterator to be printed (for debugging)+        void print(std::ostream& out) const {+            out << "iter(" << iter_core << " -> " << value << ")";+        }++        friend std::ostream& operator<<(std::ostream& out, const iterator& iter) {+            iter.print(out);+            return out;+        }+    };++    /* -------------- operator hint statistics ----------------- */++    // an aggregation of statistical values of the hint utilization+    struct hint_statistics {+        // the counter for insertion operations+        CacheAccessCounter inserts;++        // the counter for contains operations+        CacheAccessCounter contains;++        // the counter for get_boundaries operations+        CacheAccessCounter get_boundaries;+    };++protected:+    // the hint statistic of this b-tree instance+    mutable hint_statistics hint_stats;++public:+    void printStats(std::ostream& out) const {+        out << "---------------------------------\n";+        out << "  insert-hint (hits/misses/total): " << hint_stats.inserts.getHits() << "/"+            << hint_stats.inserts.getMisses() << "/" << hint_stats.inserts.getAccesses() << "\n";+        out << "  contains-hint (hits/misses/total):" << hint_stats.contains.getHits() << "/"+            << hint_stats.contains.getMisses() << "/" << hint_stats.contains.getAccesses() << "\n";+        out << "  get-boundaries-hint (hits/misses/total):" << hint_stats.get_boundaries.getHits() << "/"+            << hint_stats.get_boundaries.getMisses() << "/" << hint_stats.get_boundaries.getAccesses()+            << "\n";+        out << "---------------------------------\n";+    }+};++/**+ * A functor extracting a reference to a nested iterator core from an enclosing+ * iterator core.+ */+template <unsigned Level>+struct get_nested_iter_core {+    template <typename IterCore>+    auto operator()(IterCore& core) -> decltype(get_nested_iter_core<Level - 1>()(core.getNested())) {+        return get_nested_iter_core<Level - 1>()(core.getNested());+    }+};++template <>+struct get_nested_iter_core<0> {+    template <typename IterCore>+    IterCore& operator()(IterCore& core) {+        return core;+    }+};++/**+ * A functor initializing an iterator upon creation to reference the first+ * element in the associated Trie.+ */+template <unsigned Pos, unsigned Dim>+struct fix_first {+    template <unsigned bits, typename iterator>+    void operator()(const SparseBitMap<bits>& store, iterator& iter) const {+        // set iterator to first in store+        auto first = store.begin();+        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(first);+        iter.value[Pos] = *first;+    }++    template <typename Store, typename iterator>+    void operator()(const Store& store, iterator& iter) const {+        // set iterator to first in store+        auto first = store.begin();+        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(first);+        iter.value[Pos] = first->first;+        // and continue recursively+        fix_first<Pos + 1, Dim>()(first->second->getStore(), iter);+    }+};++template <unsigned Dim>+struct fix_first<Dim, Dim> {+    template <typename Store, typename iterator>+    void operator()(const Store&, iterator&) const {+        // terminal case => nothing to do+    }+};++/**+ * A functor initializing an iterator upon creation to reference the first element+ * exhibiting a given prefix within a given Trie.+ */+template <unsigned Len, unsigned Pos, unsigned Dim>+struct fix_binding {+    template <unsigned bits, typename iterator, typename entry_type>+    bool operator()(+            const SparseBitMap<bits>& store, iterator& begin, iterator& end, const entry_type& entry) const {+        // search in current level+        auto cur = store.find(entry[Pos]);++        // if not present => fail+        if (cur == store.end()) return false;++        // take current value+        get_nested_iter_core<Pos>()(begin.iter_core).setIterator(cur);+        ++cur;+        get_nested_iter_core<Pos>()(end.iter_core).setIterator(cur);++        // update iterator value+        begin.value[Pos] = entry[Pos];++        // no more remaining levels to fix+        return true;+    }++    template <typename Store, typename iterator, typename entry_type>+    bool operator()(const Store& store, iterator& begin, iterator& end, const entry_type& entry) const {+        // search in current level+        auto cur = store.find(entry[Pos]);++        // if not present => fail+        if (cur == store.end()) return false;++        // take current value as start+        get_nested_iter_core<Pos>()(begin.iter_core).setIterator(cur);++        // update iterator value+        begin.value[Pos] = entry[Pos];++        // fix remaining nested iterators+        auto res = fix_binding<Len - 1, Pos + 1, Dim>()(cur->second->getStore(), begin, end, entry);++        // update end of iterator+        if (get_nested_iter_core<Pos + 1>()(end.iter_core).getIterator() == cur->second->getStore().end()) {+            ++cur;+            if (cur != store.end()) {+                fix_first<Pos + 1, Dim>()(cur->second->getStore(), end);+            }+        }+        get_nested_iter_core<Pos>()(end.iter_core).setIterator(cur);++        // done+        return res;+    }+};++template <unsigned Pos, unsigned Dim>+struct fix_binding<0, Pos, Dim> {+    template <unsigned bits, typename iterator, typename entry_type>+    bool operator()(const SparseBitMap<bits>& store, iterator& begin, iterator& /* end */,+            const entry_type& /* entry */) const {+        // move begin to begin of store+        auto a = store.begin();+        get_nested_iter_core<Pos>()(begin.iter_core).setIterator(a);+        begin.value[Pos] = *a;++        return true;+    }++    template <typename Store, typename iterator, typename entry_type>+    bool operator()(const Store& store, iterator& begin, iterator& end, const entry_type& entry) const {+        // move begin to begin of store+        auto a = store.begin();+        get_nested_iter_core<Pos>()(begin.iter_core).setIterator(a);+        begin.value[Pos] = a->first;++        // continue recursively+        fix_binding<0, Pos + 1, Dim>()(a->second->getStore(), begin, end, entry);+        return true;+    }+};++template <unsigned Dim>+struct fix_binding<0, Dim, Dim> {+    template <typename Store, typename iterator, typename entry_type>+    bool operator()(const Store& /* store */, iterator& /* begin */, iterator& /* end */,+            const entry_type& /* entry */) const {+        // nothing more to do+        return true;+    }+};++/**+ * A functor initializing an iterator upon creation to reference the first element+ * within a given Trie being not less than a given value .+ */+template <unsigned Pos, unsigned Dim>+struct fix_lower_bound {+    template <unsigned bits, typename iterator, typename entry_type>+    bool operator()(const SparseBitMap<bits>& store, iterator& iter, const entry_type& entry) const {+        // search in current level+        auto cur = store.lower_bound(entry[Pos]);++        if (cur == store.end()) return false;++        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);++        assert(entry[Pos] <= RamDomain(*cur));+        iter.value[Pos] = *cur;++        // no more remaining levels to fix+        return true;+    }++    template <typename Store, typename iterator, typename entry_type>+    bool operator()(const Store& store, iterator& iter, const entry_type& entry) const {+        // search in current level+        auto cur = store.lowerBound(entry[Pos]);++        // if no lower boundary is found, be done+        if (cur == store.end()) return false;+        assert(RamDomain(cur->first) >= entry[Pos]);++        // if the lower bound is higher than the requested value, go to first in subtree+        if (RamDomain(cur->first) > entry[Pos]) {+            get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);+            iter.value[Pos] = cur->first;+            fix_first<Pos + 1, Dim>()(cur->second->getStore(), iter);+            return true;+        }++        // attempt to fix the rest+        if (!fix_lower_bound<Pos + 1, Dim>()(cur->second->getStore(), iter, entry)) {+            // if it does not work, since there are no matching elements in this branch, go to next+            entry_type sub = entry;+            sub[Pos] += 1;+            for (size_t i = Pos + 1; i < Dim; ++i) {+                sub[i] = 0;+            }+            return (*this)(store, iter, sub);+        }++        // remember result+        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);++        // update iterator value+        iter.value[Pos] = cur->first;++        // done!+        return true;+    }+};++/**+ * A functor initializing an iterator upon creation to reference the first element+ * within a given Trie being greater than a given value .+ */+template <unsigned Pos, unsigned Dim>+struct fix_upper_bound {+    template <unsigned bits, typename iterator, typename entry_type>+    bool operator()(const SparseBitMap<bits>& store, iterator& iter, const entry_type& entry) const {+        // search in current level+        auto cur = store.upper_bound(entry[Pos]);++        if (cur == store.end()) {+            return false;+        }++        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);++        assert(entry[Pos] <= RamDomain(*cur));+        iter.value[Pos] = *cur;++        // no more remaining levels to fix+        return true;+    }++    template <typename Store, typename iterator, typename entry_type>+    bool operator()(const Store& store, iterator& iter, const entry_type& entry) const {+        // search in current level (if it is not the last level, we need a lower bound)+        auto cur = store.lowerBound(entry[Pos]);++        // if no lower boundary is found, be done+        if (cur == store.end()) {+            return false;+        }+        assert(RamDomain(cur->first) >= entry[Pos]);++        // if the lower bound is higher than the requested value, go to first in subtree+        if (RamDomain(cur->first) > entry[Pos]) {+            get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);+            iter.value[Pos] = cur->first;+            fix_first<Pos + 1, Dim>()(cur->second->getStore(), iter);+            return true;+        }++        // attempt to fix the rest+        if (!fix_upper_bound<Pos + 1, Dim>()(cur->second->getStore(), iter, entry)) {+            // if it does not work, since there are no matching elements in this branch, go to next+            entry_type sub = entry;+            sub[Pos] += 1;+            for (size_t i = Pos + 1; i < Dim; ++i) {+                sub[i] = 0;+            }+            return (*this)(store, iter, sub);+        }++        // remember result+        get_nested_iter_core<Pos>()(iter.iter_core).setIterator(cur);++        // update iterator value+        iter.value[Pos] = cur->first;++        // done!+        return true;+    }+};++}  // namespace detail++/**+ * The most generic implementation of a Trie forming the top-level of any+ * Trie storing tuples of arity > 1.+ */+template <unsigned Dim>+class Trie : public detail::TrieBase<Dim, Trie<Dim>> {+    template <unsigned D>+    friend class Trie;++    template <unsigned D, typename Derived>+    friend class TrieBase;++    // a shortcut for the common base class type+    using base = typename detail::TrieBase<Dim, Trie<Dim>>;++    // the type of the nested tries (1 dimension less)+    using nested_trie_type = Trie<Dim - 1>;++    // the merge operation capable of merging two nested tries+    struct nested_trie_merger {+        nested_trie_type* operator()(nested_trie_type* a, const nested_trie_type* b) const {+            if (!b) return a;+            if (!a) return new nested_trie_type(*b);+            a->insertAll(*b);+            return a;+        }+    };++    // the operation capable of cloning a nested trie+    struct nested_trie_cloner {+        nested_trie_type* operator()(nested_trie_type* a) const {+            if (!a) return a;+            return new nested_trie_type(*a);+        }+    };++    // the data structure utilized for indexing nested tries+    using store_type = SparseArray<nested_trie_type*,+            6,  // = 2^6 entries per block+            nested_trie_merger, nested_trie_cloner>;++    // the actual data store+    store_type store;++public:+    using entry_type = typename souffle::Tuple<RamDomain, Dim>;+    using element_type = entry_type;++    // ---------------------------------------------------------------------+    //                           Iterator+    // ---------------------------------------------------------------------++    /**+     * The iterator core for trie iterators involving this level.+     */+    template <unsigned I = 0>+    class iterator_core {+        // the iterator for the current level+        using store_iter_t = typename store_type::iterator;++        // the type of the nested iterator+        using nested_iter_core = typename Trie<Dim - 1>::template iterator_core<I + 1>;++        store_iter_t iter;++        nested_iter_core nested;++    public:+        /** default end-iterator constructor */+        iterator_core() = default;++        template <typename Tuple>+        iterator_core(const store_iter_t& iter, Tuple& entry) : iter(iter) {+            entry[I] = iter->first;+            nested = iter->second->template getBeginCoreIterator<I + 1>(entry);+        }++        void setIterator(const store_iter_t& iter) {+            this->iter = iter;+        }++        store_iter_t& getIterator() {+            return this->iter;+        }++        nested_iter_core& getNested() {+            return nested;+        }++        template <typename Tuple>+        bool inc(Tuple& entry) {+            // increment nested iterator+            if (nested.inc(entry)) return true;++            // increment the iterator on this level+            ++iter;++            // check whether the end has been reached+            if (iter.isEnd()) return false;++            // otherwise update entry value+            entry[I] = iter->first;++            // and restart nested+            nested = iter->second->template getBeginCoreIterator<I + 1>(entry);+            return true;+        }++        bool operator==(const iterator_core& other) const {+            return nested == other.nested && iter == other.iter;+        }++        bool operator!=(const iterator_core& other) const {+            return !(*this == other);+        }++        // enables this iterator core to be printed (for debugging)+        void print(std::ostream& out) const {+            out << iter << " | " << nested;+        }++        friend std::ostream& operator<<(std::ostream& out, const iterator_core& iter) {+            iter.print(out);+            return out;+        }+    };++    // the type of iterator to be utilized when iterating of instances of this trie+    using iterator = typename base::template iterator<iterator_core>;++    // the operation context aggregating all operation contexts of nested structures+    struct op_context {+        using local_ctxt = typename store_type::op_context;+        using nested_ctxt = typename nested_trie_type::op_context;++        // for insert and contain+        local_ctxt local{};+        RamDomain lastQuery{};+        nested_trie_type* lastNested{nullptr};+        nested_ctxt nestedCtxt{};++        // for boundaries+        unsigned lastBoundaryLevels{Dim + 1};+        entry_type lastBoundaryRequest{};+        range<iterator> lastBoundaries{iterator(), iterator()};++        op_context() = default;+    };++    using operation_hints = op_context;++    using base::contains;+    using base::insert;++    /**+     * A simple destructore.+     */+    ~Trie() {+        for (auto& cur : store) {+            delete cur.second;  // clears all nested tries+        }+    }++    /**+     * Determines whether this trie is empty or not.+     */+    bool empty() const {+        return store.empty();+    }++    /**+     * Determines the number of entries in this trie.+     */+    std::size_t size() const {+        // the number of elements is lazy-evaluated+        std::size_t res = 0;+        for (const auto& cur : store) {+            res += cur.second->size();+        }+        return res;+    }++    /**+     * Computes the total memory usage of this data structure.+     */+    std::size_t getMemoryUsage() const {+        // compute the total memory usage of this level+        std::size_t res = sizeof(*this) - sizeof(store) + store.getMemoryUsage();++        // add the memory usage of sub-levels+        for (const auto& cur : store) {+            res += cur.second->getMemoryUsage();+        }++        // done+        return res;+    }++    /**+     * Removes all entries within this trie.+     */+    void clear() {+        // delete lower levels+        for (auto& cur : store) {+            delete cur.second;+        }++        // clear store+        store.clear();+    }++    /**+     * Inserts a new entry.+     *+     * @param tuple the entry to be added+     * @return true if the same tuple hasn't been present before, false otherwise+     */+    bool insert(const entry_type& tuple) {+        op_context ctxt;+        return insert(tuple, ctxt);+    }++    /**+     * Inserts a new entry. A operation context may be provided to exploit temporal+     * locality.+     *+     * @param tuple the entry to be added+     * @param ctxt the operation context to be utilized+     * @return true if the same tuple hasn't been present before, false otherwise+     */+    bool insert(const entry_type& tuple, op_context& ctxt) {+        return insert_internal<0>(tuple, ctxt);+    }++    /**+     * Determines whether a given tuple is present within the set specified+     * by this trie.+     *+     * @param tuple the tuple to be tested+     * @return true if present, false otherwise+     */+    bool contains(const entry_type& tuple) const {+        op_context ctxt;+        return contains(tuple, ctxt);+    }++    /**+     * Determines whether a given tuple is present within the set specified+     * by this trie. A operation context may be provided to exploit temporal+     * locality.+     *+     * @param tuple the entry to be added+     * @param ctxt the operation context to be utilized+     * @return true if the same tuple hasn't been present before, false otherwise+     */+    bool contains(const entry_type& tuple, op_context& ctxt) const {+        return contains_internal<0>(tuple, ctxt);+    }++    /**+     * Inserts all elements stored within the given trie into this trie.+     *+     * @param other the elements to be inserted into this trie+     */+    void insertAll(const Trie& other) {+        store.addAll(other.store);+    }++    /**+     * Obtains an iterator referencing the first element stored within this trie.+     */+    iterator begin() const {+        auto it = store.begin();+        if (it.isEnd()) return end();+        return iterator(it);+    }++    /**+     * Obtains an iterator referencing the position after the last element stored+     * within this trie.+     */+    iterator end() const {+        return iterator();+    }++    iterator find(const entry_type& entry) const {+        op_context ctxt;+        return find(entry, ctxt);+    }++    iterator find(const entry_type& entry, op_context& ctxt) const {+        auto range = getBoundaries<Dim>(entry, ctxt);+        return (!range.empty()) ? range.begin() : end();+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const entry_type& entry) const {+        op_context ctxt;+        return getBoundaries<levels>(entry, ctxt);+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements. A operation context may be provided to exploit temporal+     * locality.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @param ctxt the operation context to be utilized+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const entry_type& entry, op_context& ctxt) const {+        // if nothing is bound => just use begin and end+        if (levels == 0) return make_range(begin(), end());++        // check context+        if (ctxt.lastBoundaryLevels == levels) {+            bool fit = true;+            for (unsigned i = 0; i < levels; ++i) {+                fit = fit && (entry[i] == ctxt.lastBoundaryRequest[i]);+            }++            // if it fits => take it+            if (fit) {+                base::hint_stats.get_boundaries.addHit();+                return ctxt.lastBoundaries;+            }+        }++        // the hint has not been a hit+        base::hint_stats.get_boundaries.addMiss();++        // start with two end iterators+        iterator begin{};+        iterator end{};++        // adapt them level by level+        auto found = detail::fix_binding<levels, 0, Dim>()(store, begin, end, entry);+        if (!found) return make_range(iterator(), iterator());++        // update context+        ctxt.lastBoundaryLevels = levels;+        ctxt.lastBoundaryRequest = entry;+        ctxt.lastBoundaries = make_range(begin, end);++        // use the result+        return ctxt.lastBoundaries;+    }++    /**+     * Obtains an iterator to the first element not less than the given entry value.+     *+     * @param entry the lower bound for this search+     * @param ctxt the operation context to be utilized+     * @return an iterator addressing the first element in this structure not less than the given value+     */+    iterator lower_bound(const entry_type& entry, op_context& /* ctxt */) const {+        // start with a default-initialized iterator+        iterator res;++        // adapt it level by level+        bool found = detail::fix_lower_bound<0, Dim>()(store, res, entry);++        // use the result+        return found ? res : end();+    }++    /**+     * Obtains an iterator to the first element not less than the given entry value.+     *+     * @param entry the lower bound for this search+     * @return an iterator addressing the first element in this structure not less than the given value+     */+    iterator lower_bound(const entry_type& entry) const {+        op_context ctxt;+        return lower_bound(entry, ctxt);+    }++    /**+     * Obtains an iterator to the first element greater than the given entry value, or end if there is no such+     * element.+     *+     * @param entry the upper bound for this search+     * @param ctxt the operation context to be utilized+     * @return an iterator addressing the first element in this structure greater than the given value+     */+    iterator upper_bound(const entry_type& entry, op_context& /* ctxt */) const {+        // start with a default-initialized iterator+        iterator res;++        // adapt it level by level+        bool found = detail::fix_upper_bound<0, Dim>()(store, res, entry);++        // use the result+        return found ? res : end();+    }++    /**+     * Obtains an iterator to the first element greater than the given entry value, or end if there is no such+     * element.+     *+     * @param entry the upper bound for this search+     * @return an iterator addressing the first element in this structure greater than the given value+     */+    iterator upper_bound(const entry_type& entry) const {+        op_context ctxt;+        return upper_bound(entry, ctxt);+    }++    /**+     * Computes a partition of an approximate number of chunks of the content+     * of this trie. Thus, the union of the resulting set of disjoint ranges is+     * equivalent to the content of this trie.+     *+     * @param chunks the number of chunks requested+     * @return a list of sub-ranges forming a partition of the content of this trie+     */+    std::vector<range<iterator>> partition(unsigned chunks = 500) const {+        std::vector<range<iterator>> res;++        // shortcut for empty trie+        if (this->empty()) return res;++        // use top-level elements for partitioning+        int step = std::max(store.size() / chunks, size_t(1));++        int c = 1;+        auto priv = begin();+        for (auto it = store.begin(); it != store.end(); ++it, c++) {+            if (c % step != 0 || c == 1) {+                continue;+            }+            auto cur = iterator(it);+            res.push_back(make_range(priv, cur));+            priv = cur;+        }+        // add final chunk+        res.push_back(make_range(priv, end()));+        return res;+    }++    /**+     * Provides a protected access to the internally maintained store.+     */+    const store_type& getStore() const {+        return store;+    }++private:+    /**+     * Creates a core iterator for this trie level and updates component+     * I of the given entry to exhibit the corresponding first value.+     *+     * @tparam I the index of the tuple to be processed by the resulting iterator core+     * @tparam Tuple the type of the tuple to be processed by the resulting iterator core+     * @param entry a reference to the tuple to be updated to the first value+     * @return the requested iterator core instance+     */+    template <unsigned I, typename Tuple>+    iterator_core<I> getBeginCoreIterator(Tuple& entry) const {+        return iterator_core<I>(store.begin(), entry);+    }++    /**+     * The internally utilized implementation of the insert operation inserting+     * a given tuple into this sub-trie.+     *+     * @tparam I the component index associated to this level+     * @tparam Tuple the tuple type to be inserted+     * @param tuple the tuple to be inserted+     * @param ctxt a operation context to exploit temporal locality+     * @return true if this tuple wasn't contained before, false otherwise+     */+    template <unsigned I, typename Tuple>+    bool insert_internal(const Tuple& tuple, op_context& ctxt) {+        using value_t = typename store_type::value_type;+        using atomic_value_t = typename store_type::atomic_value_type;++        // check context+        if (ctxt.lastNested && ctxt.lastQuery == tuple[I]) {+            base::hint_stats.inserts.addHit();+            return ctxt.lastNested->template insert_internal<I + 1>(tuple, ctxt.nestedCtxt);+        } else {+            base::hint_stats.inserts.addMiss();+        }++        // lookup nested+        atomic_value_t& next = store.getAtomic(tuple[I], ctxt.local);++        // get pure pointer to next level+        value_t nextPtr = next;++        // conduct a lock-free lazy-creation of nested trees+        if (!nextPtr) {+            // create a new sub-tree+            auto newNested = new nested_trie_type();++            // register new sub-tree atomically+            if (next.compare_exchange_weak(nextPtr, newNested)) {+                nextPtr = newNested;  // worked+            } else {+                delete newNested;  // some other thread was faster => use its version+            }+        }++        // make sure a next has been established+        assert(nextPtr);++        // clear context if necessary+        if (nextPtr != ctxt.lastNested) {+            ctxt.lastQuery = tuple[I];+            ctxt.lastNested = nextPtr;+            ctxt.nestedCtxt = typename op_context::nested_ctxt();+        }++        // conduct recursive step+        return nextPtr->template insert_internal<I + 1>(tuple, ctxt.nestedCtxt);+    }++    /**+     * An internal implementation of the contains member function determining+     * whether a given tuple is present within this sub-trie or not.+     *+     * @tparam I the component index associated to this level+     * @tparam Tuple the tuple type to be checked+     * @param tuple the tuple to be checked+     * @param ctxt a operation context to exploit temporal locality+     * @return true if this tuple is present, false otherwise+     */+    template <unsigned I, typename Tuple>+    bool contains_internal(const Tuple& tuple, op_context& ctxt) const {+        // check context+        if (ctxt.lastNested && ctxt.lastQuery == tuple[I]) {+            base::hint_stats.contains.addHit();+            return ctxt.lastNested->template contains_internal<I + 1>(tuple, ctxt.nestedCtxt);+        } else {+            base::hint_stats.contains.addMiss();+        }++        // lookup next step+        auto next = store.lookup(tuple[I], ctxt.local);++        // clear context if necessary+        if (next != ctxt.lastNested) {+            ctxt.lastQuery = tuple[I];+            ctxt.lastNested = next;+            ctxt.nestedCtxt = typename op_context::nested_ctxt();+        }++        // conduct recursive step+        return next && next->template contains_internal<I + 1>(tuple, ctxt.nestedCtxt);+    }+};++/**+ * A template specialization for tries representing a set.+ * For improved memory efficiency, this level is the leaf-node level+ * of all tries exhibiting an arity >= 1. Internally, values are stored utilizing+ * sparse bit maps.+ */+template <>+class Trie<1u> : public detail::TrieBase<1u, Trie<1u>> {+    template <unsigned Dim>+    friend class Trie;++    template <unsigned Dim, typename Derived>+    friend class detail::TrieBase;++    // a shortcut for the base type+    using base = typename detail::TrieBase<1u, Trie<1u>>;++    // the map type utilized internally+    using map_type = SparseBitMap<>;++    // the internal data store+    map_type map;++public:+    using element_type = entry_type;+    using op_context = typename map_type::op_context;+    using operation_hints = op_context;++    using base::contains;+    using base::insert;++    /**+     * Determines whether this trie is empty or not.+     */+    bool empty() const {+        return map.empty();+    }++    /**+     * Determines the number of elements stored in this trie.+     */+    std::size_t size() const {+        return map.size();+    }++    /**+     * Computes the total memory usage of this data structure.+     */+    std::size_t getMemoryUsage() const {+        // compute the total memory usage+        return sizeof(*this) - sizeof(map_type) + map.getMemoryUsage();+    }++    /**+     * Removes all elements form this trie.+     */+    void clear() {+        map.clear();+    }++    /**+     * Inserts the given tuple into this trie.+     *+     * @param tuple the tuple to be inserted+     * @return true if the tuple has not been present before, false otherwise+     */+    bool insert(const entry_type& tuple) {+        op_context ctxt;+        return insert(tuple, ctxt);+    }++    /**+     * Inserts the given tuple into this trie.+     * An operation context can be provided to exploit temporal locality.+     *+     * @param tuple the tuple to be inserted+     * @param ctxt an operation context for exploiting temporal locality+     * @return true if the tuple has not been present before, false otherwise+     */+    bool insert(const entry_type& tuple, op_context& ctxt) {+        return insert_internal<0>(tuple, ctxt);+    }++    /**+     * Determines whether the given tuple is present in this trie or not.+     *+     * @param tuple the tuple to be tested+     * @return true if present, false otherwise+     */+    bool contains(const entry_type& tuple) const {+        op_context ctxt;+        return contains(tuple, ctxt);+    }++    /**+     * Determines whether the given tuple is present in this trie or not.+     * An operation context can be provided to exploit temporal locality.+     *+     * @param tuple the tuple to be tested+     * @param ctxt an operation context for exploiting temporal locality+     * @return true if present, false otherwise+     */+    bool contains(const entry_type& tuple, op_context& ctxt) const {+        return contains_internal<0>(tuple, ctxt);+    }++    /**+     * Inserts all tuples stored within the given trie into this trie.+     * This operation is considerably more efficient than the consecutive+     * insertion of the elements in other into this trie.+     */+    void insertAll(const Trie& other) {+        map.addAll(other.map);+    }++    // ---------------------------------------------------------------------+    //                           Iterator+    // ---------------------------------------------------------------------++    /**+     * The iterator core of this level contributing to the construction of+     * a composed trie iterator.+     */+    template <unsigned I = 0>+    class iterator_core {+        // the iterator for this level+        using iter_type = typename map_type::iterator;++        // the referenced bit-map iterator+        iter_type iter;++    public:+        /** default end-iterator constructor */+        iterator_core() = default;++        template <typename Tuple>+        iterator_core(const iter_type& iter, Tuple& entry) : iter(iter) {+            entry[I] = *iter;+        }++        void setIterator(const iter_type& iter) {+            this->iter = iter;+        }++        iter_type& getIterator() {+            return this->iter;+        }++        template <typename Tuple>+        bool inc(Tuple& entry) {+            // increment the iterator on this level+            ++iter;++            // check whether the end has been reached+            if (iter.isEnd()) return false;++            // otherwise update entry value+            entry[I] = *iter;+            return true;+        }++        bool operator==(const iterator_core& other) const {+            return iter == other.iter;+        }++        bool operator!=(const iterator_core& other) const {+            return !(*this == other);+        }++        // enables this iterator core to be printed (for debugging)+        void print(std::ostream& out) const {+            out << iter;+        }++        friend std::ostream& operator<<(std::ostream& out, const iterator_core& iter) {+            iter.print(out);+            return out;+        }+    };++    // the iterator type utilized by this trie type+    using iterator = typename base::template iterator<iterator_core>;++    /**+     * Obtains an iterator referencing the first element stored within this trie+     * or end() if this trie is empty.+     */+    iterator begin() const {+        if (map.empty()) return end();+        return iterator(map.begin());+    }++    /**+     * Obtains an iterator referencing the first position after the last element+     * within this trie.+     */+    iterator end() const {+        return iterator();+    }++    /**+     * Obtains a partition of this tire such that the resulting list of ranges+     * cover disjoint subsets of the elements stored in this trie. Their union+     * is equivalent to the content of this trie.+     */+    std::vector<range<iterator>> partition(unsigned chunks = 500) const {+        std::vector<range<iterator>> res;++        // shortcut for empty trie+        if (this->empty()) return res;++        // use top-level elements for partitioning+        int step = static_cast<int>(std::max(map.size() / chunks, size_t(1)));++        int c = 1;+        auto priv = begin();+        for (auto it = map.begin(); it != map.end(); ++it, c++) {+            if (c % step != 0 || c == 1) {+                continue;+            }+            auto cur = iterator(it);+            res.push_back(make_range(priv, cur));+            priv = cur;+        }+        // add final chunk+        res.push_back(make_range(priv, end()));+        return res;+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const entry_type& entry) const {+        op_context ctxt;+        return getBoundaries<levels>(entry, ctxt);+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements. A operation context may be provided to exploit temporal+     * locality.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @param ctxt the operation context to be utilized+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const entry_type& entry, op_context& ctxt) const {+        // for levels = 0+        if (levels == 0) return make_range(begin(), end());+        // for levels = 1+        auto pos = map.find(entry[0], ctxt);+        if (pos == map.end()) return make_range(end(), end());+        auto next = pos;+        ++next;+        return make_range(iterator(pos), iterator(next));+    }++    iterator lower_bound(const entry_type& entry, op_context&) const {+        return iterator(map.lower_bound(entry[0]));+    }++    iterator lower_bound(const entry_type& entry) const {+        op_context ctxt;+        return lower_bound(entry, ctxt);+    }++    iterator upper_bound(const entry_type& entry, op_context&) const {+        return iterator(map.upper_bound(entry[0]));+    }++    iterator upper_bound(const entry_type& entry) const {+        op_context ctxt;+        return upper_bound(entry, ctxt);+    }++    /**+     * Provides protected access to the internally maintained store.+     */+    const map_type& getStore() const {+        return map;+    }++private:+    /**+     * Creates a core iterator for this trie level and updates component+     * I of the given entry to exhibit the corresponding first value.+     *+     * @tparam I the index of the tuple to be processed by the resulting iterator core+     * @tparam Tuple the type of the tuple to be processed by the resulting iterator core+     * @param entry a reference to the tuple to be updated to the first value+     * @return the requested iterator core instance+     */+    template <unsigned I, typename Tuple>+    iterator_core<I> getBeginCoreIterator(Tuple& entry) const {+        return iterator_core<I>(map.begin(), entry);+    }++    /**+     * The internally utilized implementation of the insert operation inserting+     * a given tuple into this sub-trie.+     *+     * @tparam I the component index associated to this level+     * @tparam Tuple the tuple type to be inserted+     * @param tuple the tuple to be inserted+     * @param ctxt a operation context to exploit temporal locality+     * @return true if this tuple wasn't contained before, false otherwise+     */+    template <unsigned I, typename Tuple>+    bool insert_internal(const Tuple& tuple, op_context& ctxt) {+        return map.set(tuple[I], ctxt);+    }++    /**+     * An internal implementation of the contains member function determining+     * whether a given tuple is present within this sub-trie or not.+     *+     * @tparam I the component index associated to this level+     * @tparam Tuple the tuple type to be checked+     * @param tuple the tuple to be checked+     * @param ctxt a operation context to exploit temporal locality+     * @return true if this tuple is present, false otherwise+     */+    template <unsigned I, typename Tuple>+    bool contains_internal(const Tuple& tuple, op_context& ctxt) const {+        return map.test(tuple[I], ctxt);+    }+};++}  // end namespace souffle
+ cbits/souffle/CompiledOptions.h view
@@ -0,0 +1,257 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file CompiledOptions.h+ *+ * A header file offering command-line option support for compiled+ * RAM programs.+ *+ ***********************************************************************/++#pragma once++#include <cstdio>+#include <iostream>+#include <string>+#include <getopt.h>+#include <stdlib.h>+#include <sys/stat.h>++namespace souffle {++/**+ * A utility class for parsing command line arguments within generated+ * query programs.+ */+class CmdOptions {+protected:+    /**+     * source file+     */+    std::string src;++    /**+     * fact directory+     */+    std::string input_dir;++    /**+     * output directory+     */+    std::string output_dir;++    /**+     * profiling flag+     */+    bool profiling;++    /**+     * profile filename+     */+    std::string profile_name;++    /**+     * number of threads+     */+    size_t num_jobs;++public:+    // all argument constructor+    CmdOptions(const char* s, const char* id, const char* od, bool pe, const char* pfn, size_t nj)+            : src(s), input_dir(id), output_dir(od), profiling(pe), profile_name(pfn), num_jobs(nj) {}++    /**+     * get source code name+     */+    const std::string& getSourceFileName() const {+        return src;+    }++    /**+     * get input directory+     */+    const std::string& getInputFileDir() const {+        return input_dir;+    }++    /**+     * get output directory+     */+    const std::string& getOutputFileDir() const {+        return output_dir;+    }++    /**+     * is profiling switched on+     */+    bool isProfiling() const {+        return profiling;+    }++    /**+     * get filename of profile+     */+    const std::string& getProfileName() const {+        return profile_name;+    }++    /**+     * get number of jobs+     */+    size_t getNumJobs() const {+        return num_jobs;+    }++    /**+     * Parses the given command line parameters, handles -h help requests or errors+     * and returns whether the parsing was successful or not.+     */+    bool parse(int argc, char** argv) {+        // get executable name+        std::string exec_name = "analysis";+        if (argc > 0) {+            exec_name = argv[0];+        }++        // local options+        std::string fact_dir = input_dir;+        std::string out_dir = output_dir;++// avoid warning due to Solaris getopt.h+#pragma GCC diagnostic push+#pragma GCC diagnostic ignored "-Wwrite-strings"+        // long options+        option longOptions[] = {{"facts", true, nullptr, 'F'}, {"output", true, nullptr, 'D'},+                {"profile", true, nullptr, 'p'}, {"jobs", true, nullptr, 'j'}, {"index", true, nullptr, 'i'},+                // the terminal option -- needs to be null+                {nullptr, false, nullptr, 0}};+#pragma GCC diagnostic pop++        // check whether all options are fine+        bool ok = true;++        int c; /* command-line arguments processing */+        while ((c = getopt_long(argc, argv, "D:F:hp:j:i:", longOptions, nullptr)) != EOF) {+            switch (c) {+                /* Fact directories */+                case 'F':+                    if (!existDir(optarg)) {+                        printf("Fact directory %s does not exists!\n", optarg);+                        ok = false;+                    }+                    fact_dir = optarg;+                    break;+                /* Output directory for resulting .csv files */+                case 'D':+                    if (*optarg && !existDir(optarg)) {+                        printf("Output directory %s does not exists!\n", optarg);+                        ok = false;+                    }+                    out_dir = optarg;+                    break;+                case 'p':+                    if (!profiling) {+                        std::cerr << "\nError: profiling was not enabled in compilation\n\n";+                        printHelpPage(exec_name);+                        exit(EXIT_FAILURE);+                    }+                    profile_name = optarg;+                    break;+                case 'j':+#ifdef _OPENMP+                    if (std::string(optarg) == "auto") {+                        num_jobs = 0;+                    } else {+                        int num = atoi(optarg);+                        if (num > 0) {+                            num_jobs = num;+                        } else {+                            std::cerr << "Invalid number of jobs [-j]: " << optarg << "\n";+                            ok = false;+                        }+                    }+#else+                    std::cerr << "\nWarning: OpenMP was not enabled in compilation\n\n";+#endif+                    break;+                default: printHelpPage(exec_name); return false;+            }+        }++        // update member fields+        input_dir = fact_dir;+        output_dir = out_dir;++        // return success state+        return ok;+    }++private:+    /**+     * Prints the help page if it has been requested or there was a typo in the command line arguments.+     */+    void printHelpPage(const std::string& exec_name) const {+        std::cerr << "====================================================================\n";+        std::cerr << " Datalog Program: " << src << "\n";+        std::cerr << " Usage: " << exec_name << " [OPTION]\n\n";+        std::cerr << " Options:\n";+        std::cerr << "    -D <DIR>, --output=<DIR>     -- Specify directory for output relations\n";+        std::cerr << "                                    (default: " << output_dir << ")\n";+        std::cerr << "                                    (suppress output with \"\")\n";+        std::cerr << "    -F <DIR>, --facts=<DIR>      -- Specify directory for fact files\n";+        std::cerr << "                                    (default: " << input_dir << ")\n";+        if (profiling) {+            std::cerr << "    -p <file>, --profile=<file>  -- Specify filename for profiling\n";+            std::cerr << "                                    (default: " << profile_name << ")\n";+        }+#ifdef _OPENMP+        std::cerr << "    -j <NUM>, --jobs=<NUM>       -- Specify number of threads\n";+        if (num_jobs > 0) {+            std::cerr << "                                    (default: " << num_jobs << ")\n";+        } else {+            std::cerr << "                                    (default: auto)\n";+        }+#endif+        std::cerr << "    -h                           -- prints this help page.\n";+        std::cerr << "--------------------------------------------------------------------\n";+        std::cout << " Copyright (c) 2016-20 The Souffle Developers." << std::endl;+        std::cout << " Copyright (c) 2013-16 Oracle and/or its affiliates." << std::endl;+        std::cerr << " All rights reserved.\n";+        std::cerr << "====================================================================\n";+    }++    /**+     *  Check whether a file exists in the file system+     */+    inline bool existFile(const std::string& name) const {+        struct stat buffer;+        if (stat(name.c_str(), &buffer) == 0) {+            if ((buffer.st_mode & S_IFREG) != 0) {+                return true;+            }+        }+        return false;+    }++    /**+     *  Check whether a directory exists in the file system+     */+    bool existDir(const std::string& name) const {+        struct stat buffer;+        if (stat(name.c_str(), &buffer) == 0) {+            if ((buffer.st_mode & S_IFDIR) != 0) {+                return true;+            }+        }+        return false;+    }+};++}  // end of namespace souffle
+ cbits/souffle/CompiledSouffle.h view
@@ -0,0 +1,330 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file CompiledSouffle.h+ *+ * Main include file for generated C++ classes of Souffle+ *+ ***********************************************************************/++#pragma once++#include "souffle/Brie.h"+#include "souffle/CompiledTuple.h"+#include "souffle/EquivalenceRelation.h"+#include "souffle/IOSystem.h"+#include "souffle/IterUtils.h"+#include "souffle/RamTypes.h"+#include "souffle/RecordTable.h"+#include "souffle/SignalHandler.h"+#include "souffle/SouffleInterface.h"+#include "souffle/SymbolTable.h"+#include "souffle/Table.h"+#include "souffle/WriteStream.h"+#include "souffle/utility/CacheUtil.h"+#include "souffle/utility/ContainerUtil.h"+#include "souffle/utility/EvaluatorUtil.h"+#include "souffle/utility/FileUtil.h"+#include "souffle/utility/FunctionalUtil.h"+#include "souffle/utility/MiscUtil.h"+#include "souffle/utility/ParallelUtil.h"+#include "souffle/utility/StreamUtil.h"+#include "souffle/utility/StringUtil.h"+#ifndef __EMBEDDED_SOUFFLE__+#include "souffle/CompiledOptions.h"+#include "souffle/Logger.h"+#include "souffle/ProfileEvent.h"+#endif+#include <array>+#include <atomic>+#include <cassert>+#include <cmath>+#include <cstdint>+#include <cstdlib>+#include <exception>+#include <iostream>+#include <iterator>+#include <memory>+#include <regex>+#include <string>+#include <utility>+#include <vector>++#if defined(_OPENMP)+#include <omp.h>+#endif++namespace souffle {++extern "C" {+inline souffle::SouffleProgram* getInstance(const char* p) {+    return souffle::ProgramFactory::newInstance(p);+}+}++/**+ * Relation wrapper used internally in the generated Datalog program+ */+template <uint32_t id, class RelType, class TupleType, size_t Arity, size_t NumAuxAttributes>+class RelationWrapper : public souffle::Relation {+private:+    RelType& relation;+    SymbolTable& symTable;+    std::string name;+    std::array<const char*, Arity> tupleType;+    std::array<const char*, Arity> tupleName;++    class iterator_wrapper : public iterator_base {+        typename RelType::iterator it;+        const Relation* relation;+        tuple t;++    public:+        iterator_wrapper(uint32_t arg_id, const Relation* rel, const typename RelType::iterator& arg_it)+                : iterator_base(arg_id), it(arg_it), relation(rel), t(rel) {}+        void operator++() override {+            ++it;+        }+        tuple& operator*() override {+            t.rewind();+            for (size_t i = 0; i < Arity; i++) {+                t[i] = (*it)[i];+            }+            return t;+        }+        iterator_base* clone() const override {+            return new iterator_wrapper(*this);+        }++    protected:+        bool equal(const iterator_base& o) const override {+            const auto& casted = static_cast<const iterator_wrapper&>(o);+            return it == casted.it;+        }+    };++public:+    RelationWrapper(RelType& r, SymbolTable& s, std::string name, const std::array<const char*, Arity>& t,+            const std::array<const char*, Arity>& n)+            : relation(r), symTable(s), name(std::move(name)), tupleType(t), tupleName(n) {}+    iterator begin() const override {+        return iterator(new iterator_wrapper(id, this, relation.begin()));+    }+    iterator end() const override {+        return iterator(new iterator_wrapper(id, this, relation.end()));+    }+    void insert(const tuple& arg) override {+        TupleType t;+        assert(&arg.getRelation() == this && "wrong relation");+        assert(arg.size() == Arity && "wrong tuple arity");+        for (size_t i = 0; i < Arity; i++) {+            t[i] = arg[i];+        }+        relation.insert(t);+    }+    bool contains(const tuple& arg) const override {+        TupleType t;+        assert(arg.size() == Arity && "wrong tuple arity");+        for (size_t i = 0; i < Arity; i++) {+            t[i] = arg[i];+        }+        return relation.contains(t);+    }+    std::size_t size() const override {+        return relation.size();+    }+    std::string getName() const override {+        return name;+    }+    const char* getAttrType(size_t arg) const override {+        assert(arg < Arity && "attribute out of bound");+        return tupleType[arg];+    }+    const char* getAttrName(size_t arg) const override {+        assert(arg < Arity && "attribute out of bound");+        return tupleName[arg];+    }+    size_t getArity() const override {+        return Arity;+    }+    size_t getAuxiliaryArity() const override {+        return NumAuxAttributes;+    }+    SymbolTable& getSymbolTable() const override {+        return symTable;+    }++    /** Eliminate all the tuples in relation*/+    void purge() override {+        relation.purge();+    }+};++/** Nullary relations */+class t_nullaries {+private:+    std::atomic<bool> data{false};++public:+    t_nullaries() = default;+    using t_tuple = Tuple<RamDomain, 0>;+    struct context {};+    context createContext() {+        return context();+    }+    class iterator : public std::iterator<std::forward_iterator_tag, RamDomain*> {+        bool value;++    public:+        iterator(bool v = false) : value(v) {}++        const RamDomain* operator*() {+            return nullptr;+        }++        bool operator==(const iterator& other) const {+            return other.value == value;+        }++        bool operator!=(const iterator& other) const {+            return other.value != value;+        }++        iterator& operator++() {+            if (value) {+                value = false;+            }+            return *this;+        }+    };+    iterator begin() const {+        return iterator(data);+    }+    iterator end() const {+        return iterator();+    }+    void insert(const t_tuple& /* t */) {+        data = true;+    }+    void insert(const t_tuple& /* t */, context& /* ctxt */) {+        data = true;+    }+    void insert(const RamDomain* /* ramDomain */) {+        data = true;+    }+    bool insert() {+        bool result = data;+        data = true;+        return !result;+    }+    bool contains(const t_tuple& /* t */) const {+        return data;+    }+    bool contains(const t_tuple& /* t */, context& /* ctxt */) const {+        return data;+    }+    std::size_t size() const {+        return data ? 1 : 0;+    }+    bool empty() const {+        return !data;+    }+    void purge() {+        data = false;+    }+    void printHintStatistics(std::ostream& /* o */, std::string /* prefix */) const {}+};++/** info relations */+template <int Arity>+class t_info {+private:+    std::vector<Tuple<RamDomain, Arity>> data;+    Lock insert_lock;++public:+    t_info() = default;+    using t_tuple = Tuple<RamDomain, Arity>;+    struct context {};+    context createContext() {+        return context();+    }+    class iterator : public std::iterator<std::forward_iterator_tag, Tuple<RamDomain, Arity>> {+        typename std::vector<Tuple<RamDomain, Arity>>::const_iterator it;++    public:+        iterator(const typename std::vector<t_tuple>::const_iterator& o) : it(o) {}++        const t_tuple operator*() {+            return *it;+        }++        bool operator==(const iterator& other) const {+            return other.it == it;+        }++        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        iterator& operator++() {+            it++;+            return *this;+        }+    };+    iterator begin() const {+        return iterator(data.begin());+    }+    iterator end() const {+        return iterator(data.end());+    }+    void insert(const t_tuple& t) {+        insert_lock.lock();+        if (!contains(t)) {+            data.push_back(t);+        }+        insert_lock.unlock();+    }+    void insert(const t_tuple& t, context& /* ctxt */) {+        insert(t);+    }+    void insert(const RamDomain* ramDomain) {+        insert_lock.lock();+        t_tuple t;+        for (size_t i = 0; i < Arity; ++i) {+            t.data[i] = ramDomain[i];+        }+        data.push_back(t);+        insert_lock.unlock();+    }+    bool contains(const t_tuple& t) const {+        for (const auto& o : data) {+            if (t == o) {+                return true;+            }+        }+        return false;+    }+    bool contains(const t_tuple& t, context& /* ctxt */) const {+        return contains(t);+    }+    std::size_t size() const {+        return data.size();+    }+    bool empty() const {+        return data.size() == 0;+    }+    void purge() {+        data.clear();+    }+    void printHintStatistics(std::ostream& /* o */, std::string /* prefix */) const {}+};++}  // namespace souffle
+ cbits/souffle/CompiledTuple.h view
@@ -0,0 +1,186 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file CompiledTuple.h+ *+ * The central file covering the data structure utilized by+ * the souffle compiler for representing relations in compiled queries.+ *+ ***********************************************************************/++#pragma once++#include <cstddef>+#include <functional>+#include <iostream>+#include <system_error>++namespace souffle {++/**+ * The type of object stored within relations representing the actual+ * tuple value. Each tuple consists of a constant number of components.+ *+ * @tparam Domain the domain of the component values+ * @tparam arity the number of components within an instance+ */+template <typename Domain, std::size_t _arity>+struct Tuple {+    // some features for template meta programming+    using value_type = Domain;+    static constexpr size_t arity = _arity;++    // the stored data+    Domain data[arity];++    // constructores, destructors and assignment are default++    // provide access to components+    const Domain& operator[](std::size_t index) const {+        return data[index];+    }++    // provide access to components+    Domain& operator[](std::size_t index) {+        return data[index];+    }++    // a comparison operation+    bool operator==(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; i++) {+            if (data[i] != other.data[i]) return false;+        }+        return true;+    }++    // inequality comparison+    bool operator!=(const Tuple& other) const {+        return !(*this == other);+    }++    // required to put tuples into e.g. a std::set container+    bool operator<(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; ++i) {+            if (data[i] < other.data[i]) return true;+            if (data[i] > other.data[i]) return false;+        }+        return false;+    }++    // required to put tuples into e.g. a btree container+    bool operator>(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; ++i) {+            if (data[i] > other.data[i]) return true;+            if (data[i] < other.data[i]) return false;+        }+        return false;+    }++    // allow tuples to be printed+    friend std::ostream& operator<<(std::ostream& out, const Tuple& tuple) {+        if (arity == 0) return out << "[]";+        out << "[";+        for (std::size_t i = 0; i < (std::size_t)(arity - 1); ++i) {+            out << tuple.data[i];+            out << ",";+        }+        return out << tuple.data[arity - 1] << "]";+    }+};++#ifdef _MSC_VER+/**+ * A template specialization for 0-arity tuples when compiling with microsoft's+ * compiler, because it doesn't like the 0 length array even though it is the+ * last member of the struct.+ */+template <typename Domain>+struct Tuple<Domain, 0> {+    // some features for template meta programming+    using value_type = Domain;+    enum { arity = 0 };++    // the stored data+    Domain data[1];++    // constructores, destructors and assignment are default++    // provide access to components+    const Domain& operator[](std::size_t index) const {+        return data[index];+    }++    // provide access to components+    Domain& operator[](std::size_t index) {+        return data[index];+    }++    // a comparison operation+    bool operator==(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; i++) {+            if (data[i] != other.data[i]) return false;+        }+        return true;+    }++    // inequality comparison+    bool operator!=(const Tuple& other) const {+        return !(*this == other);+    }++    // required to put tuples into e.g. a std::set container+    bool operator<(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; ++i) {+            if (data[i] < other.data[i]) return true;+            if (data[i] > other.data[i]) return false;+        }+        return false;+    }++    // required to put tuples into e.g. a btree container+    bool operator>(const Tuple& other) const {+        for (std::size_t i = 0; i < arity; ++i) {+            if (data[i] > other.data[i]) return true;+            if (data[i] < other.data[i]) return false;+        }+        return false;+    }++    // allow tuples to be printed+    friend std::ostream& operator<<(std::ostream& out, const Tuple& tuple) {+        if (arity == 0) return out << "[]";+        out << "[";+        for (std::size_t i = 0; i < (std::size_t)(arity - 1); ++i) {+            out << tuple.data[i];+            out << ",";+        }+        return out << tuple.data[arity - 1] << "]";+    }+};+#endif  // _MSC_VER+}  // end of namespace souffle++// -- add hashing support --++namespace std {++template <typename Domain, std::size_t arity>+struct hash<souffle::Tuple<Domain, arity>> {+    size_t operator()(const souffle::Tuple<Domain, arity>& value) const {+        std::hash<Domain> hash;+        size_t res = 0;+        for (unsigned i = 0; i < arity; i++) {+            // from boost hash combine+            res ^= hash(value[i]) + 0x9e3779b9 + (res << 6) + (res >> 2);+        }+        return res;+    }+};+}  // namespace std
+ cbits/souffle/EquivalenceRelation.h view
@@ -0,0 +1,730 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2017 The Souffle Developers. All Rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file EquivalenceRelation.h+ *+ * Defines a binary relation interface to be used with Souffle as a relational store.+ * Pairs inserted into this relation implicitly store a reflexive, symmetric, and transitive relation+ * with each other.+ *+ ***********************************************************************/++#pragma once++#include "LambdaBTree.h"+#include "PiggyList.h"+#include "RamTypes.h"+#include "UnionFind.h"+#include "utility/ContainerUtil.h"+#include "utility/ParallelUtil.h"+#include <atomic>+#include <cassert>+#include <cstddef>+#include <functional>+#include <iostream>+#include <iterator>+#include <set>+#include <shared_mutex>+#include <stdexcept>+#include <tuple>+#include <utility>+#include <vector>++namespace souffle {+template <typename TupleType>+class EquivalenceRelation {+    using value_type = typename TupleType::value_type;++    // mapping from representative to disjoint set+    // just a cache, essentially, used for iteration over+    using StatesList = souffle::PiggyList<value_type>;+    using StatesBucket = StatesList*;+    using StorePair = std::pair<value_type, StatesBucket>;+    using StatesMap = souffle::LambdaBTreeSet<StorePair, std::function<StatesBucket(StorePair&)>,+            souffle::EqrelMapComparator<StorePair>>;++public:+    using element_type = TupleType;++    EquivalenceRelation() : statesMapStale(false){};+    ~EquivalenceRelation() {+        emptyPartition();+    }++    /**+     * A collection of operation hints speeding up some of the involved operations+     * by exploiting temporal locality.+     * Unused in this class, as there is no speedup to be gained.+     * This is just defined as the class expects it.+     */+    struct operation_hints {+        // resets all hints (to be triggered e.g. when deleting nodes)+        void clear() {}+    };++    /**+     * Insert the two values symbolically as a binary relation+     * @param x node to be added/paired+     * @param y node to be added/paired+     * @return true if the pair is new to the data structure+     */+    bool insert(value_type x, value_type y) {+        operation_hints z;+        return insert(x, y, z);+    };++    /**+     * Insert the tuple symbolically.+     * @param tuple The tuple to be inserted+     * @return true if the tuple is new to the data structure+     */+    bool insert(const TupleType& tuple) {+        operation_hints hints;+        return insert(tuple[0], tuple[1], hints);+    };++    /**+     * Insert the two values symbolically as a binary relation+     * @param x node to be added/paired+     * @param y node to be added/paired+     * @param z the hints to where the pair should be inserted (not applicable atm)+     * @return true if the pair is new to the data structure+     */+    bool insert(value_type x, value_type y, operation_hints) {+        // indicate that iterators will have to generate on request+        this->statesMapStale.store(true, std::memory_order_relaxed);+        bool retval = contains(x, y);+        sds.unionNodes(x, y);+        return retval;+    }++    /**+     * inserts all nodes from the other relation into this one+     * @param other the binary relation from which to add elements from+     */+    void insertAll(const EquivalenceRelation<TupleType>& other) {+        other.genAllDisjointSetLists();++        // iterate over partitions at a time+        for (typename StatesMap::chunk it : other.equivalencePartition.getChunks(MAX_THREADS)) {+            for (auto& p : it) {+                value_type rep = p.first;+                StatesList& pl = *p.second;+                const size_t ksize = pl.size();+                for (size_t i = 0; i < ksize; ++i) {+                    this->sds.unionNodes(rep, pl.get(i));+                }+            }+        }+        // invalidate iterators unconditionally+        this->statesMapStale.store(true, std::memory_order_relaxed);+    }++    /**+     * Extend this relation with another relation, expanding this equivalence relation+     * The supplied relation is the old knowledge, whilst this relation only contains+     * explicitly new knowledge. After this operation the "implicitly new tuples" are now+     * explicitly inserted this relation.+     */+    void extend(const EquivalenceRelation<TupleType>& other) {+        // nothing to extend if there's no new/original knowledge+        if (other.size() == 0 || this->size() == 0) return;++        this->genAllDisjointSetLists();+        other.genAllDisjointSetLists();++        std::set<value_type> repsCovered;++        // find all the disjoint sets that need to be added to this relation+        // that exist in other (and exist in this)+        {+            auto it = this->sds.sparseToDenseMap.begin();+            auto end = this->sds.sparseToDenseMap.end();+            value_type el;+            for (; it != end; ++it) {+                std::tie(el, std::ignore) = *it;+                if (other.containsElement(el)) {+                    value_type rep = other.sds.findNode(el);+                    if (repsCovered.count(rep) == 0) {+                        repsCovered.emplace(rep);+                    }+                }+            }+        }++        // add the intersecting dj sets into this one+        {+            value_type el;+            value_type rep;+            auto it = other.sds.sparseToDenseMap.begin();+            auto end = other.sds.sparseToDenseMap.end();+            for (; it != end; ++it) {+                std::tie(el, std::ignore) = *it;+                rep = other.sds.findNode(el);+                if (repsCovered.count(rep) != 0) {+                    this->insert(el, rep);+                }+            }+        }+    }++    /**+     * Returns whether there exists a pair with these two nodes+     * @param x front of pair+     * @param y back of pair+     */+    bool contains(value_type x, value_type y) const {+        return sds.contains(x, y);+    }++    /**+     * Returns whether there exists given tuple.+     * @param tuple The tuple to search for.+     */+    bool contains(const TupleType& tuple, operation_hints&) const {+        return contains(tuple[0], tuple[1]);+    };++    void emptyPartition() const {+        // delete the beautiful values inside (they're raw ptrs, so they need to be.)+        for (auto& pair : equivalencePartition) {+            delete pair.second;+        }+        // invalidate it my dude+        this->statesMapStale.store(true, std::memory_order_relaxed);++        equivalencePartition.clear();+    }++    /**+     * Empty the relation+     */+    void clear() {+        statesLock.lock();++        sds.clear();+        emptyPartition();++        statesLock.unlock();+    }++    /**+     * Size of relation+     * @return the sum of the number of pairs per disjoint set+     */+    size_t size() const {+        genAllDisjointSetLists();++        statesLock.lock_shared();++        size_t retVal = 0;+        for (auto& e : this->equivalencePartition) {+            const size_t s = e.second->size();+            retVal += s * s;+        }++        statesLock.unlock_shared();+        return retVal;+    }++    // an almighty iterator for several types of iteration.+    // Unfortunately, subclassing isn't an option with souffle+    //   - we don't deal with pointers (so no virtual)+    //   - and a single iter type is expected (see Relation::iterator e.g.) (i think)+    class iterator : public std::iterator<std::forward_iterator_tag, TupleType> {+    public:+        // one iterator for signalling the end (simplifies)+        explicit iterator(const EquivalenceRelation* br, bool /* signalIsEndIterator */)+                : br(br), isEndVal(true){};++        explicit iterator(const EquivalenceRelation* br)+                : br(br), ityp(IterType::ALL), djSetMapListIt(br->equivalencePartition.begin()),+                  djSetMapListEnd(br->equivalencePartition.end()) {+            // no need to fast forward if this iterator is empty+            if (djSetMapListIt == djSetMapListEnd) {+                isEndVal = true;+                return;+            }+            // grab the pointer to the list, and make it our current list+            djSetList = (*djSetMapListIt).second;+            assert(djSetList->size() != 0);++            updateAnterior();+            updatePosterior();+        }++        // WITHIN: iterator for everything within the same DJset (used for EquivalenceRelation.partition())+        explicit iterator(const EquivalenceRelation* br, const StatesBucket within)+                : br(br), ityp(IterType::WITHIN), djSetList(within) {+            // empty dj set+            if (djSetList->size() == 0) {+                isEndVal = true;+            }++            updateAnterior();+            updatePosterior();+        }++        // ANTERIOR: iterator that yields all (former, _) \in djset(former) (djset(former) === within)+        explicit iterator(const EquivalenceRelation* br, const value_type former, const StatesBucket within)+                : br(br), ityp(IterType::ANTERIOR), djSetList(within) {+            if (djSetList->size() == 0) {+                isEndVal = true;+            }++            setAnterior(former);+            updatePosterior();+        }++        // ANTPOST: iterator that yields all (former, latter) \in djset(former), (djset(former) ==+        // djset(latter) == within)+        explicit iterator(const EquivalenceRelation* br, const value_type former, value_type latter,+                const StatesBucket within)+                : br(br), ityp(IterType::ANTPOST), djSetList(within) {+            if (djSetList->size() == 0) {+                isEndVal = true;+            }++            setAnterior(former);+            setPosterior(latter);+        }++        /** explicit set first half of cPair */+        inline void setAnterior(const value_type a) {+            this->cPair[0] = a;+        }++        /** quick update to whatever the current index is pointing to */+        inline void updateAnterior() {+            this->cPair[0] = this->djSetList->get(this->cAnteriorIndex);+        }++        /** explicit set second half of cPair */+        inline void setPosterior(const value_type b) {+            this->cPair[1] = b;+        }++        /** quick update to whatever the current index is pointing to */+        inline void updatePosterior() {+            this->cPair[1] = this->djSetList->get(this->cPosteriorIndex);+        }++        // copy ctor+        iterator(const iterator& other) = default;+        // move ctor+        iterator(iterator&& other) = default;+        // assign iter+        iterator& operator=(const iterator& other) = default;++        bool operator==(const iterator& other) const {+            if (isEndVal && other.isEndVal) return br == other.br;+            return isEndVal == other.isEndVal && cPair == other.cPair;+        }++        bool operator!=(const iterator& other) const {+            return !((*this) == other);+        }++        const TupleType& operator*() const {+            return cPair;+        }++        const TupleType* operator->() const {+            return &cPair;+        }++        /* pre-increment */+        iterator& operator++() {+            if (isEndVal) {+                throw std::out_of_range("error: incrementing an out of range iterator");+            }++            switch (ityp) {+                case IterType::ALL:+                    // move posterior along one+                    // see if we can't move the posterior along+                    if (++cPosteriorIndex == djSetList->size()) {+                        // move anterior along one+                        // see if we can't move the anterior along one+                        if (++cAnteriorIndex == djSetList->size()) {+                            // move the djset it along one+                            // see if we can't move it along one (we're at the end)+                            if (++djSetMapListIt == djSetMapListEnd) {+                                isEndVal = true;+                                return *this;+                            }++                            // we can't iterate along this djset if it is empty+                            djSetList = (*djSetMapListIt).second;+                            if (djSetList->size() == 0) {+                                throw std::out_of_range("error: encountered a zero size djset");+                            }++                            // update our cAnterior and cPosterior+                            cAnteriorIndex = 0;+                            cPosteriorIndex = 0;+                            updateAnterior();+                            updatePosterior();+                        }++                        // we moved our anterior along one+                        updateAnterior();++                        cPosteriorIndex = 0;+                        updatePosterior();+                    }+                    // we just moved our posterior along one+                    updatePosterior();++                    break;+                case IterType::ANTERIOR:+                    // step posterior along one, and if we can't, then we're done.+                    if (++cPosteriorIndex == djSetList->size()) {+                        isEndVal = true;+                        return *this;+                    }+                    updatePosterior();++                    break;+                case IterType::ANTPOST:+                    // fixed anterior and posterior literally only points to one, so if we increment, its the+                    // end+                    isEndVal = true;+                    break;+                case IterType::WITHIN:+                    // move posterior along one+                    // see if we can't move the posterior along+                    if (++cPosteriorIndex == djSetList->size()) {+                        // move anterior along one+                        // see if we can't move the anterior along one+                        if (++cAnteriorIndex == djSetList->size()) {+                            isEndVal = true;+                            return *this;+                        }++                        // we moved our anterior along one+                        updateAnterior();++                        cPosteriorIndex = 0;+                        updatePosterior();+                    }+                    // we just moved our posterior along one+                    updatePosterior();+                    break;+            }++            return *this;+        }++    private:+        const EquivalenceRelation* br = nullptr;+        // special tombstone value to notify that this iter represents the end+        bool isEndVal = false;++        // all the different types of iterator this can be+        enum IterType { ALL, ANTERIOR, ANTPOST, WITHIN };+        IterType ityp;++        TupleType cPair;++        // the disjoint set that we're currently iterating through+        StatesBucket djSetList;+        typename StatesMap::iterator djSetMapListIt;+        typename StatesMap::iterator djSetMapListEnd;++        // used for ALL, and POSTERIOR (just a current index in the cList)+        size_t cAnteriorIndex = 0;+        // used for ALL, and ANTERIOR (just a current index in the cList)+        size_t cPosteriorIndex = 0;+    };++public:+    /**+     * iterator pointing to the beginning of the tuples, with no restrictions+     * @return the iterator that corresponds to the beginning of the binary relation+     */+    iterator begin() const {+        genAllDisjointSetLists();+        return iterator(this);+    }++    /**+     * iterator pointing to the end of the tuples+     * @return the iterator which represents the end of the binary rel+     */+    iterator end() const {+        return iterator(this, true);+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const TupleType& entry) const {+        operation_hints ctxt;+        return getBoundaries<levels>(entry, ctxt);+    }++    /**+     * Obtains a range of elements matching the prefix of the given entry up to+     * levels elements. A operation context may be provided to exploit temporal+     * locality.+     *+     * @tparam levels the length of the requested matching prefix+     * @param entry the entry to be looking for+     * @param ctxt the operation context to be utilized+     * @return the corresponding range of matching elements+     */+    template <unsigned levels>+    range<iterator> getBoundaries(const TupleType& entry, operation_hints&) const {+        // if nothing is bound => just use begin and end+        if (levels == 0) return make_range(begin(), end());++        // as disjoint set is exactly two args (equiv relation)+        // we only need to handle these cases++        if (levels == 1) {+            // need to test if the entry actually exists+            if (!sds.nodeExists(entry[0])) return make_range(end(), end());++            // return an iterator over all (entry[0], _)+            return make_range(anteriorIt(entry[0]), end());+        }++        if (levels == 2) {+            // need to test if the entry actually exists+            if (!sds.contains(entry[0], entry[1])) return make_range(end(), end());++            // if so return an iterator containing exactly that node+            return make_range(antpostit(entry[0], entry[1]), end());+        }++        std::cerr << "invalid state, cannot search for >2 arg start point in getBoundaries, in 2 arg tuple "+                     "store\n";+        throw "invalid state, cannot search for >2 arg start point in getBoundaries, in 2 arg tuple store";++        return make_range(end(), end());+    }++    /**+     * Act similar to getBoundaries. But non-static.+     * This function should be used ONLY by interpreter,+     * and its behavior is tightly coupling with InterpreterIndex.+     * Do Not rely on this interface outside the interpreter.+     *+     * @param entry the entry to be looking for+     * @return the corresponding range of matching elements+     */+    iterator lower_bound(const TupleType& entry, operation_hints&) const {+        if (entry[0] == MIN_RAM_SIGNED && entry[1] == MIN_RAM_SIGNED) {+            // Return an iterator over all tuples.+            return begin();+        }++        if (entry[0] != MIN_RAM_SIGNED && entry[1] == MIN_RAM_SIGNED) {+            // Return an iterator over all (entry[0], _)++            if (!sds.nodeExists(entry[0])) {+                return end();+            }+            return anteriorIt(entry[0]);+        }++        if (entry[0] != MIN_RAM_SIGNED && entry[1] != MIN_RAM_SIGNED) {+            // Return an iterator point to the exact same node.++            if (!sds.contains(entry[0], entry[1])) {+                return end();+            }+            return antpostit(entry[0], entry[1]);+        }++        return end();+    }++    /**+     * This function is only here in order to unify interfaces in InterpreterIndex.+     * Unlike the name suggestes, it omit the arguments and simply return the end+     * iterator of the relation.+     *+     * @param omitted+     * @return the end iterator.+     */+    iterator upper_bound(const TupleType&, operation_hints&) const {+        return end();+    }++    /**+     * Check emptiness.+     */+    bool empty() const {+        return this->size() == 0;+    }++    /**+     * Creates an iterator that generates all pairs (A, X)+     * for a given A, and X are elements within A's disjoint set.+     * @param anteriorVal: The first value of the tuple to be generated for+     * @return the iterator representing this.+     */+    iterator anteriorIt(value_type anteriorVal) const {+        genAllDisjointSetLists();++        // locate the blocklist that the anterior val resides in+        auto found = equivalencePartition.find({sds.findNode(anteriorVal), nullptr});+        assert(found != equivalencePartition.end() && "iterator called on partition that doesn't exist");++        return iterator(this, anteriorVal, (*found).second);+    }++    /**+     * Creates an iterator that generates the pair (A, B)+     * for a given A and B. If A and B don't exist, or aren't in the same set,+     * then the end() iterator is returned.+     * @param anteriorVal: the A value of the tuple+     * @param posteriorVal: the B value of the tuple+     * @return the iterator representing this+     */+    iterator antpostit(value_type anteriorVal, value_type posteriorVal) const {+        // obv if they're in diff sets, then iteration for this pair just ends.+        if (!sds.sameSet(anteriorVal, posteriorVal)) return end();++        genAllDisjointSetLists();++        // locate the blocklist that the val resides in+        auto found = equivalencePartition.find({sds.findNode(posteriorVal), nullptr});+        assert(found != equivalencePartition.end() && "iterator called on partition that doesn't exist");++        return iterator(this, anteriorVal, posteriorVal, (*found).second);+    }++    /**+     * Begin an iterator over all pairs within a single disjoint set - This is used for partition().+     * @param rep the representative of (or element within) a disjoint set of which to generate all pairs+     * @return an iterator that will generate all pairs within the disjoint set+     */+    iterator closure(value_type rep) const {+        genAllDisjointSetLists();++        // locate the blocklist that the val resides in+        auto found = equivalencePartition.find({sds.findNode(rep), nullptr});+        return iterator(this, (*found).second);+    }++    /**+     * Generate an approximate number of iterators for parallel iteration+     * The iterators returned are not necessarily equal in size, but in practise are approximately similarly+     * sized+     * Depending on the structure of the data, there can be more or less partitions returned than requested.+     * @param chunks the number of requested partitions+     * @return a list of the iterators as ranges+     */+    std::vector<souffle::range<iterator>> partition(size_t chunks) const {+        // generate all reps+        genAllDisjointSetLists();++        size_t numPairs = this->size();+        if (numPairs == 0) return {};+        if (numPairs == 1 || chunks <= 1) return {souffle::make_range(begin(), end())};++        // if there's more dj sets than requested chunks, then just return an iter per dj set+        std::vector<souffle::range<iterator>> ret;+        if (chunks <= equivalencePartition.size()) {+            for (auto& p : equivalencePartition) {+                ret.push_back(souffle::make_range(closure(p.first), end()));+            }+            return ret;+        }++        // keep it simple stupid+        // just go through and if the size of the binrel is > numpairs/chunks, then generate an anteriorIt for+        // each+        const size_t perchunk = numPairs / chunks;+        for (const auto& itp : equivalencePartition) {+            const size_t s = itp.second->size();+            if (s * s > perchunk) {+                for (const auto& i : *itp.second) {+                    ret.push_back(souffle::make_range(anteriorIt(i), end()));+                }+            } else {+                ret.push_back(souffle::make_range(closure(itp.first), end()));+            }+        }++        return ret;+    }++    iterator find(const TupleType&, operation_hints&) const {+        throw std::runtime_error("error: find() is not compatible with equivalence relations");+        return begin();+    }++    iterator find(const TupleType& t) const {+        operation_hints context;+        return find(t, context);+    }++protected:+    bool containsElement(value_type e) const {+        return this->sds.nodeExists(e);+    }++private:+    // marked as mutable due to difficulties with the const enforcement via the Relation API+    // const operations *may* safely change internal state (i.e. collapse djset forest)+    mutable souffle::SparseDisjointSet<value_type> sds;++    // read/write lock on equivalencePartition+    mutable std::shared_mutex statesLock;++    mutable StatesMap equivalencePartition;+    // whether the cache is stale+    mutable std::atomic<bool> statesMapStale;++    /**+     * Generate a cache of the sets such that they can be iterated over efficiently.+     * Each set is partitioned into a PiggyList.+     */+    void genAllDisjointSetLists() const {+        statesLock.lock();++        // no need to generate again, already done.+        if (!this->statesMapStale.load(std::memory_order_acquire)) {+            statesLock.unlock();+            return;+        }++        // btree version+        emptyPartition();++        size_t dSetSize = this->sds.ds.a_blocks.size();+        for (size_t i = 0; i < dSetSize; ++i) {+            typename TupleType::value_type sparseVal = this->sds.toSparse(i);+            parent_t rep = this->sds.findNode(sparseVal);++            StorePair p = {rep, nullptr};+            StatesList* mapList = equivalencePartition.insert(p, [&](StorePair& sp) {+                auto* r = new StatesList(1);+                sp.second = r;+                return r;+            });+            mapList->append(sparseVal);+        }++        statesMapStale.store(false, std::memory_order_release);+        statesLock.unlock();+    }+};+}  // namespace souffle
+ cbits/souffle/EventProcessor.h view
@@ -0,0 +1,572 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2018, The Souffle Developers. All rights reserved.+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file EventProcessor.h+ *+ * Declares classes for event processor that parse profile events and+ * populate the profile database+ *+ ***********************************************************************/++#pragma once++#include "ProfileDatabase.h"+#include "utility/MiscUtil.h"+#include "utility/StreamUtil.h"+#include <cassert>+#include <chrono>+#include <cstdarg>+#include <cstdint>+#include <cstdlib>+#include <iostream>+#include <map>+#include <string>+#include <vector>++namespace souffle {+namespace profile {+/**+ * Abstract Class for EventProcessor+ */+class EventProcessor {+public:+    virtual ~EventProcessor() = default;++    /** abstract interface for processing an profile event */+    virtual void process(ProfileDatabase&, const std::vector<std::string>& signature, va_list&) {+        fatal("Unknown profiling processing event: %s", join(signature, " "));+    }+};++/**+ * Event Processor Singleton+ *+ * Singleton that is the connection point for events+ */+class EventProcessorSingleton {+public:+    /** get instance */+    static EventProcessorSingleton& instance() {+        static EventProcessorSingleton singleton;+        return singleton;+    }++    /** register an event processor with its keyword */+    void registerEventProcessor(const std::string& keyword, EventProcessor* processor) {+        registry[keyword] = processor;+    }++    /** process a profile event */+    void process(ProfileDatabase& db, const char* txt, ...) {+        va_list args;+        va_start(args, txt);++        // escape signature+        std::string escapedText = escape(txt);+        // obtain event signature by splitting event text+        std::vector<std::string> eventSignature = splitSignature(escapedText);++        // invoke the event processor of the event+        const std::string& keyword = eventSignature[0];+        assert(eventSignature.size() > 0 && "no keyword in event description");+        assert(registry.find(keyword) != registry.end() && "EventProcessor not found!");+        registry[keyword]->process(db, eventSignature, args);++        // terminate access to variadic arguments+        va_end(args);+    }++private:+    /** keyword / event processor mapping */+    std::map<std::string, EventProcessor*> registry;++    EventProcessorSingleton() = default;++    /**+     * Escape escape characters.+     *+     * Remove all escapes, then escape double quotes.+     */+    std::string escape(const std::string& text) {+        std::string str(text);+        size_t start_pos = 0;+        // replace backslashes with double backslash+        while ((start_pos = str.find('\\', start_pos)) != std::string::npos) {+            if (start_pos == str.size()) {+                break;+            }+            ++start_pos;+            if (str[start_pos] == 't' || str[start_pos] == '"' || str[start_pos] == '\\' ||+                    str[start_pos] == 'n' || str[start_pos] == ';') {+                continue;+            }+            str.replace(start_pos - 1, 1, "\\\\");+            ++start_pos;+        }+        return str;+    }++    /** split string */+    static std::vector<std::string> split(std::string str, std::string split_str) {+        // repeat value when splitting so "a   b" -> ["a","b"] not ["a","","","","b"]+        bool repeat = (split_str == " ");++        std::vector<std::string> elems;++        std::string temp;+        std::string hold;+        for (size_t i = 0; i < str.size(); i++) {+            if (repeat) {+                if (str.at(i) == split_str.at(0)) {+                    while (str.at(++i) == split_str.at(0)) {+                        ;  // set i to be at the end of the search string+                    }+                    elems.push_back(temp);+                    temp = "";+                }+                temp += str.at(i);+            } else {+                temp += str.at(i);+                hold += str.at(i);+                for (size_t j = 0; j < hold.size(); j++) {+                    if (hold[j] != split_str[j]) {+                        hold = "";+                    }+                }+                if (hold.size() == split_str.size()) {+                    elems.push_back(temp.substr(0, temp.size() - hold.size()));+                    hold = "";+                    temp = "";+                }+            }+        }+        if (!temp.empty()) {+            elems.push_back(temp);+        }++        return elems;+    }++    /** split string separated by semi-colon */+    static std::vector<std::string> splitSignature(std::string str) {+        for (size_t i = 0; i < str.size(); i++) {+            if (i > 0 && str[i] == ';' && str[i - 1] == '\\') {+                // I'm assuming this isn't a thing that will be naturally found in souffle profiler files+                str[i - 1] = '\b';+                str.erase(i--, 1);+            }+        }+        std::vector<std::string> result = split(str, ";");+        for (auto& i : result) {+            for (char& j : i) {+                if (j == '\b') {+                    j = ';';+                }+            }+        }+        return result;+    }+};++/**+ * Non-Recursive Rule Timing Profile Event Processor+ */+const class NonRecursiveRuleTimingProcessor : public EventProcessor {+public:+    NonRecursiveRuleTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@t-nonrecursive-rule", this);+    }+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        const std::string& rule = signature[3];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        size_t startMaxRSS = va_arg(args, size_t);+        size_t endMaxRSS = va_arg(args, size_t);+        size_t size = va_arg(args, size_t);+        db.addSizeEntry(+                {"program", "relation", relation, "non-recursive-rule", rule, "maxRSS", "pre"}, startMaxRSS);+        db.addSizeEntry(+                {"program", "relation", relation, "non-recursive-rule", rule, "maxRSS", "post"}, endMaxRSS);+        db.addTextEntry(+                {"program", "relation", relation, "non-recursive-rule", rule, "source-locator"}, srcLocator);+        db.addDurationEntry(+                {"program", "relation", relation, "non-recursive-rule", rule, "runtime"}, start, end);+        db.addSizeEntry({"program", "relation", relation, "non-recursive-rule", rule, "num-tuples"}, size);+    }+} nonRecursiveRuleTimingProcessor;++/**+ * Non-Recursive Rule Number Profile Event Processor+ */+const class NonRecursiveRuleNumberProcessor : public EventProcessor {+public:+    NonRecursiveRuleNumberProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@n-nonrecursive-rule", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        const std::string& rule = signature[3];+        size_t num = va_arg(args, size_t);+        db.addTextEntry(+                {"program", "relation", relation, "non-recursive-rule", rule, "source-locator"}, srcLocator);+        db.addSizeEntry({"program", "relation", relation, "non-recursive-rule", rule, "num-tuples"}, num);+    }+} nonRecursiveRuleNumberProcessor;++/**+ * Recursive Rule Timing Profile Event Processor+ */+const class RecursiveRuleTimingProcessor : public EventProcessor {+public:+    RecursiveRuleTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@t-recursive-rule", this);+    }+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& version = signature[2];+        const std::string& srcLocator = signature[3];+        const std::string& rule = signature[4];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        size_t startMaxRSS = va_arg(args, size_t);+        size_t endMaxRSS = va_arg(args, size_t);+        size_t size = va_arg(args, size_t);+        std::string iteration = std::to_string(va_arg(args, size_t));+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "maxRSS", "pre"},+                startMaxRSS);+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "maxRSS", "post"},+                endMaxRSS);+        db.addTextEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "source-locator"},+                srcLocator);+        db.addDurationEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                    version, "runtime"},+                start, end);+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "num-tuples"},+                size);+    }+} recursiveRuleTimingProcessor;++/**+ * Recursive Rule Number Profile Event Processor+ */+const class RecursiveRuleNumberProcessor : public EventProcessor {+public:+    RecursiveRuleNumberProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@n-recursive-rule", this);+    }+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& version = signature[2];+        const std::string& srcLocator = signature[3];+        const std::string& rule = signature[4];+        size_t number = va_arg(args, size_t);+        std::string iteration = std::to_string(va_arg(args, size_t));+        db.addTextEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "source-locator"},+                srcLocator);+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "recursive-rule", rule,+                                version, "num-tuples"},+                number);+    }+} recursiveRuleNumberProcessor;++/**+ * Non-Recursive Relation Number Profile Event Processor+ */+const class NonRecursiveRelationTimingProcessor : public EventProcessor {+public:+    NonRecursiveRelationTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@t-nonrecursive-relation", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        size_t startMaxRSS = va_arg(args, size_t);+        size_t endMaxRSS = va_arg(args, size_t);+        size_t size = va_arg(args, size_t);+        db.addSizeEntry({"program", "relation", relation, "maxRSS", "pre"}, startMaxRSS);+        db.addSizeEntry({"program", "relation", relation, "maxRSS", "post"}, endMaxRSS);+        db.addSizeEntry({"program", "relation", relation, "num-tuples"}, size);+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addDurationEntry({"program", "relation", relation, "runtime"}, start, end);+    }+} nonRecursiveRelationTimingProcessor;++/**+ * Non-Recursive Relation Number Profile Event Processor+ */+const class NonRecursiveRelationNumberProcessor : public EventProcessor {+public:+    NonRecursiveRelationNumberProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@n-nonrecursive-relation", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        size_t num = va_arg(args, size_t);+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addSizeEntry({"program", "relation", relation, "num-tuples"}, num);+    }+} nonRecursiveRelationNumberProcessor;++/**+ * Recursive Relation Timing Profile Event Processor+ */+const class RecursiveRelationTimingProcessor : public EventProcessor {+public:+    RecursiveRelationTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@t-recursive-relation", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        size_t startMaxRSS = va_arg(args, size_t);+        size_t endMaxRSS = va_arg(args, size_t);+        size_t size = va_arg(args, size_t);+        std::string iteration = std::to_string(va_arg(args, size_t));+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addDurationEntry({"program", "relation", relation, "iteration", iteration, "runtime"}, start, end);+        db.addSizeEntry(+                {"program", "relation", relation, "iteration", iteration, "maxRSS", "pre"}, startMaxRSS);+        db.addSizeEntry(+                {"program", "relation", relation, "iteration", iteration, "maxRSS", "post"}, endMaxRSS);+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "num-tuples"}, size);+    }+} recursiveRelationTimingProcessor;++/**+ * Recursive Relation Timing Profile Event Processor+ */+const class RecursiveRelationNumberProcessor : public EventProcessor {+public:+    RecursiveRelationNumberProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@n-recursive-relation", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        size_t number = va_arg(args, size_t);+        std::string iteration = std::to_string(va_arg(args, size_t));+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addSizeEntry({"program", "relation", relation, "iteration", iteration, "num-tuples"}, number);+    }+} recursiveRelationNumberProcessor;++/**+ * Recursive Relation Copy Timing Profile Event Processor+ */+const class RecursiveRelationCopyTimingProcessor : public EventProcessor {+public:+    RecursiveRelationCopyTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@c-recursive-relation", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        size_t startMaxRSS = va_arg(args, size_t);+        size_t endMaxRSS = va_arg(args, size_t);+        va_arg(args, size_t);+        std::string iteration = std::to_string(va_arg(args, size_t));+        db.addSizeEntry(+                {"program", "relation", relation, "iteration", iteration, "maxRSS", "pre"}, startMaxRSS);+        db.addSizeEntry(+                {"program", "relation", relation, "iteration", iteration, "maxRSS", "post"}, endMaxRSS);+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addDurationEntry(+                {"program", "relation", relation, "iteration", iteration, "copytime"}, start, end);+    }+} recursiveRelationCopyTimingProcessor;++/**+ * Recursive Relation Copy Timing Profile Event Processor+ */+const class RelationIOTimingProcessor : public EventProcessor {+public:+    RelationIOTimingProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@t-relation-savetime", this);+        EventProcessorSingleton::instance().registerEventProcessor("@t-relation-loadtime", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& srcLocator = signature[2];+        const std::string ioType = signature[3];+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        db.addTextEntry({"program", "relation", relation, "source-locator"}, srcLocator);+        db.addDurationEntry({"program", "relation", relation, ioType}, start, end);+    }+} relationIOTimingProcessor;++/**+ * Program Run Event Processor+ */+const class ProgramTimepointProcessor : public EventProcessor {+public:+    ProgramTimepointProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@time", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        microseconds time = va_arg(args, microseconds);+        auto path = signature;+        path[0] = "program";+        db.addTimeEntry(path, time);+    }+} programTimepointProcessor;++/**+ * Program Run Event Processor+ */+const class ProgramRuntimeProcessor : public EventProcessor {+public:+    ProgramRuntimeProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@runtime", this);+    }+    /** process event input */+    void process(+            ProfileDatabase& db, const std::vector<std::string>& /* signature */, va_list& args) override {+        microseconds start = va_arg(args, microseconds);+        microseconds end = va_arg(args, microseconds);+        db.addDurationEntry({"program", "runtime"}, start, end);+    }+} programRuntimeProcessor;++/**+ * Program Resource Utilisation Event Processor+ */+const class ProgramResourceUtilisationProcessor : public EventProcessor {+public:+    ProgramResourceUtilisationProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@utilisation", this);+    }+    /** process event input */+    void process(+            ProfileDatabase& db, const std::vector<std::string>& /* signature */, va_list& args) override {+        microseconds time = va_arg(args, microseconds);+        uint64_t systemTime = va_arg(args, uint64_t);+        uint64_t userTime = va_arg(args, uint64_t);+        size_t maxRSS = va_arg(args, size_t);+        std::string timeString = std::to_string(time.count());+        db.addSizeEntry({"program", "usage", "timepoint", timeString, "systemtime"}, systemTime);+        db.addSizeEntry({"program", "usage", "timepoint", timeString, "usertime"}, userTime);+        db.addSizeEntry({"program", "usage", "timepoint", timeString, "maxRSS"}, maxRSS);+    }+} programResourceUtilisationProcessor;++/**+ * Frequency Atom Processor+ */+const class FrequencyAtomProcessor : public EventProcessor {+public:+    FrequencyAtomProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@frequency-atom", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        const std::string& version = signature[2];+        const std::string& rule = signature[3];+        const std::string& atom = signature[4];+        const std::string& originalRule = signature[5];+        size_t level = std::stoi(signature[6]);+        size_t number = va_arg(args, size_t);+        size_t iteration = va_arg(args, size_t);+        // non-recursive rule+        if (rule == originalRule) {+            db.addSizeEntry({"program", "relation", relation, "non-recursive-rule", rule, "atom-frequency",+                                    rule, atom, "level"},+                    level);+            db.addSizeEntry({"program", "relation", relation, "non-recursive-rule", rule, "atom-frequency",+                                    rule, atom, "num-tuples"},+                    number);+        } else {+            db.addSizeEntry(+                    {"program", "relation", relation, "iteration", std::to_string(iteration),+                            "recursive-rule", originalRule, version, "atom-frequency", rule, atom, "level"},+                    level);+            db.addSizeEntry({"program", "relation", relation, "iteration", std::to_string(iteration),+                                    "recursive-rule", originalRule, version, "atom-frequency", rule, atom,+                                    "num-tuples"},+                    number);+        }+    }+} frequencyAtomProcessor;++/**+ * Reads Processor+ */+const class RelationReadsProcessor : public EventProcessor {+public:+    RelationReadsProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@relation-reads", this);+    }+    /** process event input */+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string& relation = signature[1];+        size_t reads = va_arg(args, size_t);+        db.addSizeEntry({"program", "relation", relation, "reads"}, reads);+    }++} relationReadsProcessor;++/**+ * Config entry processor+ */+const class ConfigProcessor : public EventProcessor {+public:+    ConfigProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@config", this);+    }+    void process(+            ProfileDatabase& db, const std::vector<std::string>& /* signature */, va_list& args) override {+        const std::string key = va_arg(args, char*);+        const std::string& value = va_arg(args, char*);+        db.addTextEntry({"program", "configuration", key}, value);+    }+} configProcessor;++/**+ * Text entry processor+ */+const class TextProcessor : public EventProcessor {+public:+    TextProcessor() {+        EventProcessorSingleton::instance().registerEventProcessor("@text", this);+    }+    void process(ProfileDatabase& db, const std::vector<std::string>& signature, va_list& args) override {+        const std::string text = va_arg(args, char*);+        auto path = signature;+        path.front() = "program";+        db.addTextEntry(path, text);+    }+} textProcessor;++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/IOSystem.h view
@@ -0,0 +1,92 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file IOSystem.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "ReadStream.h"+#include "ReadStreamCSV.h"+#include "SymbolTable.h"+#include "WriteStream.h"+#include "WriteStreamCSV.h"++#ifdef USE_SQLITE+#include "ReadStreamSQLite.h"+#include "WriteStreamSQLite.h"+#endif++#include <map>+#include <memory>+#include <stdexcept>+#include <string>++namespace souffle {+class RecordTable;++class IOSystem {+public:+    static IOSystem& getInstance() {+        static IOSystem singleton;+        return singleton;+    }++    void registerWriteStreamFactory(const std::shared_ptr<WriteStreamFactory>& factory) {+        outputFactories[factory->getName()] = factory;+    }++    void registerReadStreamFactory(const std::shared_ptr<ReadStreamFactory>& factory) {+        inputFactories[factory->getName()] = factory;+    }++    /**+     * Return a new WriteStream+     */+    std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable& symbolTable, const RecordTable& recordTable) const {+        std::string ioType = rwOperation.at("IO");+        if (outputFactories.count(ioType) == 0) {+            throw std::invalid_argument("Requested output type <" + ioType + "> is not supported.");+        }+        return outputFactories.at(ioType)->getWriter(rwOperation, symbolTable, recordTable);+    }+    /**+     * Return a new ReadStream+     */+    std::unique_ptr<ReadStream> getReader(const std::map<std::string, std::string>& rwOperation,+            SymbolTable& symbolTable, RecordTable& recordTable) const {+        std::string ioType = rwOperation.at("IO");+        if (inputFactories.count(ioType) == 0) {+            throw std::invalid_argument("Requested input type <" + ioType + "> is not supported.");+        }+        return inputFactories.at(ioType)->getReader(rwOperation, symbolTable, recordTable);+    }+    ~IOSystem() = default;++private:+    IOSystem() {+        registerReadStreamFactory(std::make_shared<ReadFileCSVFactory>());+        registerReadStreamFactory(std::make_shared<ReadCinCSVFactory>());+        registerWriteStreamFactory(std::make_shared<WriteFileCSVFactory>());+        registerWriteStreamFactory(std::make_shared<WriteCoutCSVFactory>());+        registerWriteStreamFactory(std::make_shared<WriteCoutPrintSizeFactory>());+#ifdef USE_SQLITE+        registerReadStreamFactory(std::make_shared<ReadSQLiteFactory>());+        registerWriteStreamFactory(std::make_shared<WriteSQLiteFactory>());+#endif+    };+    std::map<std::string, std::shared_ptr<WriteStreamFactory>> outputFactories;+    std::map<std::string, std::shared_ptr<ReadStreamFactory>> inputFactories;+};++} /* namespace souffle */
+ cbits/souffle/IterUtils.h view
@@ -0,0 +1,137 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file IterUtils.h+ *+ * A (growing) collection of generic iterator utilities.+ *+ ***********************************************************************/++#pragma once++#include <iterator>+#include <type_traits>++namespace souffle {++/**+ * A wrapper for an iterator obtaining pointers of a certain type,+ * dereferencing values before forwarding them to the consumer.+ *+ * @tparam Iter ... the type of wrapped iterator+ * @tparam T    ... the value to be accessed by the resulting iterator+ */+template <typename Iter, typename T = typename std::remove_pointer<typename Iter::value_type>::type>+struct IterDerefWrapper : public std::iterator<std::forward_iterator_tag, T> {+    /* The nested iterator. */+    Iter iter;++public:+    // some constructores+    IterDerefWrapper() = default;+    IterDerefWrapper(const Iter& iter) : iter(iter) {}++    // defaulted copy and move constructors+    IterDerefWrapper(const IterDerefWrapper&) = default;+    IterDerefWrapper(IterDerefWrapper&&) = default;++    // default assignment operators+    IterDerefWrapper& operator=(const IterDerefWrapper&) = default;+    IterDerefWrapper& operator=(IterDerefWrapper&&) = default;++    /* The equality operator as required by the iterator concept. */+    bool operator==(const IterDerefWrapper& other) const {+        return iter == other.iter;+    }++    /* The not-equality operator as required by the iterator concept. */+    bool operator!=(const IterDerefWrapper& other) const {+        return iter != other.iter;+    }++    /* The deref operator as required by the iterator concept. */+    const T& operator*() const {+        return **iter;+    }++    /* Support for the pointer operator. */+    const T* operator->() const {+        return &(**iter);+    }++    /* The increment operator as required by the iterator concept. */+    IterDerefWrapper& operator++() {+        ++iter;+        return *this;+    }+};++/**+ * A factory function enabling the construction of a dereferencing+ * iterator utilizing the automated deduction of template parameters.+ */+template <typename Iter>+IterDerefWrapper<Iter> derefIter(const Iter& iter) {+    return IterDerefWrapper<Iter>(iter);+}++// ---------------------------------------------------------------------+//                        Single-Value-Iterator+// ---------------------------------------------------------------------++/**+ * An iterator to be utilized if there is only a single element to iterate over.+ */+template <typename T>+class SingleValueIterator : public std::iterator<std::forward_iterator_tag, T> {+    T value;++    bool end = true;++public:+    SingleValueIterator() = default;++    SingleValueIterator(const T& value) : value(value), end(false) {}++    // a copy constructor+    SingleValueIterator(const SingleValueIterator& other) = default;++    // an assignment operator+    SingleValueIterator& operator=(const SingleValueIterator& other) = default;++    // the equality operator as required by the iterator concept+    bool operator==(const SingleValueIterator& other) const {+        // only equivalent if pointing to the end+        return end && other.end;+    }++    // the not-equality operator as required by the iterator concept+    bool operator!=(const SingleValueIterator& other) const {+        return !(*this == other);+    }++    // the deref operator as required by the iterator concept+    const T& operator*() const {+        return value;+    }++    // support for the pointer operator+    const T* operator->() const {+        return &value;+    }++    // the increment operator as required by the iterator concept+    SingleValueIterator& operator++() {+        end = true;+        return *this;+    }+};++}  // end of namespace souffle
+ cbits/souffle/LICENSE view
@@ -0,0 +1,25 @@+Copyright (c) 2016 Oracle and/or its affiliates. All Rights reserved++The Universal Permissive License (UPL), Version 1.0++Subject to the condition set forth below, permission is hereby granted to any person obtaining a copy of this software,+associated documentation and/or data (collectively the "Software"), free of charge and under any and all copyright rights in the +Software, and any and all patent rights owned or freely licensable by each licensor hereunder covering either (i) the unmodified +Software as contributed to or provided by such licensor, or (ii) the Larger Works (as defined below), to deal in both++(a) the Software, and+(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if one is included with the Software (each a “Larger+Work” to which the Software is contributed by such licensors),++without restriction, including without limitation the rights to copy, create derivative works of, display, perform, and +distribute the Software and make, use, sell, offer for sale, import, export, have made, and have sold the Software and the +Larger Work(s), and to sublicense the foregoing rights on either these or other terms.++This license is subject to the following condition:+The above copyright notice and either this complete permission notice or at a minimum a reference to the UPL must be included in +all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR+IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ cbits/souffle/LambdaBTree.h view
@@ -0,0 +1,620 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2018, Souffle Developers+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file LambdaBTree.h+ *+ * An implementation of a generic B-tree data structure including+ * interfaces for utilizing instances as set or multiset containers.+ * Allows the user to provide a function to execute on successful insert+ * Be careful using this, it currently expects a pair as the key.+ *+ ***********************************************************************/++#pragma once++#include "BTree.h"+#include "utility/ContainerUtil.h"+#include "utility/ParallelUtil.h"+#include <atomic>+#include <cassert>+#include <typeinfo>+#include <vector>++namespace souffle {++namespace detail {+/**+ * The actual implementation of a b-tree data structure.+ *+ * @tparam Key             .. the element type to be stored in this tree+ * @tparam Comparator     .. a class defining an order on the stored elements+ * @tparam Allocator     .. utilized for allocating memory for required nodes+ * @tparam blockSize    .. determines the number of bytes/block utilized by leaf nodes+ * @tparam SearchStrategy .. enables switching between linear, binary or any other search strategy+ * @tparam isSet        .. true = set, false = multiset+ * @tparam Functor      .. a std::function that is called on successful (new) insert+ */+template <typename Key, typename Comparator,+        typename Allocator,  // is ignored so far - TODO: add support+        unsigned blockSize, typename SearchStrategy, bool isSet, typename Functor,+        typename WeakComparator = Comparator, typename Updater = detail::updater<Key>>+class LambdaBTree : public btree<Key, Comparator, Allocator, blockSize, SearchStrategy, isSet, WeakComparator,+                            Updater> {+public:+    using parenttype =+            btree<Key, Comparator, Allocator, blockSize, SearchStrategy, isSet, WeakComparator, Updater>;++    LambdaBTree(const Comparator& comp = Comparator(), const WeakComparator& weak_comp = WeakComparator())+            : parenttype(comp, weak_comp) {}++    /**+     * Inserts the given key into this tree.+     */+    typename Functor::result_type insert(Key& k, const Functor& f) {+        typename parenttype::operation_hints hints;+        return insert(k, hints, f);+    }++    // rewriting this because of david's changes+    typename Functor::result_type insert(+            Key& k, typename parenttype::operation_hints& hints, const Functor& f) {+#ifdef IS_PARALLEL++        // special handling for inserting first element+        while (this->root == nullptr) {+            // try obtaining root-lock+            if (!this->root_lock.try_start_write()) {+                // somebody else was faster => re-check+                continue;+            }++            // check loop condition again+            if (this->root != nullptr) {+                // somebody else was faster => normal insert+                this->root_lock.abort_write();+                break;+            }++            // create new node+            this->leftmost = new typename parenttype::leaf_node();+            this->leftmost->numElements = 1;+            // call the functor as we've successfully inserted+            typename Functor::result_type res = f(k);++            this->leftmost->keys[0] = k;+            this->root = this->leftmost;++            // operation complete => we can release the root lock+            this->root_lock.end_write();++            hints.last_insert.access(this->leftmost);++            return res;+        }++        // insert using iterative implementation++        typename parenttype::node* cur = nullptr;++        // test last insert hints+        typename parenttype::lock_type::Lease cur_lease;++        auto checkHint = [&](typename parenttype::node* last_insert) {+            // ignore null pointer+            if (!last_insert) return false;+            // get a read lease on indicated node+            auto hint_lease = last_insert->lock.start_read();+            // check whether it covers the key+            if (!this->weak_covers(last_insert, k)) return false;+            // and if there was no concurrent modification+            if (!last_insert->lock.validate(hint_lease)) return false;+            // use hinted location+            cur = last_insert;+            // and keep lease+            cur_lease = hint_lease;+            // we found a hit+            return true;+        };++        if (hints.last_insert.any(checkHint)) {+            // register this as a hit+            this->hint_stats.inserts.addHit();+        } else {+            // register this as a miss+            this->hint_stats.inserts.addMiss();+        }++        // if there is no valid hint ..+        if (!cur) {+            do {+                // get root - access lock+                auto root_lease = this->root_lock.start_read();++                // start with root+                cur = this->root;++                // get lease of the next node to be accessed+                cur_lease = cur->lock.start_read();++                // check validity of root pointer+                if (this->root_lock.end_read(root_lease)) {+                    break;+                }++            } while (true);+        }++        while (true) {+            // handle inner nodes+            if (cur->inner) {+                auto a = &(cur->keys[0]);+                auto b = &(cur->keys[cur->numElements]);++                auto pos = this->search.lower_bound(k, a, b, this->weak_comp);+                auto idx = pos - a;++                // early exit for sets+                if (isSet && pos != b && this->weak_equal(*pos, k)) {+                    // validate results+                    if (!cur->lock.validate(cur_lease)) {+                        // start over again+                        return insert(k, hints, f);+                    }++                    // update provenance information+                    if (typeid(Comparator) != typeid(WeakComparator) && this->less(k, *pos)) {+                        if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                            // start again+                            return insert(k, hints, f);+                        }+                        this->update(*pos, k);++                        // get result before releasing lock+                        auto res = (*pos).second;++                        cur->lock.end_write();+                        return res;+                    }++                    // get the result before releasing lock+                    auto res = (*pos).second;++                    // check validity+                    if (!cur->lock.validate(cur_lease)) {+                        // start over again+                        return insert(k, hints, f);+                    }++                    // we found the element => return the result+                    return res;+                }++                // get next pointer+                auto next = cur->getChild(idx);++                // get lease on next level+                auto next_lease = next->lock.start_read();++                // check whether there was a write+                if (!cur->lock.end_read(cur_lease)) {+                    // start over+                    return insert(k, hints, f);+                }++                // go to next+                cur = next;++                // move on lease+                cur_lease = next_lease;++                continue;+            }++            // the rest is for leaf nodes+            assert(!cur->inner);++            // -- insert node in leaf node --++            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = this->search.upper_bound(k, a, b, this->weak_comp);+            auto idx = pos - a;++            // early exit for sets+            if (isSet && pos != a && this->weak_equal(*(pos - 1), k)) {+                // validate result+                if (!cur->lock.validate(cur_lease)) {+                    // start over again+                    return insert(k, hints, f);+                }++                // TODO (pnappa): remove provenance from LambdaBTree - no use for it+                // update provenance information+                if (typeid(Comparator) != typeid(WeakComparator) && this->less(k, *(pos - 1))) {+                    if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                        // start again+                        return insert(k, hints, f);+                    }+                    this->update(*(pos - 1), k);++                    // retrieve result before releasing lock+                    auto res = (*(pos - 1)).second;++                    cur->lock.end_write();+                    return res;+                }++                // read result (atomic) -- just as a proof of concept, this is actually not valid!!+                std::atomic<typename Functor::result_type>& loc =+                        *reinterpret_cast<std::atomic<typename Functor::result_type>*>(&(*(pos - 1)).second);+                auto res = loc.load(std::memory_order_relaxed);++                // check validity+                if (!cur->lock.validate(cur_lease)) {+                    // start over again+                    return insert(k, hints, f);+                }++                // we found the element => done+                return res;+            }++            // upgrade to write-permission+            if (!cur->lock.try_upgrade_to_write(cur_lease)) {+                // something has changed => restart+                hints.last_insert.access(cur);+                return insert(k, hints, f);+            }++            if (cur->numElements >= parenttype::node::maxKeys) {+                // -- lock parents --+                auto priv = cur;+                auto parent = priv->parent;+                std::vector<typename parenttype::node*> parents;+                do {+                    if (parent) {+                        parent->lock.start_write();+                        while (true) {+                            // check whether parent is correct+                            if (parent == priv->parent) {+                                break;+                            }+                            // switch parent+                            parent->lock.abort_write();+                            parent = priv->parent;+                            parent->lock.start_write();+                        }+                    } else {+                        // lock root lock => since cur is root+                        this->root_lock.start_write();+                    }++                    // record locked node+                    parents.push_back(parent);++                    // stop at "sphere of influence"+                    if (!parent || !parent->isFull()) {+                        break;+                    }++                    // go one step higher+                    priv = parent;+                    parent = parent->parent;++                } while (true);++                // split this node+                auto old_root = this->root;+                idx -= cur->rebalance_or_split(+                        const_cast<typename parenttype::node**>(&this->root), this->root_lock, idx, parents);++                // release parent lock+                for (auto it = parents.rbegin(); it != parents.rend(); ++it) {+                    auto parent = *it;++                    // release this lock+                    if (parent) {+                        parent->lock.end_write();+                    } else {+                        if (old_root != this->root) {+                            this->root_lock.end_write();+                        } else {+                            this->root_lock.abort_write();+                        }+                    }+                }++                // insert element in right fragment+                if (((typename parenttype::size_type)idx) > cur->numElements) {+                    // release current lock+                    cur->lock.end_write();++                    // insert in sibling+                    return insert(k, hints, f);+                }+            }++            // ok - no split necessary+            assert(cur->numElements < parenttype::node::maxKeys && "Split required!");++            // move keys+            for (int j = cur->numElements; j > idx; --j) {+                cur->keys[j] = cur->keys[j - 1];+            }++            // insert new element+            typename Functor::result_type res = f(k);+            cur->keys[idx] = k;+            cur->numElements++;++            // release lock on current node+            cur->lock.end_write();++            // remember last insertion position+            hints.last_insert.access(cur);+            return res;+        }++#else+        // special handling for inserting first element+        if (this->empty()) {+            // create new node+            this->leftmost = new typename parenttype::leaf_node();+            this->leftmost->numElements = 1;+            // call the functor as we've successfully inserted+            typename Functor::result_type res = f(k);+            this->leftmost->keys[0] = k;+            this->root = this->leftmost;++            hints.last_insert.access(this->leftmost);++            return res;+        }++        // insert using iterative implementation+        typename parenttype::node* cur = this->root;++        auto checkHints = [&](typename parenttype::node* last_insert) {+            if (!last_insert) return false;+            if (!this->weak_covers(last_insert, k)) return false;+            cur = last_insert;+            return true;+        };++        // test last insert+        if (hints.last_insert.any(checkHints)) {+            this->hint_stats.inserts.addHit();+        } else {+            this->hint_stats.inserts.addMiss();+        }++        while (true) {+            // handle inner nodes+            if (cur->inner) {+                auto a = &(cur->keys[0]);+                auto b = &(cur->keys[cur->numElements]);++                auto pos = this->search.lower_bound(k, a, b, this->weak_comp);+                auto idx = pos - a;++                // early exit for sets+                if (isSet && pos != b && this->weak_equal(*pos, k)) {+                    // update provenance information+                    if (typeid(Comparator) != typeid(WeakComparator) && this->less(k, *pos)) {+                        this->update(*pos, k);+                        return (*pos).second;+                    }++                    return (*pos).second;+                }++                cur = cur->getChild(idx);+                continue;+            }++            // the rest is for leaf nodes+            assert(!cur->inner);++            // -- insert node in leaf node --++            auto a = &(cur->keys[0]);+            auto b = &(cur->keys[cur->numElements]);++            auto pos = this->search.upper_bound(k, a, b, this->weak_comp);+            auto idx = pos - a;++            // early exit for sets+            if (isSet && pos != a && this->weak_equal(*(pos - 1), k)) {+                // update provenance information+                if (typeid(Comparator) != typeid(WeakComparator) && this->less(k, *(pos - 1))) {+                    this->update(*(pos - 1), k);+                    return (*(pos - 1)).second;+                }++                return (*(pos - 1)).second;+            }++            if (cur->numElements >= parenttype::node::maxKeys) {+                // split this node+                idx -= cur->rebalance_or_split(+                        const_cast<typename parenttype::node**>(&this->root), this->root_lock, idx);++                // insert element in right fragment+                if (((typename parenttype::size_type)idx) > cur->numElements) {+                    idx -= cur->numElements + 1;+                    cur = cur->parent->getChild(cur->position + 1);+                }+            }++            // ok - no split necessary+            assert(cur->numElements < parenttype::node::maxKeys && "Split required!");++            // move keys+            for (int j = cur->numElements; j > idx; --j) {+                cur->keys[j] = cur->keys[j - 1];+            }++            // call the functor as we've successfully inserted+            typename Functor::result_type res = f(k);+            // insert new element+            cur->keys[idx] = k;+            cur->numElements++;++            // remember last insertion position+            hints.last_insert.access(cur);+            return res;+        }+#endif+    }++    /**+     * Inserts the given range of elements into this tree.+     */+    template <typename Iter>+    void insert(const Iter& a, const Iter& b) {+        // TODO: improve this beyond a naive insert+        typename parenttype::operation_hints hints;+        // a naive insert so far .. seems to work fine+        for (auto it = a; it != b; ++it) {+            // use insert with hint+            insert(*it, hints);+        }+    }++    /**+     * Swaps the content of this tree with the given tree. This+     * is a much more efficient operation than creating a copy and+     * realizing the swap utilizing assignment operations.+     */+    void swap(LambdaBTree& other) {+        // swap the content+        std::swap(this->root, other.root);+        std::swap(this->leftmost, other.leftmost);+    }++    // Implementation of the assignment operation for trees.+    LambdaBTree& operator=(const LambdaBTree& other) {+        // check identity+        if (this == &other) {+            return *this;+        }++        // create a deep-copy of the content of the other tree+        // shortcut for empty sets+        if (other.empty()) {+            return *this;+        }++        // clone content (deep copy)+        this->root = other.root->clone();++        // update leftmost reference+        auto tmp = this->root;+        while (!tmp->isLeaf()) {+            tmp = tmp->getChild(0);+        }+        this->leftmost = static_cast<typename parenttype::leaf_node*>(tmp);++        // done+        return *this;+    }++    // Implementation of an equality operation for trees.+    bool operator==(const LambdaBTree& other) const {+        // check identity+        if (this == &other) {+            return true;+        }++        // check size+        if (this->size() != other.size()) {+            return false;+        }+        if (this->size() < other.size()) {+            return other == *this;+        }++        // check content+        for (const auto& key : other) {+            if (!contains(key)) {+                return false;+            }+        }+        return true;+    }++    // Implementation of an inequality operation for trees.+    bool operator!=(const LambdaBTree& other) const {+        return !(*this == other);+    }+};++}  // end namespace detail++/**+ * A b-tree based set implementation.+ *+ * @tparam Key             .. the element type to be stored in this set+ * @tparam Functor         .. a std::function that is invoked on successful insert+ * @tparam Comparator     .. a class defining an order on the stored elements+ * @tparam Allocator     .. utilized for allocating memory for required nodes+ * @tparam blockSize    .. determines the number of bytes/block utilized by leaf nodes+ * @tparam SearchStrategy .. enables switching between linear, binary or any other search strategy+ */+template <typename Key, typename Functor, typename Comparator = detail::comparator<Key>,+        typename Allocator = std::allocator<Key>,  // is ignored so far+        unsigned blockSize = 256, typename SearchStrategy = typename detail::default_strategy<Key>::type>+class LambdaBTreeSet+        : public detail::LambdaBTree<Key, Comparator, Allocator, blockSize, SearchStrategy, true, Functor> {+    using super = detail::LambdaBTree<Key, Comparator, Allocator, blockSize, SearchStrategy, true, Functor>;++    friend class detail::LambdaBTree<Key, Comparator, Allocator, blockSize, SearchStrategy, true, Functor>;++public:+    /**+     * A default constructor creating an empty set.+     */+    LambdaBTreeSet(const Comparator& comp = Comparator()) : super(comp) {}++    /**+     * A constructor creating a set based on the given range.+     */+    template <typename Iter>+    LambdaBTreeSet(const Iter& a, const Iter& b) {+        this->insert(a, b);+    }++    // A copy constructor.+    LambdaBTreeSet(const LambdaBTreeSet& other) : super(other) {}++    // A move constructor.+    LambdaBTreeSet(LambdaBTreeSet&& other) : super(std::move(other)) {}++private:+    // A constructor required by the bulk-load facility.+    template <typename s, typename n, typename l>+    LambdaBTreeSet(s size, n* root, l* leftmost) : super::parenttype(size, root, leftmost) {}++public:+    // Support for the assignment operator.+    LambdaBTreeSet& operator=(const LambdaBTreeSet& other) {+        super::operator=(other);+        return *this;+    }++    // Support for the bulk-load operator.+    template <typename Iter>+    static LambdaBTreeSet load(const Iter& a, const Iter& b) {+        return super::template load<LambdaBTreeSet>(a, b);+    }+};++}  // end of namespace souffle
+ cbits/souffle/Logger.h view
@@ -0,0 +1,67 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file Logger.h+ *+ * A logger is the utility utilized by RAM programs to create logs and+ * traces.+ *+ ***********************************************************************/++#pragma once++#include "ProfileEvent.h"+#include "utility/MiscUtil.h"+#include <cstddef>+#include <functional>+#include <string>+#include <utility>+#include <sys/resource.h>++namespace souffle {++/**+ * The class utilized to times for the souffle profiling tool. This class+ * is utilized by both -- the interpreted and compiled version -- to conduct+ * the corresponding measurements.+ *+ * To far, only execution times are logged. More events, e.g. the number of+ * processed tuples may be added in the future.+ */+class Logger {+public:+    Logger(std::string label, size_t iteration) : Logger(label, iteration, []() { return 0; }) {}++    Logger(std::string label, size_t iteration, std::function<size_t()> size)+            : label(std::move(label)), start(now()), iteration(iteration), size(size), preSize(size()) {+        struct rusage ru {};+        getrusage(RUSAGE_SELF, &ru);+        startMaxRSS = ru.ru_maxrss;+        // Assume that if we are logging the progress of an event then we care about usage during that time.+        ProfileEventSingleton::instance().resetTimerInterval();+    }++    ~Logger() {+        struct rusage ru {};+        getrusage(RUSAGE_SELF, &ru);+        size_t endMaxRSS = ru.ru_maxrss;+        ProfileEventSingleton::instance().makeTimingEvent(+                label, start, now(), startMaxRSS, endMaxRSS, size() - preSize, iteration);+    }++private:+    std::string label;+    time_point start;+    size_t startMaxRSS;+    size_t iteration;+    std::function<size_t()> size;+    size_t preSize;+};+}  // end of namespace souffle
+ cbits/souffle/PiggyList.h view
@@ -0,0 +1,327 @@+#pragma once++#include "utility/ParallelUtil.h"+#include <array>+#include <atomic>+#include <cstring>+#include <iostream>+#include <iterator>++#ifdef _WIN32+/**+ * MSVC does not provide a builtin for counting leading zeroes like gcc,+ * so we have to implement it ourselves.+ */+unsigned long __inline __builtin_clzll(unsigned long long value) {+    unsigned long msb = 0;++    if (_BitScanReverse64(&msb, value))+        return 63 - msb;+    else+        return 64;+}+#endif  // _WIN32++using std::size_t;+namespace souffle {++/**+ * A PiggyList that allows insertAt functionality.+ * This means we can't append, as we don't know the next available element.+ * insertAt is dangerous. You must be careful not to call it for the same index twice!+ */+template <class T>+class RandomInsertPiggyList {+public:+    RandomInsertPiggyList() = default;+    // an instance where the initial size is not 65k, and instead is user settable (to a power of+    // initialbitsize)+    RandomInsertPiggyList(size_t initialbitsize) : BLOCKBITS(initialbitsize) {}++    /** copy constructor */+    RandomInsertPiggyList(const RandomInsertPiggyList& other) : BLOCKBITS(other.BLOCKBITS) {+        this->numElements.store(other.numElements.load());++        // copy blocks from the old lookup table to this one+        for (size_t i = 0; i < maxContainers; ++i) {+            if (other.blockLookupTable[i].load() != nullptr) {+                // calculate the size of that block+                const size_t blockSize = INITIALBLOCKSIZE << i;++                // allocate that in the new container+                this->blockLookupTable[i].store(new T[blockSize]);++                // then copy the stuff over+                std::memcpy(this->blockLookupTable[i].load(), other.blockLookupTable[i].load(),+                        blockSize * sizeof(T));+            }+        }+    }++    // move ctr+    RandomInsertPiggyList(RandomInsertPiggyList&& other) = delete;+    // copy assign ctor+    RandomInsertPiggyList& operator=(RandomInsertPiggyList& other) = delete;+    // move assign ctor+    RandomInsertPiggyList& operator=(RandomInsertPiggyList&& other) = delete;++    ~RandomInsertPiggyList() {+        freeList();+    }++    inline size_t size() const {+        return numElements.load();+    }++    inline T* getBlock(size_t blockNum) const {+        return blockLookupTable[blockNum];+    }++    inline T& get(size_t index) const {+        size_t nindex = index + INITIALBLOCKSIZE;+        size_t blockNum = (63 - __builtin_clzll(nindex));+        size_t blockInd = (nindex) & ((1 << blockNum) - 1);+        return this->getBlock(blockNum - BLOCKBITS)[blockInd];+    }++    void insertAt(size_t index, T value) {+        // starting with an initial blocksize requires some shifting to transform into a nice powers of two+        // series+        size_t blockNum = (63 - __builtin_clzll(index + INITIALBLOCKSIZE)) - BLOCKBITS;++        // allocate the block if not allocated+        if (blockLookupTable[blockNum].load() == nullptr) {+            slock.lock();+            if (blockLookupTable[blockNum].load() == nullptr) {+                blockLookupTable[blockNum].store(new T[INITIALBLOCKSIZE << blockNum]);+            }+            slock.unlock();+        }++        this->get(index) = value;+        // we ALWAYS increment size, even if there was something there before (its impossible to tell!)+        // the onus is up to the user to not call this for an index twice+        ++numElements;+    }++    void clear() {+        freeList();+        numElements.store(0);+    }+    const size_t BLOCKBITS = 16ul;+    const size_t INITIALBLOCKSIZE = (1ul << BLOCKBITS);++    // number of elements currently stored within+    std::atomic<size_t> numElements{0};++    // 2^64 - 1 elements can be stored (default initialised to nullptrs)+    static constexpr size_t maxContainers = 64;+    std::array<std::atomic<T*>, maxContainers> blockLookupTable = {};++    // for parallel node insertions+    mutable SpinLock slock;++    /**+     * Free the arrays allocated within the linked list nodes+     */+    void freeList() {+        slock.lock();+        // delete all - deleting a nullptr is a no-op+        for (size_t i = 0; i < maxContainers; ++i) {+            delete[] blockLookupTable[i].load();+            // reset the container within to be empty.+            blockLookupTable[i].store(nullptr);+        }+        slock.unlock();+    }+};++template <class T>+class PiggyList {+public:+    PiggyList() : num_containers(0), container_size(0), m_size(0) {}+    PiggyList(size_t initialbitsize)+            : BLOCKBITS(initialbitsize), num_containers(0), container_size(0), m_size(0) {}++    /** copy constructor */+    PiggyList(const PiggyList& other) : BLOCKBITS(other.BLOCKBITS) {+        num_containers.store(other.num_containers.load());+        container_size.store(other.container_size.load());+        m_size.store(other.m_size.load());+        // copy each chunk from other into this+        // the size of the next container to allocate+        size_t cSize = BLOCKSIZE;+        for (size_t i = 0; i < other.num_containers; ++i) {+            this->blockLookupTable[i] = new T[cSize];+            std::memcpy(this->blockLookupTable[i], other.blockLookupTable[i], cSize * sizeof(T));+            cSize <<= 1;+        }+        // if this isn't the case, uhh+        assert((cSize >> 1) == container_size.load());+    }++    /** move constructor */+    PiggyList(PiggyList&& other) = delete;+    /** copy assign ctor **/+    PiggyList& operator=(const PiggyList& other) = delete;++    ~PiggyList() {+        freeList();+    }++    /**+     * Well, returns the number of nodes exist within the list + number of nodes queued to be inserted+     *  The reason for this, is that there may be many nodes queued up+     *  that haven't had time to had containers created and updated+     * @return the number of nodes exist within the list + number of nodes queued to be inserted+     */+    inline size_t size() const {+        return m_size.load();+    };++    inline T* getBlock(size_t blocknum) const {+        return this->blockLookupTable[blocknum];+    }++    size_t append(T element) {+        size_t new_index = m_size.fetch_add(1, std::memory_order_acquire);++        // will this not fit?+        if (container_size < new_index + 1) {+            sl.lock();+            // check and add as many containers as required+            while (container_size < new_index + 1) {+                blockLookupTable[num_containers] = new T[allocsize];+                num_containers += 1;+                container_size += allocsize;+                // double the number elements that will be allocated next time+                allocsize <<= 1;+            }+            sl.unlock();+        }++        this->get(new_index) = element;+        return new_index;+    }++    size_t createNode() {+        size_t new_index = m_size.fetch_add(1, std::memory_order_acquire);++        // will this not fit?+        if (container_size < new_index + 1) {+            sl.lock();+            // check and add as many containers as required+            while (container_size < new_index + 1) {+                blockLookupTable[num_containers] = new T[allocsize];+                num_containers += 1;+                container_size += allocsize;+                // double the number elements that will be allocated next time+                allocsize <<= 1;+            }+            sl.unlock();+        }++        return new_index;+    }++    /**+     * Retrieve a reference to the stored value at index+     * @param index position to search+     * @return the value at index+     */+    inline T& get(size_t index) const {+        // supa fast 2^16 size first block+        size_t nindex = index + BLOCKSIZE;+        size_t blockNum = (63 - __builtin_clzll(nindex));+        size_t blockInd = (nindex) & ((1 << blockNum) - 1);+        return this->getBlock(blockNum - BLOCKBITS)[blockInd];+    }++    /**+     * Clear all elements from the PiggyList+     */+    void clear() {+        freeList();+        m_size = 0;+        num_containers = 0;++        allocsize = BLOCKSIZE;+        container_size = 0;+    }++    class iterator : std::iterator<std::forward_iterator_tag, T> {+        size_t cIndex = 0;+        PiggyList* bl;++    public:+        // default ctor, to silence+        iterator() = default;++        /* begin iterator for iterating over all elements */+        iterator(PiggyList* bl) : bl(bl){};+        /* ender iterator for marking the end of the iteration */+        iterator(PiggyList* bl, size_t beginInd) : cIndex(beginInd), bl(bl){};++        T operator*() {+            return bl->get(cIndex);+        };+        const T operator*() const {+            return bl->get(cIndex);+        };++        iterator& operator++(int) {+            ++cIndex;+            return *this;+        };++        iterator operator++() {+            iterator ret(*this);+            ++cIndex;+            return ret;+        };++        bool operator==(const iterator& x) const {+            return x.cIndex == this->cIndex && x.bl == this->bl;+        };++        bool operator!=(const iterator& x) const {+            return !(x == *this);+        };+    };++    iterator begin() {+        return iterator(this);+    }+    iterator end() {+        return iterator(this, size());+    }+    const size_t BLOCKBITS = 16ul;+    const size_t BLOCKSIZE = (1ul << BLOCKBITS);++    // number of inserted+    std::atomic<size_t> num_containers = 0;+    size_t allocsize = BLOCKSIZE;+    std::atomic<size_t> container_size = 0;+    std::atomic<size_t> m_size = 0;++    // > 2^64 elements can be stored (default initialise to nullptrs)+    static constexpr size_t max_conts = 64;+    std::array<T*, max_conts> blockLookupTable = {};++    // for parallel node insertions+    mutable SpinLock sl;++    /**+     * Free the arrays allocated within the linked list nodes+     */+    void freeList() {+        sl.lock();+        // we don't know which ones are taken up!+        for (size_t i = 0; i < num_containers; ++i) {+            delete[] blockLookupTable[i];+        }+        sl.unlock();+    }+};++}  // namespace souffle
+ cbits/souffle/ProfileDatabase.h view
@@ -0,0 +1,465 @@+#pragma once++#include "json11.h"+#include "utility/MiscUtil.h"+#include <cassert>+#include <chrono>+#include <cstddef>+#include <fstream>+#include <iostream>+#include <iterator>+#include <map>+#include <memory>+#include <mutex>+#include <set>+#include <stdexcept>+#include <string>+#include <utility>+#include <vector>++namespace souffle {+namespace profile {++class DirectoryEntry;+class DurationEntry;+class SizeEntry;+class TextEntry;+class TimeEntry;++/**+ * Visitor Interface+ */+class Visitor {+public:+    virtual ~Visitor() = default;++    // visit entries in a directory+    virtual void visit(DirectoryEntry& e);++    // visit entries+    virtual void visit(DurationEntry&) {}+    virtual void visit(SizeEntry&) {}+    virtual void visit(TextEntry&) {}+    virtual void visit(TimeEntry&) {}+};++/**+ * Entry class+ *+ * abstract class for a key/value entry in a hierarchical database+ */+class Entry {+private:+    // entry key+    std::string key;++public:+    Entry(std::string key) : key(std::move(key)) {}+    virtual ~Entry() = default;++    // get key+    const std::string& getKey() const {+        return key;+    };++    // accept visitor+    virtual void accept(Visitor& v) = 0;++    // print+    virtual void print(std::ostream& os, int tabpos) const = 0;+};++/**+ * DirectoryEntry entry+ */+class DirectoryEntry : public Entry {+private:+    std::map<std::string, std::unique_ptr<Entry>> entries;+    mutable std::mutex lock;++public:+    DirectoryEntry(const std::string& name) : Entry(name) {}++    // get keys+    const std::set<std::string> getKeys() const {+        std::set<std::string> result;+        std::lock_guard<std::mutex> guard(lock);+        for (auto const& cur : entries) {+            result.insert(cur.first);+        }+        return result;+    }++    // write entry+    Entry* writeEntry(std::unique_ptr<Entry> entry) {+        assert(entry != nullptr && "null entry");+        std::lock_guard<std::mutex> guard(lock);+        const std::string& key = entry->getKey();+        // Don't rewrite an existing entry+        if (entries.count(key) == 0) {+            entries[key] = std::move(entry);+        }+        return entries[key].get();+    }++    // read entry+    Entry* readEntry(const std::string& key) const {+        std::lock_guard<std::mutex> guard(lock);+        auto it = entries.find(key);+        if (it != entries.end()) {+            return (*it).second.get();+        } else {+            return nullptr;+        }+    }++    // read directory+    DirectoryEntry* readDirectoryEntry(const std::string& key) const {+        return dynamic_cast<DirectoryEntry*>(readEntry(key));+    }++    // accept visitor+    void accept(Visitor& v) override {+        v.visit(*this);+    }++    // print directory+    void print(std::ostream& os, int tabpos) const override {+        os << std::string(tabpos, ' ') << '"' << getKey() << "\": {" << std::endl;+        bool first{true};+        for (auto const& cur : entries) {+            if (!first) {+                os << ',' << std::endl;+            } else {+                first = false;+            }+            cur.second->print(os, tabpos + 1);+        }+        os << std::endl << std::string(tabpos, ' ') << '}';+    }+};++/**+ * SizeEntry+ */+class SizeEntry : public Entry {+private:+    size_t size;  // size+public:+    SizeEntry(const std::string& key, size_t size) : Entry(key), size(size) {}++    // get size+    size_t getSize() const {+        return size;+    }++    // accept visitor+    void accept(Visitor& v) override {+        v.visit(*this);+    }++    // print entry+    void print(std::ostream& os, int tabpos) const override {+        os << std::string(tabpos, ' ') << "\"" << getKey() << "\": " << size;+    }+};++/**+ * TextEntry+ */+class TextEntry : public Entry {+private:+    // entry text+    std::string text;++public:+    TextEntry(const std::string& key, std::string text) : Entry(key), text(std::move(text)) {}++    // get text+    const std::string& getText() const {+        return text;+    }++    // accept visitor+    void accept(Visitor& v) override {+        v.visit(*this);+    }++    // write size entry+    void print(std::ostream& os, int tabpos) const override {+        os << std::string(tabpos, ' ') << "\"" << getKey() << "\": \"" << text << "\"";+    }+};++/**+ * Duration Entry+ */+class DurationEntry : public Entry {+private:+    // duration start+    microseconds start;++    // duration end+    microseconds end;++public:+    DurationEntry(const std::string& key, microseconds start, microseconds end)+            : Entry(key), start(start), end(end) {}++    // get start+    microseconds getStart() const {+        return start;+    }++    // get end+    microseconds getEnd() const {+        return end;+    }++    // accept visitor+    void accept(Visitor& v) override {+        v.visit(*this);+    }++    // write size entry+    void print(std::ostream& os, int tabpos) const override {+        os << std::string(tabpos, ' ') << '"' << getKey();+        os << R"_(": { "start": )_";+        os << start.count();+        os << ", \"end\": ";+        os << end.count();+        os << '}';+    }+};++/**+ * Time Entry+ */+class TimeEntry : public Entry {+private:+    // time since start+    microseconds time;++public:+    TimeEntry(const std::string& key, microseconds time) : Entry(key), time(time) {}++    // get start+    microseconds getTime() const {+        return time;+    }++    // accept visitor+    void accept(Visitor& v) override {+        v.visit(*this);+    }++    // write size entry+    void print(std::ostream& os, int tabpos) const override {+        os << std::string(tabpos, ' ') << '"' << getKey();+        os << R"_(": { "time": )_";+        os << time.count();+        os << '}';+    }+};++inline void Visitor::visit(DirectoryEntry& e) {+    std::cout << "Dir " << e.getKey() << "\n";+    for (const auto& cur : e.getKeys()) {+        std::cout << "\t :" << cur << "\n";+        e.readEntry(cur)->accept(*this);+    }+}++class Counter : public Visitor {+private:+    size_t ctr{0};+    std::string key;++public:+    Counter(std::string key) : key(std::move(key)) {}+    void visit(SizeEntry& e) override {+        std::cout << "Size entry : " << e.getKey() << " " << e.getSize() << "\n";+        if (e.getKey() == key) {+            ctr += e.getSize();+        }+    }+    size_t getCounter() const {+        return ctr;+    }+};++/**+ * Hierarchical databas+ */+class ProfileDatabase {+private:+    std::unique_ptr<DirectoryEntry> root;++protected:+    /**+     * Find path: if directories along the path do not exist, create them.+     */+    DirectoryEntry* lookupPath(const std::vector<std::string>& path) {+        DirectoryEntry* dir = root.get();+        for (const std::string& key : path) {+            assert(!key.empty() && "Key is empty!");+            DirectoryEntry* newDir = dir->readDirectoryEntry(key);+            if (newDir == nullptr) {+                newDir =+                        dynamic_cast<DirectoryEntry*>(dir->writeEntry(std::make_unique<DirectoryEntry>(key)));+            }+            assert(newDir != nullptr && "Attempting to overwrite an existing entry");+            dir = newDir;+        }+        return dir;+    }++    void parseJson(const json11::Json& json, std::unique_ptr<DirectoryEntry>& node) {+        for (auto& cur : json.object_items()) {+            if (cur.second.is_object()) {+                std::string err;+                // Duration entries are also maps+                if (cur.second.has_shape(+                            {{"start", json11::Json::NUMBER}, {"end", json11::Json::NUMBER}}, err)) {+                    auto start = std::chrono::microseconds(cur.second["start"].long_value());+                    auto end = std::chrono::microseconds(cur.second["end"].long_value());+                    node->writeEntry(std::make_unique<DurationEntry>(cur.first, start, end));+                } else if (cur.second.has_shape({{"time", json11::Json::NUMBER}}, err)) {+                    auto time = std::chrono::microseconds(cur.second["time"].long_value());+                    node->writeEntry(std::make_unique<TimeEntry>(cur.first, time));+                } else {+                    auto dir = std::make_unique<DirectoryEntry>(cur.first);+                    parseJson(cur.second, dir);+                    node->writeEntry(std::move(dir));+                }+            } else if (cur.second.is_string()) {+                node->writeEntry(std::make_unique<TextEntry>(cur.first, cur.second.string_value()));+            } else if (cur.second.is_number()) {+                node->writeEntry(std::make_unique<SizeEntry>(cur.first, cur.second.long_value()));+            } else {+                std::string err;+                cur.second.dump(err);+                std::cerr << "Unknown types in profile log: " << cur.first << ": " << err << std::endl;+            }+        }+    }++public:+    ProfileDatabase() : root(std::make_unique<DirectoryEntry>("root")) {}++    ProfileDatabase(const std::string& filename) : root(std::make_unique<DirectoryEntry>("root")) {+        std::ifstream file(filename);+        if (!file.is_open()) {+            throw std::runtime_error("Log file could not be opened.");+        }+        std::string jsonString((std::istreambuf_iterator<char>(file)), (std::istreambuf_iterator<char>()));+        std::string error;+        json11::Json json = json11::Json::parse(jsonString, error);+        if (!error.empty()) {+            throw std::runtime_error("Parse error: " + error);+        }+        parseJson(json["root"], root);+    }++    // add size entry+    void addSizeEntry(std::vector<std::string> qualifier, size_t size) {+        assert(qualifier.size() > 0 && "no qualifier");+        std::vector<std::string> path(qualifier.begin(), qualifier.end() - 1);+        DirectoryEntry* dir = lookupPath(path);++        const std::string& key = qualifier.back();+        std::unique_ptr<SizeEntry> entry = std::make_unique<SizeEntry>(key, size);+        dir->writeEntry(std::move(entry));+    }++    // add text entry+    void addTextEntry(std::vector<std::string> qualifier, const std::string& text) {+        assert(qualifier.size() > 0 && "no qualifier");+        std::vector<std::string> path(qualifier.begin(), qualifier.end() - 1);+        DirectoryEntry* dir = lookupPath(path);++        const std::string& key = qualifier.back();+        std::unique_ptr<TextEntry> entry = std::make_unique<TextEntry>(key, text);+        dir->writeEntry(std::move(entry));+    }++    // add duration entry+    void addDurationEntry(std::vector<std::string> qualifier, microseconds start, microseconds end) {+        assert(qualifier.size() > 0 && "no qualifier");+        std::vector<std::string> path(qualifier.begin(), qualifier.end() - 1);+        DirectoryEntry* dir = lookupPath(path);++        const std::string& key = qualifier.back();+        std::unique_ptr<DurationEntry> entry = std::make_unique<DurationEntry>(key, start, end);+        dir->writeEntry(std::move(entry));+    }++    // add time entry+    void addTimeEntry(std::vector<std::string> qualifier, microseconds time) {+        assert(qualifier.size() > 0 && "no qualifier");+        std::vector<std::string> path(qualifier.begin(), qualifier.end() - 1);+        DirectoryEntry* dir = lookupPath(path);++        const std::string& key = qualifier.back();+        std::unique_ptr<TimeEntry> entry = std::make_unique<TimeEntry>(key, time);+        dir->writeEntry(std::move(entry));+    }++    // compute sum+    size_t computeSum(const std::vector<std::string>& qualifier) {+        assert(qualifier.size() > 0 && "no qualifier");+        std::vector<std::string> path(qualifier.begin(), qualifier.end() - 1);+        DirectoryEntry* dir = lookupPath(path);++        const std::string& key = qualifier.back();+        std::cout << "Key: " << key << std::endl;+        Counter ctr(key);+        dir->accept(ctr);+        return ctr.getCounter();+    }++    /**+     * Return the entry at the given path.+     */+    Entry* lookupEntry(const std::vector<std::string>& path) const {+        DirectoryEntry* dir = root.get();+        auto last = --path.end();+        for (auto it = path.begin(); it != last; ++it) {+            dir = dir->readDirectoryEntry(*it);+            if (dir == nullptr) {+                return nullptr;+            }+        }+        return dir->readEntry(*last);+    }++    /**+     * Return a map of string keys to string values.+     */+    std::map<std::string, std::string> getStringMap(const std::vector<std::string>& path) const {+        std::map<std::string, std::string> kvps;+        auto* parent = dynamic_cast<DirectoryEntry*>(lookupEntry(path));+        if (parent == nullptr) {+            return kvps;+        }++        for (const auto& key : parent->getKeys()) {+            auto* text = dynamic_cast<TextEntry*>(parent->readEntry(key));+            if (text != nullptr) {+                kvps[key] = text->getText();+            }+        }++        return kvps;+    }++    // print database+    void print(std::ostream& os) const {+        os << '{' << std::endl;+        root->print(os, 1);+        os << std::endl << '}' << std::endl;+    };+};++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/ProfileEvent.h view
@@ -0,0 +1,226 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2018, The Souffle Developers. All rights reserved.+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ProfileEvent.h+ *+ * Declares classes for profile events+ *+ ***********************************************************************/++#pragma once++#include "EventProcessor.h"+#include "ProfileDatabase.h"+#include "utility/MiscUtil.h"+#include <atomic>+#include <chrono>+#include <condition_variable>+#include <cstdint>+#include <ctime>+#include <iostream>+#include <mutex>+#include <sstream>+#include <string>+#include <thread>+#include <sys/resource.h>+#include <sys/time.h>++namespace souffle {++/**+ * Profile Event Singleton+ */+class ProfileEventSingleton {+    /** profile database */+    profile::ProfileDatabase database;+    std::string filename{""};++    ProfileEventSingleton() = default;++public:+    ~ProfileEventSingleton() {+        stopTimer();+        ProfileEventSingleton::instance().dump();+    }++    /** get instance */+    static ProfileEventSingleton& instance() {+        static ProfileEventSingleton singleton;+        return singleton;+    }++    /** create config record */+    void makeConfigRecord(const std::string& key, const std::string& value) {+        profile::EventProcessorSingleton::instance().process(database, "@config", key.c_str(), value.c_str());+    }++    /** create time event */+    void makeTimeEvent(const std::string& txt) {+        profile::EventProcessorSingleton::instance().process(+                database, txt.c_str(), std::chrono::duration_cast<microseconds>(now().time_since_epoch()));+    }++    /** create an event for recording start and end times */+    void makeTimingEvent(const std::string& txt, time_point start, time_point end, size_t startMaxRSS,+            size_t endMaxRSS, size_t size, size_t iteration) {+        microseconds start_ms = std::chrono::duration_cast<microseconds>(start.time_since_epoch());+        microseconds end_ms = std::chrono::duration_cast<microseconds>(end.time_since_epoch());+        profile::EventProcessorSingleton::instance().process(+                database, txt.c_str(), start_ms, end_ms, startMaxRSS, endMaxRSS, size, iteration);+    }++    /** create quantity event */+    void makeQuantityEvent(const std::string& txt, size_t number, int iteration) {+        profile::EventProcessorSingleton::instance().process(database, txt.c_str(), number, iteration);+    }++    /** create utilisation event */+    void makeUtilisationEvent(const std::string& txt) {+        /* current time */+        microseconds time = std::chrono::duration_cast<microseconds>(now().time_since_epoch());+        /* system CPU time used */+        struct rusage ru {};+        getrusage(RUSAGE_SELF, &ru);+        /* system CPU time used */+        uint64_t systemTime = ru.ru_stime.tv_sec * 1000000 + ru.ru_stime.tv_usec;+        /* user CPU time used */+        uint64_t userTime = ru.ru_utime.tv_sec * 1000000 + ru.ru_utime.tv_usec;+        /* Maximum resident set size (kb) */+        size_t maxRSS = ru.ru_maxrss;++        profile::EventProcessorSingleton::instance().process(+                database, txt.c_str(), time, systemTime, userTime, maxRSS);+    }++    void setOutputFile(std::string filename) {+        this->filename = filename;+    }+    /** Dump all events */+    void dump() {+        if (!filename.empty()) {+            std::ofstream os(filename);+            if (!os.is_open()) {+                std::cerr << "Cannot open profile log file <" + filename + ">";+            } else {+                database.print(os);+            }+        }+    }++    /** Start timer */+    void startTimer() {+        timer.start();+    }++    /** Stop timer */+    void stopTimer() {+        timer.stop();+    }++    void resetTimerInterval(uint32_t interval = 1) {+        timer.resetTimerInterval(interval);+    }+    const profile::ProfileDatabase& getDB() const {+        return database;+    }++    void setDBFromFile(const std::string& filename) {+        database = profile::ProfileDatabase(filename);+    }++private:+    /**  Profile Timer */+    class ProfileTimer {+    private:+        /** time interval between per utilisation read */+        uint32_t t;++        /** timer is running */+        std::atomic<bool> running{false};++        /** thread timer runs on */+        std::thread th;++        std::condition_variable conditionVariable;+        std::mutex timerMutex;++        /** number of utilisation events */+        std::atomic<uint32_t> runCount{0};++        /** run method for thread th */+        void run() {+            ProfileEventSingleton::instance().makeUtilisationEvent("@utilisation");+            ++runCount;+            if (runCount % 128 == 0) {+                increaseInterval();+            }+        }++        uint32_t getInterval() {+            return t;+        }++        /** Increase value of time interval by factor of 2 */+        void increaseInterval() {+            // Don't increase time interval past 60 seconds+            if (t < 60000) {+                t = t * 2;+            }+        }++    public:+        /*+         *  @param interval the size of the timing interval in milliseconds+         */+        ProfileTimer(uint32_t interval = 10) : t(interval) {}++        /** start timer on the thread th */+        void start() {+            if (running) {+                return;+            }+            running = true;++            th = std::thread([this]() {+                while (running) {+                    run();+                    std::unique_lock<std::mutex> lock(timerMutex);+                    conditionVariable.wait_for(lock, std::chrono::milliseconds(getInterval()));+                }+            });+        }++        /** stop timer on the thread th */+        void stop() {+            running = false;+            conditionVariable.notify_all();+            if (th.joinable()) {+                th.join();+            }+        }++        /** Reset timer interval.+         *+         *  The timer interval increases as the program executes. Resetting the interval is useful to+         *  ensure that detailed usage information is gathered even in long running programs, if desired.+         *+         *  @param interval the size of the timing interval in milliseconds+         */+        void resetTimerInterval(uint32_t interval = 10) {+            t = interval;+            runCount = 0;+            conditionVariable.notify_all();+        }+    };++    ProfileTimer timer;+};++}  // namespace souffle
+ cbits/souffle/RamTypes.h view
@@ -0,0 +1,110 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file RamTypes.h+ *+ * Defines tuple element type and data type for keys on table columns+ *+ ***********************************************************************/++#pragma once++#include <cstdint>+#include <cstring>+#include <iostream>+#include <limits>+#include <type_traits>++namespace souffle {++enum class TypeAttribute {+    Symbol,+    Signed,    // Signed number+    Unsigned,  // Unsigned number+    Float,     // Floating point number.+    Record,+};++// Printing of the TypeAttribute Enum.+// To be utilised in synthesizer.+std::ostream& operator<<(std::ostream& os, TypeAttribute T);++/**+ * Check if type is numeric.+ */+bool isNumericType(TypeAttribute ramType);++/**+ * Types of elements in a tuple.+ *+ * Default domain has size of 32 bits; may be overridden by user+ * defining RAM_DOMAIN_SIZE.+ */++#ifndef RAM_DOMAIN_SIZE+#define RAM_DOMAIN_SIZE 32+#endif++#if RAM_DOMAIN_SIZE == 64+using RamDomain = int64_t;+using RamSigned = RamDomain;+using RamUnsigned = uint64_t;+// There is not standard fixed size double/float.+using RamFloat = double;+#else+using RamDomain = int32_t;+using RamSigned = RamDomain;+using RamUnsigned = uint32_t;+// There is no standard - fixed size double/float.+using RamFloat = float;+#endif++// Compile time sanity checks+static_assert(std::is_integral<RamSigned>::value && std::is_signed<RamSigned>::value,+        "RamSigned must be represented by a signed type.");+static_assert(std::is_integral<RamUnsigned>::value && !std::is_signed<RamUnsigned>::value,+        "RamUnsigned must be represented by an unsigned type.");+static_assert(std::is_floating_point<RamFloat>::value && sizeof(RamFloat) * 8 == RAM_DOMAIN_SIZE,+        "RamFloat must be represented by a floating point and have the same size as other types.");++template <typename T>+constexpr bool isRamType = (std::is_same<T, RamDomain>::value || std::is_same<T, RamSigned>::value ||+                            std::is_same<T, RamUnsigned>::value || std::is_same<T, RamFloat>::value);++/**+In C++20 there will be a new way to cast between types by reinterpreting bits (std::bit_cast),+but as of January 2020 it is not yet supported.+**/++/** Cast a type by reinterpreting its bits. Domain is restricted to Ram Types only.+ * Template takes two types (second type is never necessary because it can be deduced from the argument)+ * The following always holds+ * For type T and a : T+ * ramBitCast<T>(ramBitCast<RamDomain>(a)) == a+ **/+template <typename To = RamDomain, typename From>+To ramBitCast(From source) {+    static_assert(isRamType<From> && isRamType<To>, "Bit casting should only be used on Ram Types.");+    static_assert(sizeof(To) == sizeof(From), "Can't bit cast types with different size.");+    To destination;+    memcpy(&destination, &source, sizeof(destination));+    return destination;+}++/** lower and upper boundaries for the ram types **/+constexpr RamSigned MIN_RAM_SIGNED = std::numeric_limits<RamSigned>::min();+constexpr RamSigned MAX_RAM_SIGNED = std::numeric_limits<RamSigned>::max();++constexpr RamUnsigned MIN_RAM_UNSIGNED = std::numeric_limits<RamUnsigned>::min();+constexpr RamUnsigned MAX_RAM_UNSIGNED = std::numeric_limits<RamUnsigned>::max();++constexpr RamFloat MIN_RAM_FLOAT = std::numeric_limits<RamFloat>::min();+constexpr RamFloat MAX_RAM_FLOAT = std::numeric_limits<RamFloat>::max();+}  // end of namespace souffle
+ cbits/souffle/ReadStream.h view
@@ -0,0 +1,179 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ReadStream.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "RecordTable.h"+#include "SerialisationStream.h"+#include "SymbolTable.h"+#include "json11.h"+#include "utility/MiscUtil.h"+#include "utility/StringUtil.h"+#include <cctype>+#include <cstddef>+#include <map>+#include <memory>+#include <ostream>+#include <stdexcept>+#include <string>+#include <vector>++namespace souffle {++class ReadStream : public SerialisationStream<false> {+protected:+    ReadStream(+            const std::map<std::string, std::string>& rwOperation, SymbolTable& symTab, RecordTable& recTab)+            : SerialisationStream(symTab, recTab, rwOperation) {}++public:+    template <typename T>+    void readAll(T& relation) {+        auto lease = symbolTable.acquireLock();+        (void)lease;+        while (const auto next = readNextTuple()) {+            const RamDomain* ramDomain = next.get();+            relation.insert(ramDomain);+        }+    }++protected:+    /**+     * Read a record from a string.+     *+     * @param source - string containing a record+     * @param recordTypeName - record type.+     * @parem pos - start parsing from this position.+     * @param consumed - if not nullptr: number of characters read.+     *+     */+    RamDomain readRecord(const std::string& source, const std::string& recordTypeName, size_t pos = 0,+            size_t* charactersRead = nullptr) {+        const size_t initial_position = pos;++        // Check if record type information are present+        auto&& recordInfo = types["records"][recordTypeName];+        if (recordInfo.is_null()) {+            throw std::invalid_argument("Missing record type information: " + recordTypeName);+        }++        // Handle nil case+        consumeWhiteSpace(source, pos);+        if (source.substr(pos, 3) == "nil") {+            if (charactersRead != nullptr) {+                *charactersRead = 3;+            }+            return 0;+        }++        auto&& recordTypes = recordInfo["types"];+        const size_t recordArity = recordInfo["arity"].long_value();++        std::vector<RamDomain> recordValues(recordArity);++        consumeChar(source, '[', pos);++        for (size_t i = 0; i < recordArity; ++i) {+            const std::string& recordType = recordTypes[i].string_value();+            size_t consumed = 0;++            if (i > 0) {+                consumeChar(source, ',', pos);+            }+            consumeWhiteSpace(source, pos);+            switch (recordType[0]) {+                case 's': {+                    recordValues[i] = readStringInRecord(source, pos, &consumed);+                    break;+                }+                case 'i': {+                    recordValues[i] = RamSignedFromString(source.substr(pos), &consumed);+                    break;+                }+                case 'u': {+                    recordValues[i] = ramBitCast(RamUnsignedFromString(source.substr(pos), &consumed));+                    break;+                }+                case 'f': {+                    recordValues[i] = ramBitCast(RamFloatFromString(source.substr(pos), &consumed));+                    break;+                }+                case 'r': {+                    recordValues[i] = readRecord(source, recordType, pos, &consumed);+                    break;+                }+                default: fatal("Invalid type attribute");+            }+            pos += consumed;+        }+        consumeChar(source, ']', pos);++        if (charactersRead != nullptr) {+            *charactersRead = pos - initial_position;+        }++        return recordTable.pack(recordValues.data(), recordValues.size());+    }++    RamDomain readStringInRecord(const std::string& source, const size_t pos, size_t* charactersRead) {+        size_t endOfSymbol = source.find_first_of(",]", pos);++        if (endOfSymbol == std::string::npos) {+            throw std::invalid_argument("Unexpected end of input in record");+        }++        *charactersRead = endOfSymbol - pos;+        std::string str = source.substr(pos, *charactersRead);++        return symbolTable.unsafeLookup(str);+    }++    /**+     * Read past given character, consuming any preceding whitespace.+     */+    void consumeChar(const std::string& str, char c, size_t& pos) {+        consumeWhiteSpace(str, pos);+        if (pos >= str.length()) {+            throw std::invalid_argument("Unexpected end of input in record");+        }+        if (str[pos] != c) {+            std::stringstream error;+            error << "Expected: \'" << c << "\', got: " << str[pos];+            throw std::invalid_argument(error.str());+        }+        ++pos;+    }++    /**+     * Advance position in the string until first non-whitespace character.+     */+    void consumeWhiteSpace(const std::string& str, size_t& pos) {+        while (pos < str.length() && std::isspace(static_cast<unsigned char>(str[pos]))) {+            ++pos;+        }+    }++    virtual std::unique_ptr<RamDomain[]> readNextTuple() = 0;+};++class ReadStreamFactory {+public:+    virtual std::unique_ptr<ReadStream> getReader(+            const std::map<std::string, std::string>&, SymbolTable&, RecordTable&) = 0;+    virtual const std::string& getName() const = 0;+    virtual ~ReadStreamFactory() = default;+};++} /* namespace souffle */
+ cbits/souffle/ReadStreamCSV.h view
@@ -0,0 +1,317 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ReadStreamCSV.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "ReadStream.h"+#include "SymbolTable.h"+#include "utility/ContainerUtil.h"+#include "utility/FileUtil.h"+#include "utility/StringUtil.h"++#ifdef USE_LIBZ+#include "gzfstream.h"+#else+#include <fstream>+#endif++#include <algorithm>+#include <cassert>+#include <cstddef>+#include <cstdint>+#include <iostream>+#include <map>+#include <memory>+#include <sstream>+#include <stdexcept>+#include <string>+#include <vector>++namespace souffle {+class RecordTable;++class ReadStreamCSV : public ReadStream {+public:+    ReadStreamCSV(std::istream& file, const std::map<std::string, std::string>& rwOperation,+            SymbolTable& symbolTable, RecordTable& recordTable)+            : ReadStream(rwOperation, symbolTable, recordTable),+              delimiter(getOr(rwOperation, "delimiter", "\t")), file(file), lineNumber(0),+              inputMap(getInputColumnMap(rwOperation, arity)) {+        while (inputMap.size() < arity) {+            int size = static_cast<int>(inputMap.size());+            inputMap[size] = size;+        }+    }++protected:+    /**+     * Read and return the next tuple.+     *+     * Returns nullptr if no tuple was readable.+     * @return+     */+    std::unique_ptr<RamDomain[]> readNextTuple() override {+        if (file.eof()) {+            return nullptr;+        }+        std::string line;+        std::unique_ptr<RamDomain[]> tuple = std::make_unique<RamDomain[]>(typeAttributes.size());++        if (!getline(file, line)) {+            return nullptr;+        }+        // Handle Windows line endings on non-Windows systems+        if (!line.empty() && line.back() == '\r') {+            line = line.substr(0, line.length() - 1);+        }+        ++lineNumber;++        size_t start = 0;+        size_t end = 0;+        size_t columnsFilled = 0;+        for (uint32_t column = 0; columnsFilled < arity; column++) {+            size_t charactersRead = 0;+            std::string element = nextElement(line, start, end);+            if (inputMap.count(column) == 0) {+                continue;+            }+            ++columnsFilled;++            try {+                auto&& ty = typeAttributes.at(inputMap[column]);+                switch (ty[0]) {+                    case 's': {+                        tuple[inputMap[column]] = symbolTable.unsafeLookup(element);+                        charactersRead = element.size();+                        break;+                    }+                    case 'r': {+                        tuple[inputMap[column]] = readRecord(element, ty, 0, &charactersRead);+                        break;+                    }+                    case 'i': {+                        tuple[inputMap[column]] = RamSignedFromString(element, &charactersRead);+                        break;+                    }+                    case 'u': {+                        tuple[inputMap[column]] = ramBitCast(readRamUnsigned(element, charactersRead));+                        break;+                    }+                    case 'f': {+                        tuple[inputMap[column]] = ramBitCast(RamFloatFromString(element, &charactersRead));+                        break;+                    }+                    default: fatal("invalid type attribute: `%c`", ty[0]);+                }+                // Check if everything was read.+                if (charactersRead != element.size()) {+                    throw std::invalid_argument(+                            "Expected: " + delimiter + " or \\n. Got: " + element[charactersRead]);+                }+            } catch (...) {+                std::stringstream errorMessage;+                errorMessage << "Error converting <" + element + "> in column " << column + 1 << " in line "+                             << lineNumber << "; ";+                throw std::invalid_argument(errorMessage.str());+            }+        }++        return tuple;+    }++    /**+     * Read an unsigned element. Possible bases are 2, 10, 16+     * Base is indicated by the first two chars.+     */+    RamUnsigned readRamUnsigned(const std::string& element, size_t& charactersRead) {+        // Sanity check+        assert(element.size() > 0);++        RamSigned value = 0;++        // Check prefix and parse the input.+        if (isPrefix("0b", element)) {+            value = RamUnsignedFromString(element, &charactersRead, 2);+        } else if (isPrefix("0x", element)) {+            value = RamUnsignedFromString(element, &charactersRead, 16);+        } else {+            value = RamUnsignedFromString(element, &charactersRead);+        }+        return value;+    }++    std::string nextElement(const std::string& line, size_t& start, size_t& end) {+        std::string element;++        // Handle record/tuple delimiter coincidence.+        if (delimiter.find(',') != std::string::npos) {+            int record_parens = 0;+            size_t next_delimiter = line.find(delimiter, start);++            // Find first delimiter after the record.+            while (end < std::min(next_delimiter, line.length()) || record_parens != 0) {+                // Track the number of parenthesis.+                if (line[end] == '[') {+                    ++record_parens;+                } else if (line[end] == ']') {+                    --record_parens;+                }++                // Check for unbalanced parenthesis.+                if (record_parens < 0) {+                    break;+                };++                ++end;++                // Find a next delimiter if the old one is invalid.+                // But only if inside the unbalance parenthesis.+                if (end == next_delimiter && record_parens != 0) {+                    next_delimiter = line.find(delimiter, end);+                }+            }++            // Handle the end-of-the-line case where parenthesis are unbalanced.+            if (record_parens != 0) {+                std::stringstream errorMessage;+                errorMessage << "Unbalanced record parenthesis " << lineNumber << "; ";+                throw std::invalid_argument(errorMessage.str());+            }+        } else {+            end = std::min(line.find(delimiter, start), line.length());+        }++        // Check for missing value.+        if (start > end) {+            std::stringstream errorMessage;+            errorMessage << "Values missing in line " << lineNumber << "; ";+            throw std::invalid_argument(errorMessage.str());+        }++        element = line.substr(start, end - start);+        start = end + delimiter.size();++        return element;+    }++    std::map<int, int> getInputColumnMap(+            const std::map<std::string, std::string>& rwOperation, const unsigned arity_) const {+        std::string columnString = getOr(rwOperation, "columns", "");+        std::map<int, int> inputColumnMap;++        if (!columnString.empty()) {+            std::istringstream iss(columnString);+            std::string mapping;+            int index = 0;+            while (std::getline(iss, mapping, ':')) {+                inputColumnMap[stoi(mapping)] = index++;+            }+            if (inputColumnMap.size() < arity_) {+                throw std::invalid_argument("Invalid column set was given: <" + columnString + ">");+            }+        } else {+            while (inputColumnMap.size() < arity_) {+                int size = static_cast<int>(inputColumnMap.size());+                inputColumnMap[size] = size;+            }+        }+        return inputColumnMap;+    }++    const std::string delimiter;+    std::istream& file;+    size_t lineNumber;+    std::map<int, int> inputMap;+};++class ReadFileCSV : public ReadStreamCSV {+public:+    ReadFileCSV(const std::map<std::string, std::string>& rwOperation, SymbolTable& symbolTable,+            RecordTable& recordTable)+            : ReadStreamCSV(fileHandle, rwOperation, symbolTable, recordTable),+              baseName(souffle::baseName(getFileName(rwOperation))),+              fileHandle(getFileName(rwOperation), std::ios::in | std::ios::binary) {+        if (!fileHandle.is_open()) {+            throw std::invalid_argument("Cannot open fact file " + baseName + "\n");+        }+        // Strip headers if we're using them+        if (getOr(rwOperation, "headers", "false") == "true") {+            std::string line;+            getline(file, line);+        }+    }++    /**+     * Read and return the next tuple.+     *+     * Returns nullptr if no tuple was readable.+     * @return+     */+    std::unique_ptr<RamDomain[]> readNextTuple() override {+        try {+            return ReadStreamCSV::readNextTuple();+        } catch (std::exception& e) {+            std::stringstream errorMessage;+            errorMessage << e.what();+            errorMessage << "cannot parse fact file " << baseName << "!\n";+            throw std::invalid_argument(errorMessage.str());+        }+    }++    ~ReadFileCSV() override = default;++protected:+    static std::string getFileName(const std::map<std::string, std::string>& rwOperation) {+        return getOr(rwOperation, "filename", rwOperation.at("name") + ".facts");+    }++    std::string baseName;+#ifdef USE_LIBZ+    gzfstream::igzfstream fileHandle;+#else+    std::ifstream fileHandle;+#endif+};++class ReadCinCSVFactory : public ReadStreamFactory {+public:+    std::unique_ptr<ReadStream> getReader(const std::map<std::string, std::string>& rwOperation,+            SymbolTable& symbolTable, RecordTable& recordTable) override {+        return std::make_unique<ReadStreamCSV>(std::cin, rwOperation, symbolTable, recordTable);+    }++    const std::string& getName() const override {+        static const std::string name = "stdin";+        return name;+    }+    ~ReadCinCSVFactory() override = default;+};++class ReadFileCSVFactory : public ReadStreamFactory {+public:+    std::unique_ptr<ReadStream> getReader(const std::map<std::string, std::string>& rwOperation,+            SymbolTable& symbolTable, RecordTable& recordTable) override {+        return std::make_unique<ReadFileCSV>(rwOperation, symbolTable, recordTable);+    }++    const std::string& getName() const override {+        static const std::string name = "file";+        return name;+    }++    ~ReadFileCSVFactory() override = default;+};++} /* namespace souffle */
+ cbits/souffle/ReadStreamSQLite.h view
@@ -0,0 +1,175 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ReadStreamSQLite.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "ReadStream.h"+#include "SymbolTable.h"+#include "utility/MiscUtil.h"+#include "utility/StringUtil.h"+#include <cassert>+#include <cstdint>+#include <fstream>+#include <map>+#include <memory>+#include <stdexcept>+#include <string>+#include <vector>+#include <sqlite3.h>++namespace souffle {+class RecordTable;++class ReadStreamSQLite : public ReadStream {+public:+    ReadStreamSQLite(const std::map<std::string, std::string>& rwOperation, SymbolTable& symbolTable,+            RecordTable& recordTable)+            : ReadStream(rwOperation, symbolTable, recordTable), dbFilename(rwOperation.at("filename")),+              relationName(rwOperation.at("name")) {+        openDB();+        checkTableExists();+        prepareSelectStatement();+    }++    ~ReadStreamSQLite() override {+        sqlite3_finalize(selectStatement);+        sqlite3_close(db);+    }++protected:+    /**+     * Read and return the next tuple.+     *+     * Returns nullptr if no tuple was readable.+     * @return+     */+    std::unique_ptr<RamDomain[]> readNextTuple() override {+        if (sqlite3_step(selectStatement) != SQLITE_ROW) {+            return nullptr;+        }++        std::unique_ptr<RamDomain[]> tuple = std::make_unique<RamDomain[]>(arity + auxiliaryArity);++        uint32_t column;+        for (column = 0; column < arity; column++) {+            std::string element(reinterpret_cast<const char*>(sqlite3_column_text(selectStatement, column)));++            if (element.empty()) {+                element = "n/a";+            }++            try {+                auto&& ty = typeAttributes.at(column);+                switch (ty[0]) {+                    case 's': tuple[column] = symbolTable.unsafeLookup(element); break;+                    case 'i':+                    case 'u':+                    case 'f':+                    case 'r': tuple[column] = RamSignedFromString(element); break;+                    default: fatal("invalid type attribute: `%c`", ty[0]);+                }+            } catch (...) {+                std::stringstream errorMessage;+                errorMessage << "Error converting number in column " << (column) + 1;+                throw std::invalid_argument(errorMessage.str());+            }+        }++        return tuple;+    }++    void executeSQL(const std::string& sql) {+        assert(db && "Database connection is closed");++        char* errorMessage = nullptr;+        /* Execute SQL statement */+        int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errorMessage);+        if (rc != SQLITE_OK) {+            std::stringstream error;+            error << "SQLite error in sqlite3_exec: " << sqlite3_errmsg(db) << "\n";+            error << "SQL error: " << errorMessage << "\n";+            error << "SQL: " << sql << "\n";+            sqlite3_free(errorMessage);+            throw std::invalid_argument(error.str());+        }+    }++    void throwError(const std::string& message) {+        std::stringstream error;+        error << message << sqlite3_errmsg(db) << "\n";+        throw std::invalid_argument(error.str());+    }++    void prepareSelectStatement() {+        std::stringstream selectSQL;+        selectSQL << "SELECT * FROM '" << relationName << "'";+        const char* tail = nullptr;+        if (sqlite3_prepare_v2(db, selectSQL.str().c_str(), -1, &selectStatement, &tail) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_prepare_v2: ");+        }+    }++    void openDB() {+        if (sqlite3_open(dbFilename.c_str(), &db) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_open: ");+        }+        sqlite3_extended_result_codes(db, 1);+        executeSQL("PRAGMA synchronous = OFF");+        executeSQL("PRAGMA journal_mode = MEMORY");+    }++    void checkTableExists() {+        sqlite3_stmt* tableStatement;+        std::stringstream selectSQL;+        selectSQL << "SELECT count(*) FROM sqlite_master WHERE type IN ('table', 'view') AND ";+        selectSQL << " name = '" << relationName << "';";+        const char* tail = nullptr;++        if (sqlite3_prepare_v2(db, selectSQL.str().c_str(), -1, &tableStatement, &tail) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_prepare_v2: ");+        }++        if (sqlite3_step(tableStatement) == SQLITE_ROW) {+            int count = sqlite3_column_int(tableStatement, 0);+            if (count > 0) {+                sqlite3_finalize(tableStatement);+                return;+            }+        }+        sqlite3_finalize(tableStatement);+        throw std::invalid_argument(+                "Required table or view does not exist in " + dbFilename + " for relation " + relationName);+    }+    const std::string& dbFilename;+    const std::string& relationName;+    sqlite3_stmt* selectStatement = nullptr;+    sqlite3* db = nullptr;+};++class ReadSQLiteFactory : public ReadStreamFactory {+public:+    std::unique_ptr<ReadStream> getReader(const std::map<std::string, std::string>& rwOperation,+            SymbolTable& symbolTable, RecordTable& recordTable) override {+        return std::make_unique<ReadStreamSQLite>(rwOperation, symbolTable, recordTable);+    }++    const std::string& getName() const override {+        static const std::string name = "sqlite";+        return name;+    }+    ~ReadSQLiteFactory() override = default;+};++} /* namespace souffle */
+ cbits/souffle/RecordTable.h view
@@ -0,0 +1,147 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2020, The Souffle Developers. All rights reserved.+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file RecordTable.h+ *+ * Data container implementing a map between records and their references.+ * Records are separated by arity, i.e., stored in different RecordMaps.+ *+ ***********************************************************************/++#pragma once++#include "CompiledTuple.h"+#include "RamTypes.h"+#include <cassert>+#include <cstddef>+#include <limits>+#include <memory>+#include <unordered_map>+#include <utility>+#include <vector>++namespace souffle {++/** @brief Bidirectional mappping between records and record references */+class RecordMap {+    /** arity of record */+    const size_t arity;++    /** hash function for unordered record map */+    struct RecordHash {+        std::size_t operator()(std::vector<RamDomain> record) const {+            std::size_t seed = 0;+            std::hash<RamDomain> domainHash;+            for (RamDomain value : record) {+                seed ^= domainHash(value) + 0x9e3779b9 + (seed << 6) + (seed >> 2);+            }+            return seed;+        }+    };++    /** map from records to references */+    // TODO (b-scholz): replace vector<RamDomain> with something more memory-frugal+    std::unordered_map<std::vector<RamDomain>, RamDomain, RecordHash> recordToIndex;++    /** array of records; index represents record reference */+    // TODO (b-scholz): replace vector<RamDomain> with something more memory-frugal+    std::vector<std::vector<RamDomain>> indexToRecord;++public:+    explicit RecordMap(size_t arity) : arity(arity), indexToRecord(1) {}  // note: index 0 element left free++    /** @brief converts record to a record reference */+    // TODO (b-scholz): replace vector<RamDomain> with something more memory-frugal+    RamDomain pack(const std::vector<RamDomain>& vector) {+        RamDomain index;+#pragma omp critical(record_pack)+        {+            auto pos = recordToIndex.find(vector);+            if (pos != recordToIndex.end()) {+                index = pos->second;+            } else {+#pragma omp critical(record_unpack)+                {+                    indexToRecord.push_back(vector);+                    index = indexToRecord.size() - 1;+                    recordToIndex[vector] = index;++                    // assert that new index is smaller than the range+                    assert(index != std::numeric_limits<RamDomain>::max());+                }+            }+        }+        return index;+    }++    /** @brief convert record pointer to a record reference */+    RamDomain pack(const RamDomain* tuple) {+        // TODO (b-scholz): data is unnecessarily copied+        // for a successful lookup. To avoid this, we should+        // compute a hash of the pointer-array and traverse through+        // the bucket list of the unordered map finding the record.+        // Note that in case of non-existence, the record still needs to be+        // copied for the newly created entry but this will be the less+        // frequent case.+        std::vector<RamDomain> tmp(arity);+        for (size_t i = 0; i < arity; i++) {+            tmp[i] = tuple[i];+        }+        return pack(tmp);+    }++    /** @brief convert record reference to a record pointer */+    const RamDomain* unpack(RamDomain index) const {+        const RamDomain* res;+#pragma omp critical(record_unpack)+        res = indexToRecord[index].data();+        return res;+    }+};++class RecordTable {+public:+    RecordTable() = default;+    virtual ~RecordTable() = default;++    /** @brief convert record to record reference */+    RamDomain pack(RamDomain* tuple, size_t arity) {+        return lookupArity(arity).pack(tuple);+    }+    /** @brief convert record reference to a record */+    const RamDomain* unpack(RamDomain ref, size_t arity) const {+        auto iter = maps.find(arity);+        assert(iter != maps.end() && "Attempting to unpack non-existing record");+        return (iter->second).unpack(ref);+    }++private:+    /** @brief lookup RecordMap for a given arity; if it does not exist, create new RecordMap */+    RecordMap& lookupArity(size_t arity) {+        std::unordered_map<size_t, RecordMap>::iterator mapsIterator;+#pragma omp critical(RecordTableGetForArity)+        {+            // This will create a new map if it doesn't exist yet.+            mapsIterator = maps.emplace(arity, arity).first;+        }+        return mapsIterator->second;+    }++    /** Arity/RecordMap association */+    std::unordered_map<size_t, RecordMap> maps;+};++/** @brief helper to convert tuple to record reference for the synthesiser */+template <std::size_t Arity>+inline RamDomain pack(RecordTable& recordTab, Tuple<RamDomain, Arity> tuple) {+    return recordTab.pack(static_cast<RamDomain*>(tuple.data), Arity);+}++}  // namespace souffle
+ cbits/souffle/SerialisationStream.h view
@@ -0,0 +1,95 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2020, The Souffle Developers. All rights reserved.+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file SerialisationStream.h+ *+ * Defines a common base class for relation serialisation streams.+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "json11.h"++#include <cassert>+#include <cstddef>+#include <map>+#include <string>+#include <utility>+#include <vector>++namespace souffle {++class RecordTable;+class SymbolTable;++using json11::Json;++template <bool readOnlyTables>+class SerialisationStream {+public:+    virtual ~SerialisationStream() = default;++protected:+    template <typename A>+    using RO = std::conditional_t<readOnlyTables, const A, A>;++    SerialisationStream(RO<SymbolTable>& symTab, RO<RecordTable>& recTab, Json types,+            std::vector<std::string> relTypes, size_t auxArity = 0)+            : symbolTable(symTab), recordTable(recTab), types(std::move(types)),+              typeAttributes(std::move(relTypes)), arity(typeAttributes.size() - auxArity),+              auxiliaryArity(auxArity) {}++    SerialisationStream(+            RO<SymbolTable>& symTab, RO<RecordTable>& recTab, Json types, char const* const relationName)+            : symbolTable(symTab), recordTable(recTab), types(std::move(types)) {+        setupFromJson(relationName);+    }++    SerialisationStream(RO<SymbolTable>& symTab, RO<RecordTable>& recTab,+            const std::map<std::string, std::string>& rwOperation)+            : symbolTable(symTab), recordTable(recTab) {+        auto&& relationName = rwOperation.at("name");++        std::string parseErrors;+        types = Json::parse(rwOperation.at("types"), parseErrors);+        assert(parseErrors.size() == 0 && "Internal JSON parsing failed.");++        setupFromJson(relationName.c_str());+    }++    RO<SymbolTable>& symbolTable;+    RO<RecordTable>& recordTable;+    Json types;+    std::vector<std::string> typeAttributes;++    size_t arity = 0;+    size_t auxiliaryArity = 0;++private:+    void setupFromJson(char const* const relationName) {+        auto&& relInfo = types[relationName];+        arity = static_cast<size_t>(relInfo["arity"].long_value());+        auxiliaryArity = static_cast<size_t>(relInfo["auxArity"].long_value());++        assert(relInfo["types"].is_array());+        auto&& relTypes = relInfo["types"].array_items();+        assert(relTypes.size() == (arity + auxiliaryArity));++        for (size_t i = 0; i < arity + auxiliaryArity; ++i) {+            auto&& type = relTypes[i].string_value();+            assert(!type.empty() && "malformed types tag");+            typeAttributes.push_back(type);+        }+    }+};++}  // namespace souffle
+ cbits/souffle/SignalHandler.h view
@@ -0,0 +1,172 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2017, The Souffle Developers. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file SignalHandler.h+ *+ * A signal handler for Souffle's interpreter and compiler.+ *+ ***********************************************************************/++#pragma once++#include <atomic>+#include <csignal>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <iostream>+#include <mutex>+#include <string>++namespace souffle {++/**+ * Class SignalHandler captures signals+ * and reports the context where the signal occurs.+ * The signal handler is implemented as a singleton.+ */+class SignalHandler {+public:+    // get singleton+    static SignalHandler* instance() {+        static SignalHandler singleton;+        return &singleton;+    }++    // Enable logging+    void enableLogging() {+        logMessages = true;+    }+    // set signal message+    void setMsg(const char* m) {+        if (logMessages && m != nullptr) {+            static std::mutex outputMutex;+            static bool sameLine = false;+            std::lock_guard<std::mutex> guard(outputMutex);+            if (msg != nullptr && strcmp(m, msg) == 0) {+                std::cout << ".";+                sameLine = true;+            } else {+                if (sameLine) {+                    sameLine = false;+                    std::cout << std::endl;+                }+                std::string outputMessage(m);+                for (char& c : outputMessage) {+                    if (c == '\n' || c == '\t') {+                        c = ' ';+                    }+                }+                std::cout << "Starting work on " << outputMessage << std::endl;+            }+        }+        msg = m;+    }++    /***+     * set signal handlers+     */+    void set() {+        if (!isSet && std::getenv("SOUFFLE_ALLOW_SIGNALS") == nullptr) {+            // register signals+            // floating point exception+            if ((prevFpeHandler = signal(SIGFPE, handler)) == SIG_ERR) {+                perror("Failed to set SIGFPE signal handler.");+                exit(1);+            }+            // user interrupts+            if ((prevIntHandler = signal(SIGINT, handler)) == SIG_ERR) {+                perror("Failed to set SIGINT signal handler.");+                exit(1);+            }+            // memory issues+            if ((prevSegVHandler = signal(SIGSEGV, handler)) == SIG_ERR) {+                perror("Failed to set SIGSEGV signal handler.");+                exit(1);+            }+            isSet = true;+        }+    }++    /***+     * reset signal handlers+     */+    void reset() {+        if (isSet) {+            // reset floating point exception+            if (signal(SIGFPE, prevFpeHandler) == SIG_ERR) {+                perror("Failed to reset SIGFPE signal handler.");+                exit(1);+            }+            // user interrupts+            if (signal(SIGINT, prevIntHandler) == SIG_ERR) {+                perror("Failed to reset SIGINT signal handler.");+                exit(1);+            }+            // memory issues+            if (signal(SIGSEGV, prevSegVHandler) == SIG_ERR) {+                perror("Failed to reset SIGSEGV signal handler.");+                exit(1);+            }+            isSet = false;+        }+    }++    /***+     * error handling routine that prints the rule context.+     */++    void error(const std::string& error) {+        if (msg != nullptr) {+            std::cerr << error << " in rule:\n" << msg << std::endl;+        } else {+            std::cerr << error << std::endl;+        }+        exit(1);+    }++private:+    // signal context information+    std::atomic<const char*> msg;++    // state of signal handler+    bool isSet = false;++    bool logMessages = false;++    // previous signal handler routines+    void (*prevFpeHandler)(int) = nullptr;+    void (*prevIntHandler)(int) = nullptr;+    void (*prevSegVHandler)(int) = nullptr;++    /**+     * Signal handler for various types of signals.+     */+    static void handler(int signal) {+        const char* msg = instance()->msg;+        std::string error;+        switch (signal) {+            case SIGINT: error = "Interrupt"; break;+            case SIGFPE: error = "Floating-point arithmetic exception"; break;+            case SIGSEGV: error = "Segmentation violation"; break;+            default: error = "Unknown"; break;+        }+        if (msg != nullptr) {+            std::cerr << error << " signal in rule:\n" << msg << std::endl;+        } else {+            std::cerr << error << " signal." << std::endl;+        }+        exit(1);+    }++    SignalHandler() : msg(nullptr) {}+};++}  // namespace souffle
+ cbits/souffle/SouffleInterface.h view
@@ -0,0 +1,1055 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file CompiledSouffle.h+ *+ * Main include file for generated C++ classes of Souffle+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "SymbolTable.h"+#include <algorithm>+#include <cassert>+#include <cstddef>+#include <cstdint>+#include <initializer_list>+#include <iostream>+#include <map>+#include <memory>+#include <string>+#include <tuple>+#include <utility>+#include <vector>++namespace souffle {++class tuple;++/**+ * Object-oriented wrapper class for Souffle's templatized relations.+ */+class Relation {+protected:+    /**+     * Abstract iterator class.+     *+     * When tuples are inserted into a relation, they will be stored contiguously.+     * Intially, the iterator_base of a relation will point to the first tuple inserted.+     * iterator_base can be moved to point to the next tuple until the end.+     * The tuple iterator_base is pointing to can be accessed.+     * However, users can not use this to access tuples since iterator class is protected.+     * Instead, they should use the public class - iterator which interacts with iterator_base.+     */+    class iterator_base {+    protected:+        /**+         * Required for identifying type of iterator+         * (NB: LLVM has no typeinfo).+         *+         * TODO (Honghyw) : Provide a clear documentation of what id is used for.+         */+        uint32_t id;++    public:+        /**+         * Get the ID of the iterator_base object.+         *+         * @return ID of the iterator_base object (unit32_t)+         */+        virtual uint32_t getId() const {+            return id;+        }++        /**+         * Constructor.+         *+         * Create an instance of iterator_base and set its ID to be arg_id.+         *+         * @param arg_id ID of an iterator object (unit32_t)+         */+        iterator_base(uint32_t arg_id) : id(arg_id) {}++        /**+         * Destructor.+         */+        virtual ~iterator_base() = default;++        /**+         * Overload the "++" operator.+         *+         * Increment the iterator_base so that the iterator_base will now point to the next tuple.+         * The definition of this overloading has to be defined by the child class of iterator_base.+         */+        virtual void operator++() = 0;++        /**+         * Overload the "*" operator.+         *+         * Return the tuple that is pointed to by the iterator_base.+         * The definition of this overloading has to be defined by the child class of iterator_base.+         *+         * @return tuple Reference to a tuple object+         */+        virtual tuple& operator*() = 0;++        /**+         * Overload the "==" operator.+         *+         * @param o Reference to an object of the iterator_base class+         * @return A boolean value, if the ID of o is the same as the ID of the current object and equal(o)+         * returns true. Otherwise return false+         */+        bool operator==(const iterator_base& o) const {+            return this->getId() == o.getId() && equal(o);+        }++        /**+         * Clone the iterator_base.+         * The definition of clone has to be defined by the child class of iterator_base.+         *+         * @return An iterator_base pointer+         */+        virtual iterator_base* clone() const = 0;++    protected:+        /**+         * Check if the passed-in object of o is the the same as the current iterator_base.+         *+         * TODO (Honghyw) : Provide a clear documentation of what equal function does.+         *+         * @param o Reference to an object of the iterator_base class+         * @return A boolean value. If two iterator_base are the same return true. Otherwise return false+         */+        virtual bool equal(const iterator_base& o) const = 0;+    };++public:+    /**+     * Destructor.+     */+    virtual ~Relation() = default;++    /**+     * Wrapper class for abstract iterator.+     *+     * Users must use iterator class to access the tuples stored in a relation.+     */+    class iterator {+    protected:+        /*+         * iterator_base class pointer.+         *+         */+        std::unique_ptr<iterator_base> iter = nullptr;++    public:+        /**+         * Constructor.+         */+        iterator() = default;++        /**+         * Move constructor.+         *+         * The new iterator now has ownerhsip of the iterator base.+         *+         * @param arg lvalue reference to an iterator object+         */+        iterator(iterator&& arg) = default;++        /**+         * Constructor.+         *+         * Initialise this iterator with a given iterator base+         *+         * The new iterator has ownership of the iterator base.+         *+         * @param arg An iterator_base class pointer+         */+        iterator(iterator_base* arg) : iter(arg) {}++        /**+         * Destructor.+         *+         * The iterator_base instance iter is pointing is destructed.+         */+        ~iterator() = default;++        /**+         * Constructor.+         *+         * Initialise the iter to be the clone of arg.+         *+         * @param o Reference to an iterator object+         */+        iterator(const iterator& o) : iter(o.iter->clone()) {}++        /**+         * Overload the "=" operator.+         *+         * The original iterator_base instance is destructed.+         */+        iterator& operator=(const iterator& o) {+            iter.reset(o.iter->clone());+            return *this;+        }++        iterator& operator=(iterator&& o) {+            iter.swap(o.iter);+            return *this;+        }++        /**+         * Overload the "++" operator.+         *+         * Increment the iterator_base object that iter is pointing to so that iterator_base object points to+         * next tuple.+         *+         * @return Reference to the iterator object which points to the next tuple in a relation+         */+        iterator& operator++() {+            ++(*iter);+            return *this;+        }++        /**+         * Overload the "*" operator.+         *+         * This will return the tuple that the iterator is pointing to.+         *+         * @return Reference to a tuple object+         */++        tuple& operator*() const {+            return *(*iter);+        }++        /**+         * Overload the "==" operator.+         *+         * Check if either the iter of o and the iter of current object are the same or the corresponding+         * iterator_base objects are the same.+         *+         * @param o Reference to a iterator object+         * @return Boolean. True, if either of them is true. False, otherwise+         */+        bool operator==(const iterator& o) const {+            return (iter == o.iter) || (*iter == *o.iter);+        }++        /**+         * Overload the "!=" operator.+         *+         * Check if the iterator object o is not the same as the current object.+         *+         * @param o Reference to a iterator object+         * @return Boolean. True, if they are not the same. False, otherwise+         */+        bool operator!=(const iterator& o) const {+            return !(*this == o);+        }+    };++    /**+     * Insert a new tuple into the relation.+     * The definition of insert function has to be defined by the child class of relation class.+     *+     * @param t Reference to a tuple class object+     */+    virtual void insert(const tuple& t) = 0;++    /**+     * Check whether a tuple exists in a relation.+     * The definition of contains has to be defined by the child class of relation class.+     *+     * @param t Reference to a tuple object+     * @return Boolean. True, if the tuple exists. False, otherwise+     */+    virtual bool contains(const tuple& t) const = 0;++    /**+     * Return an iterator pointing to the first tuple of the relation.+     * This iterator is used to access the tuples of the relation.+     *+     * @return Iterator+     */+    virtual iterator begin() const = 0;++    /**+     * Return an iterator pointing to next to the last tuple of the relation.+     *+     * @return Iterator+     */+    virtual iterator end() const = 0;++    /**+     * Get the number of tuples in a relation.+     *+     * @return The number of tuples in a relation (std::size_t)+     */+    virtual std::size_t size() const = 0;++    /**+     * Get the name of a relation.+     *+     * @return The name of a relation (std::string)+     */+    virtual std::string getName() const = 0;++    /**+     * Get the attribute type of a relation at the column specified by the parameter.+     * The attribute type is in the form "<primitive type>:<type name>".+     * <primitive type> can be s or n standing for symbol or number which are two primitive types in Souffle.+     * <type name> is the name given by the user in the Souffle Program after ".type", "symbol_type" or+     * "number_type". For example, for ".type Node", the type name is "Node".+     *+     * @param The index of the column starting starting from 0 (size_t)+     * @return The constant string of the attribute type+     */+    virtual const char* getAttrType(size_t) const = 0;++    /**+     * Get the attribute name of a relation at the column specified by the parameter.+     * The attribute name is the name given to the type by the user in the .decl statement. For example, for+     * ".decl edge (node1:Node, node2:Node)", the attribute names are node1 and node2.+     *+     * @param The index of the column starting starting from 0 (size_t)+     * @return The constant string of the attribute name+     */+    virtual const char* getAttrName(size_t) const = 0;++    /**+     * Return the arity of a relation.+     * For example for a tuple (1 2) the arity is 2 and for a tuple (1 2 3) the arity is 3.+     *+     * @return Arity of a relation (size_t)+     */+    virtual size_t getArity() const = 0;++    /**+     * Return the number of auxiliary attributes. Auxiliary attributes+     * are used for provenance and and other alternative evaluation+     * strategies. They are stored as the last attributes of a tuple.+     *+     * @return Number of auxiliary attributes of a relation (size_t)+     */+    virtual size_t getAuxiliaryArity() const = 0;++    /**+     * Get the symbol table of a relation.+     * The symbols in a tuple to be stored into a relation are stored and assigned with a number in a table+     * called symbol table. For example, to insert ("John","Student") to a relation, "John" and "Student" are+     * stored in symbol table and they are assigned with number say 0 and 1. After this, instead of inserting+     * ("John","Student"), (0, 1) is inserted. When accessing this tuple, 0 and 1 will be looked up in the+     * table and replaced by "John" and "Student". This is done so to save memory space if same symbols are+     * inserted many times. Symbol table has many rows where each row contains a symbol and its corresponding+     * assigned number.+     *+     * @return Reference to a symbolTable object+     */+    virtual SymbolTable& getSymbolTable() const = 0;++    /**+     * Get the signature of a relation.+     * The signature is in the form <<primitive type 1>:<type name 1>,<primitive type 2>:<type name 2>...> for+     * all the attributes in a relation. For example, <s:Node,s:Node>. The primitive type and type name are+     * explained in getAttrType.+     *+     * @return String of the signature of a relation+     */+    std::string getSignature() {+        if (getArity() == 0) {+            return "<>";+        }++        std::string signature = "<" + std::string(getAttrType(0));+        for (size_t i = 1; i < getArity(); i++) {+            signature += "," + std::string(getAttrType(i));+        }+        signature += ">";+        return signature;+    }++    /**+     * Delete all the tuples in relation.+     *+     * When purge() is called, it sets the head and tail of the table (table is a+     * singly-linked list structure) to nullptr, and for every elements+     * in the table, set the next element pointer points to the current element itself.+     */+    virtual void purge() = 0;+};++/**+ * Defines a tuple for the OO interface such that+ * relations with varying columns can be accessed.+ *+ * Tuples are stored in relations.+ * In Souffle, one row of data to be stored into a relation is represented as a tuple.+ * For example if we have a relation called dog with attributes name, colour and age which are string, string+ * and interger type respectively. One row of data a relation to be stored can be (mydog, black, 3). However,+ * this is not directly stored as a tuple. There will be a symbol table storing the actual content and+ * associate them with numbers (For example, |1|mydog| |2|black| |3|3|). And when this row of data is stored+ * as a tuple, (1, 2, 3) will be stored.+ */+class tuple {+    /**+     * The relation to which the tuple belongs.+     */+    const Relation& relation;++    /**+     * Dynamic array used to store the elements in a tuple.+     */+    std::vector<RamDomain> array;++    /**+     * pos shows what the current position of a tuple is.+     * Initially, pos is 0 meaning we are at the head of the tuple.+     * If we have an empty tuple and try to insert things, pos lets us know where to insert the element. After+     * the element is inserted, pos will be incremented by 1. If we have a tuple with content, pos lets us+     * know where to read the element. After we have read one element, pos will be incremented by 1. pos also+     * helps to make sure we access an insert a tuple within the bound by making sure pos never exceeds the+     * arity of the relation.+     */+    size_t pos;++public:+    /**+     * Constructor.+     *+     * Tuples are constructed here by passing a relation, then may be subsequently inserted into that same+     * passed relation. The passed relation pointer will be stored within the tuple instance, while the arity+     * of the relation will be used to initialize the vector holding the elements of the tuple. Where such an+     * element is of integer type, it will be stored directly within the vector. Otherwise, if the element is+     * of a string type, the index of that string within the associated symbol table will be stored instead.+     * The tuple also stores the index of some "current" element, referred to as its position. The constructor+     * initially sets this position to the first (zeroth) element of the tuple, while subsequent methods of+     * this class use that position for element access and modification.+     *+     * @param r Relation pointer pointing to a relation+     */+    tuple(const Relation* r) : relation(*r), array(r->getArity()), pos(0), data(array.data()) {}++    /**+     * Constructor.+     *+     * Tuples are constructed here by passing a tuple.+     * The relation to which the passed tuple belongs to will be stored.+     * The array of the passed tuple, which stores the elements will be stroed.+     * The pos will be set to be the same as the pos of passed tuple.+     * belongs to.+     *+     * @param Reference to a tuple object.+     */+    tuple(const tuple& t) : relation(t.relation), array(t.array), pos(t.pos), data(array.data()) {}++    /**+     * Allows printing using WriteStream.+     */+    const RamDomain* data = nullptr;++    /**+     * Get the reference to the relation to which the tuple belongs.+     *+     * @return Reference to a relation.+     */+    const Relation& getRelation() const {+        return relation;+    }++    /**+     * Return the number of elements in the tuple.+     *+     * @return the number of elements in the tuple (size_t).+     */+    size_t size() const {+        return array.size();+    }++    /**+     * Overload the operator [].+     *+     * Direct access to tuple elements via index and+     * return the element in idx position of a tuple.+     *+     * TODO (Honghyw) : This interface should be hidden and+     * only be used by friendly classes such as+     * iterators; users should not use this interface.+     *+     * @param idx This is the idx of element in a tuple (size_t).+     */+    RamDomain& operator[](size_t idx) {+        return array[idx];+    }++    /**+     * Overload the operator [].+     *+     * Direct access to tuple elements via index and+     * Return the element in idx position of a tuple. The returned element can not be changed.+     *+     * TODO (Honghyw) : This interface should be hidden and+     * only be used by friendly classes such as+     * iterators; users should not use this interface.+     *+     * @param idx This is the idx of element in a tuple (size_t).+     */+    const RamDomain& operator[](size_t idx) const {+        return array[idx];+    }++    /**+     * Reset the index giving the "current element" of the tuple to zero.+     */+    void rewind() {+        pos = 0;+    }++    /**+     * Set the "current element" of the tuple to the given string, then increment the index giving the current+     * element.+     *+     * @param str Symbol to be added (std::string)+     * @return Reference to the tuple+     */+    tuple& operator<<(const std::string& str) {+        assert(pos < size() && "exceeded tuple's size");+        assert(*relation.getAttrType(pos) == 's' && "wrong element type");+        array[pos++] = relation.getSymbolTable().lookup(str);+        return *this;+    }++    /**+     * Set the "current element" of the tuple to the given int, then increment the index giving the current+     * element.+     *+     * @param integer Integer to be added+     * @return Reference to the tuple+     */+    tuple& operator<<(RamSigned integer) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'i' || *relation.getAttrType(pos) == 'r') &&+                "wrong element type");+        array[pos++] = integer;+        return *this;+    }++    /**+     * Set the "current element" of the tuple to the given uint, then increment the index giving the current+     * element.+     *+     * @param uint Unsigned number to be added+     * @return Reference to the tuple+     */+    tuple& operator<<(RamUnsigned uint) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'u') && "wrong element type");+        array[pos++] = ramBitCast(uint);+        return *this;+    }++    /**+     * Set the "current element" of the tuple to the given float, then increment the index giving the current+     * element.+     *+     * @param float float to be added+     * @return Reference to the tuple+     */+    tuple& operator<<(RamFloat ramFloat) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'f') && "wrong element type");+        array[pos++] = ramBitCast(ramFloat);+        return *this;+    }++    /**+     * Get the "current element" of the tuple as a string, then increment the index giving the current+     * element.+     *+     * @param str Symbol to be loaded from the tuple(std::string)+     * @return Reference to the tuple+     */+    tuple& operator>>(std::string& str) {+        assert(pos < size() && "exceeded tuple's size");+        assert(*relation.getAttrType(pos) == 's' && "wrong element type");+        str = relation.getSymbolTable().resolve(array[pos++]);+        return *this;+    }++    /**+     * Get the "current element" of the tuple as a int, then increment the index giving the current+     * element.+     *+     * @param integer Integer to be loaded from the tuple+     * @return Reference to the tuple+     */+    tuple& operator>>(RamSigned& integer) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'i' || *relation.getAttrType(pos) == 'r') &&+                "wrong element type");+        integer = ramBitCast<RamSigned>(array[pos++]);+        return *this;+    }++    /**+     * Get the "current element" of the tuple as a unsigned, then increment the index giving the current+     * element.+     *+     * @param uint Unsigned number to be loaded from the tuple+     * @return Reference to the tuple+     */+    tuple& operator>>(RamUnsigned& uint) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'u') && "wrong element type");+        uint = ramBitCast<RamUnsigned>(array[pos++]);+        return *this;+    }++    /**+     * Get the "current element" of the tuple as a float, then increment the index giving the current+     * element.+     *+     * @param ramFloat Float to be loaded from the tuple+     * @return Reference to the tuple+     */+    tuple& operator>>(RamFloat& ramFloat) {+        assert(pos < size() && "exceeded tuple's size");+        assert((*relation.getAttrType(pos) == 'f') && "wrong element type");+        ramFloat = ramBitCast<RamFloat>(array[pos++]);+        return *this;+    }++    /**+     * Iterator for direct access to tuple's data.+     *+     * @see Relation::iteraor::begin()+     */+    decltype(array)::iterator begin() {+        return array.begin();+    }++    /**+     * Construct using initialisation list.+     */+    tuple(const Relation* relation, std::initializer_list<RamDomain> tupleList)+            : relation(*relation), array(tupleList), pos(tupleList.size()), data(array.data()) {+        assert(tupleList.size() == relation->getArity() && "tuple arity does not match relation arity");+    }+};++/**+ * Abstract base class for generated Datalog programs.+ */+class SouffleProgram {+private:+    /**+     * Define a relation map for external access, when getRelation(name) is called,+     * the relation with the given name will be returned from this map,+     * relationMap stores all the relations in a map with its name+     * as the key and relation as the value.+     */+    std::map<std::string, Relation*> relationMap;++    /**+     * inputRelations stores all the input relation in a vector.+     */+    std::vector<Relation*> inputRelations;++    /**+     * outputRelations stores all the output relation in a vector.+     */+    std::vector<Relation*> outputRelations;++    /**+     * internalRelation stores all the relation in a vector that are neither an input or an output.+     */+    std::vector<Relation*> internalRelations;++    /**+     * allRelations store all the relation in a vector.+     */+    std::vector<Relation*> allRelations;+    std::size_t numThreads = 1;++protected:+    /**+     * Add the relation to relationMap (with its name) and allRelations,+     * depends on the properties of the relation, if the relation is an input relation, it will be added to+     * inputRelations, else if the relation is an output relation, it will be added to outputRelations,+     * otherwise will add to internalRelations. (a relation could be both input and output at the same time.)+     *+     * @param name the name of the relation (std::string)+     * @param rel a pointer to the relation (std::string)+     * @param isInput a bool argument, true if the relation is a input relation, else false (bool)+     * @param isOnput a bool argument, true if the relation is a ouput relation, else false (bool)+     */+    void addRelation(const std::string& name, Relation* rel, bool isInput, bool isOutput) {+        relationMap[name] = rel;+        allRelations.push_back(rel);+        if (isInput) {+            inputRelations.push_back(rel);+        }+        if (isOutput) {+            outputRelations.push_back(rel);+        }+        if (!isInput && !isOutput) {+            internalRelations.push_back(rel);+        }+    }++public:+    /**+     * Destructor.+     *+     * Destructor of SouffleProgram.+     */+    virtual ~SouffleProgram() = default;++    /**+     * Execute the souffle program, without any loads or stores.+     */+    virtual void run() {}++    /**+     * Execute program, loading inputs and storing outputs as required.+     * Read all input relations and store all output relations from the given directory.+     * First argument is the input directory, second argument is the output directory,+     * if no directory is given, the default is to use the current working directory.+     * The implementation of this function occurs in the C++ code generated by Souffle.+     * To view the generated C++ code, run Souffle with the `-g` option.+     */+    virtual void runAll(std::string inputDirectory = ".", std::string outputDirectory = ".") = 0;++    /**+     * Read all input relations from the given directory. If no directory is given, the+     * default is to use the current working directory. The implementation of this function+     * occurs in the C++ code generated by Souffle. To view the generated C++ code, run+     * Souffle with the `-g` option.+     */+    virtual void loadAll(std::string inputDirectory = ".") = 0;++    /**+     * Store all output relations individually in .CSV file in the given directory, with+     * the relation name as the file name of the .CSV file. If no directory is given, the+     * default is to use the current working directory. The implementation of this function+     * occurs in the C++ code generated by Souffle. To view the generated C++ code, run+     * Souffle with the `-g` option.+     */+    virtual void printAll(std::string outputDirectory = ".") = 0;++    /**+     * Output all the input relations in stdout, without generating any files. (for debug purposes).+     */+    virtual void dumpInputs(std::ostream& out = std::cout) = 0;++    /**+     * Output all the output relations in stdout, without generating any files. (for debug purposes).+     */+    virtual void dumpOutputs(std::ostream& out = std::cout) = 0;++    /**+     * Set the number of threads to be used+     */+    void setNumThreads(std::size_t numThreadsValue) {+        this->numThreads = numThreadsValue;+    }++    /**+     * Get the number of threads to be used+     */+    std::size_t getNumThreads() {+        return numThreads;+    }++    /**+     * Get Relation by its name from relationMap, if relation not found, return a nullptr.+     *+     * @param name The name of the target relation (const std::string)+     * @return The pointer of the target relation, or null pointer if the relation not found (Relation*)+     */+    Relation* getRelation(const std::string& name) const {+        auto it = relationMap.find(name);+        if (it != relationMap.end()) {+            return (*it).second;+        } else {+            return nullptr;+        }+    };++    /**+     * Return the size of the target relation from relationMap.+     *+     * @param name The name of the target relation (const std::string)+     * @return The size of the target relation (std::size_t)+     */+    std::size_t getRelationSize(const std::string& name) const {+        return getRelation(name)->size();+    }++    /**+     * Return the name of the target relation from relationMap.+     *+     * @param name The name of the target relation (const std::string)+     * @return The name of the target relation (std::string)+     */+    std::string getRelationName(const std::string& name) const {+        return getRelation(name)->getName();+    }++    /**+     * Getter of outputRelations, which this vector structure contains all output relations.+     *+     * @return outputRelations (std::vector)+     * @see outputRelations+     */+    std::vector<Relation*> getOutputRelations() const {+        return outputRelations;+    }++    /**+     * Getter of inputRelations, which this vector structure contains all input relations.+     *+     * @return intputRelations (std::vector)+     * @see inputRelations+     */+    std::vector<Relation*> getInputRelations() const {+        return inputRelations;+    }++    /**+     * Getter of internalRelations, which this vector structure contains all relations+     * that are neither an input relation or an output relation.+     *+     * @return internalRelations (std::vector)+     * @see internalRelations+     */+    std::vector<Relation*> getInternalRelations() const {+        return internalRelations;+    }++    /**+     * Getter of allRelations, which this vector structure contains all relations.+     *+     * @return allRelations (std::vector)+     * @see allRelations+     */+    std::vector<Relation*> getAllRelations() const {+        return allRelations;+    }++    /**+     * Execute a subroutine+     * @param name  Name of a subroutine (std:string)+     * @param arg Arguments of the subroutine (std::vector<RamDomain>&)+     * @param ret Return values of the subroutine (std::vector<RamDomain>&)+     */+    virtual void executeSubroutine(std::string /* name */, const std::vector<RamDomain>& /* args */,+            std::vector<RamDomain>& /* ret */) {+        fatal("unknown subroutine");+    }++    /**+     * Get the symbol table of the program.+     */+    virtual SymbolTable& getSymbolTable() = 0;++    /**+     * Remove all the tuples from the outputRelations, calling the purge method of each.+     *+     * @see Relation::purge()+     */+    void purgeOutputRelations() {+        for (Relation* relation : outputRelations) {+            relation->purge();+        }+    }++    /**+     * Remove all the tuples from the inputRelations, calling the purge method of each.+     *+     * @see Relation::purge()+     */+    void purgeInputRelations() {+        for (Relation* relation : inputRelations) {+            relation->purge();+        }+    }++    /**+     * Remove all the tuples from the internalRelations, calling the purge method of each.+     *+     * @see Relation::purge()+     */+    void purgeInternalRelations() {+        for (Relation* relation : internalRelations) {+            relation->purge();+        }+    }++    /**+     * Helper function for the wrapper function Relation::insert() and Relation::contains().+     */+    template <typename Tuple, size_t N>+    struct tuple_insert {+        static void add(const Tuple& t, souffle::tuple& t1) {+            tuple_insert<Tuple, N - 1>::add(t, t1);+            t1 << std::get<N - 1>(t);+        }+    };++    /**+     * Helper function for the wrapper function Relation::insert() and Relation::contains() for the+     * first element of the tuple.+     */+    template <typename Tuple>+    struct tuple_insert<Tuple, 1> {+        static void add(const Tuple& t, souffle::tuple& t1) {+            t1 << std::get<0>(t);+        }+    };++    /**+     * Insert function with std::tuple as input (wrapper)+     *+     * @param t The insert tuple (std::tuple)+     * @param relation The relation that perform insert operation (Relation*)+     * @see Relation::insert()+     */+    template <typename... Args>+    void insert(const std::tuple<Args...>& t, Relation* relation) {+        tuple t1(relation);+        tuple_insert<decltype(t), sizeof...(Args)>::add(t, t1);+        relation->insert(t1);+    }++    /**+     * Contains function with std::tuple as input (wrapper)+     *+     * @param t The existence searching tuple (std::tuple)+     * @param relation The relation that perform contains operation (Relation*)+     * @return A boolean value, return true if the tuple found, otherwise return false+     * @see Relation::contains()+     */+    template <typename... Args>+    bool contains(const std::tuple<Args...>& t, Relation* relation) {+        tuple t1(relation);+        tuple_insert<decltype(t), sizeof...(Args)>::add(t, t1);+        return relation->contains(t1);+    }+};++/**+ * Abstract program factory class.+ */+class ProgramFactory {+protected:+    /**+     * Singly linked-list to store all program factories+     * Note that STL data-structures are not possible due+     * to "static initialization order fiasco (problem)".+     * (The problem of the order static objects get initialized, causing effect+     * such as program access static variables before they initialized.)+     * The static container needs to be a primitive type such as pointer+     * set to NULL.+     * Link to next factory.+     */+    ProgramFactory* link = nullptr;++    /**+     * The name of factory.+     */+    std::string name;++protected:+    /**+     * Constructor.+     *+     * Constructor adds factory to static singly-linked list+     * for registration.+     */+    ProgramFactory(std::string name) : name(std::move(name)) {+        registerFactory(this);+    }++private:+    /**+     * Helper method for creating a factory map, which map key is the name of the program factory, map value+     * is the pointer of the ProgramFactory.+     *+     * TODO (NubKel) : Improve documentation of use and interaction between inline and static, here and for+     * the whole class.+     *+     * @return The factory registration map (std::map)+     */+    static inline std::map<std::string, ProgramFactory*>& getFactoryRegistry() {+        static std::map<std::string, ProgramFactory*> factoryReg;+        return factoryReg;+    }++protected:+    /**+     * Create and insert a factory into the factoryReg map.+     *+     * @param factory Pointer of the program factory (ProgramFactory*)+     */+    static inline void registerFactory(ProgramFactory* factory) {+        auto& entry = getFactoryRegistry()[factory->name];+        assert(!entry && "double-linked/defined souffle analyis");+        entry = factory;+    }++    /**+     * Find a factory by its name, return the fatory if found, return nullptr if the+     * factory not found.+     *+     * @param factoryName The factory name (const std::string)+     * @return The pointer of the target program factory, or null pointer if the program factory not found+     * (ProgramFactory*)+     */+    static inline ProgramFactory* find(const std::string& factoryName) {+        const auto& reg = getFactoryRegistry();+        auto pos = reg.find(factoryName);+        return (pos == reg.end()) ? nullptr : pos->second;+    }++    /**+     * Create new instance (abstract).+     */+    virtual SouffleProgram* newInstance() = 0;++public:+    /**+     * Destructor.+     *+     * Destructor of ProgramFactory.+     */+    virtual ~ProgramFactory() = default;++    /**+     * Create an instance by finding the name of the program factory, return nullptr if the instance not+     * found.+     *+     * @param name Instance name (const std::string)+     * @return The new instance(SouffleProgram*), or null pointer if the instance not found+     */+    static SouffleProgram* newInstance(const std::string& name) {+        ProgramFactory* factory = find(name);+        if (factory != nullptr) {+            return factory->newInstance();+        } else {+            return nullptr;+        }+    }+};+}  // namespace souffle
+ cbits/souffle/SymbolTable.h view
@@ -0,0 +1,243 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file SymbolTable.h+ *+ * Data container to store symbols of the Datalog program.+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "utility/MiscUtil.h"+#include "utility/ParallelUtil.h"+#include "utility/StreamUtil.h"+#include <algorithm>+#include <cstdlib>+#include <deque>+#include <initializer_list>+#include <iostream>+#include <string>+#include <unordered_map>+#include <utility>+#include <vector>++namespace souffle {++/**+ * @class SymbolTable+ *+ * Global pool of re-usable strings+ *+ * SymbolTable stores Datalog symbols and converts them to numbers and vice versa.+ */+class SymbolTable {+private:+    /** A lock to synchronize parallel accesses */+    mutable Lock access;++    /** Map indices to strings. */+    std::deque<std::string> numToStr;++    /** Map strings to indices. */+    std::unordered_map<std::string, size_t> strToNum;++    /** Convenience method to place a new symbol in the table, if it does not exist, and return the index of+     * it. */+    inline size_t newSymbolOfIndex(const std::string& symbol) {+        size_t index;+        auto it = strToNum.find(symbol);+        if (it == strToNum.end()) {+            index = numToStr.size();+            strToNum[symbol] = index;+            numToStr.push_back(symbol);+        } else {+            index = it->second;+        }+        return index;+    }++    /** Convenience method to place a new symbol in the table, if it does not exist. */+    inline void newSymbol(const std::string& symbol) {+        if (strToNum.find(symbol) == strToNum.end()) {+            strToNum[symbol] = numToStr.size();+            numToStr.push_back(symbol);+        }+    }++public:+    /** Empty constructor. */+    SymbolTable() = default;++    /** Copy constructor, performs a deep copy. */+    SymbolTable(const SymbolTable& other) : numToStr(other.numToStr), strToNum(other.strToNum) {}++    /** Copy constructor for r-value reference. */+    SymbolTable(SymbolTable&& other) noexcept {+        numToStr.swap(other.numToStr);+        strToNum.swap(other.strToNum);+    }++    SymbolTable(std::initializer_list<std::string> symbols) {+        strToNum.reserve(symbols.size());+        for (const auto& symbol : symbols) {+            newSymbol(symbol);+        }+    }++    /** Destructor, frees memory allocated for all strings. */+    virtual ~SymbolTable() = default;++    /** Assignment operator, performs a deep copy and frees memory allocated for all strings. */+    SymbolTable& operator=(const SymbolTable& other) {+        if (this == &other) {+            return *this;+        }+        numToStr = other.numToStr;+        strToNum = other.strToNum;+        return *this;+    }++    /** Assignment operator for r-value references. */+    SymbolTable& operator=(SymbolTable&& other) noexcept {+        numToStr.swap(other.numToStr);+        strToNum.swap(other.strToNum);+        return *this;+    }++    /** Find the index of a symbol in the table, inserting a new symbol if it does not exist there+     * already. */+    RamDomain lookup(const std::string& symbol) {+        {+            auto lease = access.acquire();+            (void)lease;  // avoid warning;+            return static_cast<RamDomain>(newSymbolOfIndex(symbol));+        }+    }++    /** Finds the index of a symbol in the table, giving an error if it's not found */+    RamDomain lookupExisting(const std::string& symbol) const {+        {+            auto lease = access.acquire();+            (void)lease;  // avoid warning;+            auto result = strToNum.find(symbol);+            if (result == strToNum.end()) {+                fatal("Error string not found in call to `SymbolTable::lookupExisting`: `%s`", symbol);+            }+            return static_cast<RamDomain>(result->second);+        }+    }++    /** Find the index of a symbol in the table, inserting a new symbol if it does not exist there+     * already. */+    RamDomain unsafeLookup(const std::string& symbol) {+        return newSymbolOfIndex(symbol);+    }++    /** Find a symbol in the table by its index, note that this gives an error if the index is out of+     * bounds.+     */+    const std::string& resolve(const RamDomain index) const {+        {+            auto lease = access.acquire();+            (void)lease;  // avoid warning;+            auto pos = static_cast<size_t>(index);+            if (pos >= size()) {+                // TODO: use different error reporting here!!+                fatal("Error index out of bounds in call to `SymbolTable::resolve`. index = `%d`", index);+            }+            return numToStr[pos];+        }+    }++    const std::string& unsafeResolve(const RamDomain index) const {+        return numToStr[static_cast<size_t>(index)];+    }++    /* Return the size of the symbol table, being the number of symbols it currently holds. */+    size_t size() const {+        return numToStr.size();+    }++    /** Bulk insert symbols into the table, note that this operation is more efficient than repeated+     * inserts+     * of single symbols. */+    void insert(const std::vector<std::string>& symbols) {+        {+            auto lease = access.acquire();+            (void)lease;  // avoid warning;+            strToNum.reserve(size() + symbols.size());+            for (auto& symbol : symbols) {+                newSymbol(symbol);+            }+        }+    }++    /** Insert a single symbol into the table, not that this operation should not be used if inserting+     * symbols+     * in bulk. */+    void insert(const std::string& symbol) {+        {+            auto lease = access.acquire();+            (void)lease;  // avoid warning;+            newSymbol(symbol);+        }+    }++    /** Print the symbol table to the given stream. */+    void print(std::ostream& out) const {+        {+            out << "SymbolTable: {\n\t";+            out << join(strToNum, "\n\t",+                           [](std::ostream& out, const std::pair<std::string, std::size_t>& entry) {+                               out << entry.first << "\t => " << entry.second;+                           })+                << "\n";+            out << "}\n";+        }+    }++    /** Check if the symbol table contains a string */+    bool contains(const std::string& symbol) const {+        auto lease = access.acquire();+        (void)lease;  // avoid warning;+        auto result = strToNum.find(symbol);+        if (result == strToNum.end()) {+            return false;+        } else {+            return true;+        }+    }++    /** Check if the symbol table contains an index */+    bool contains(const RamDomain index) const {+        auto lease = access.acquire();+        (void)lease;  // avoid warning;+        auto pos = static_cast<size_t>(index);+        if (pos >= size()) {+            return false;+        } else {+            return true;+        }+    }++    Lock::Lease acquireLock() const {+        return access.acquire();+    }++    /** Stream operator, used as a convenience for print. */+    friend std::ostream& operator<<(std::ostream& out, const SymbolTable& table) {+        table.print(out);+        return out;+    }+};++}  // namespace souffle
+ cbits/souffle/Table.h view
@@ -0,0 +1,145 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file Table.h+ *+ * An implementation of a generic Table storing a position-fixed collection+ * of objects in main memory.+ *+ ***********************************************************************/++#pragma once++#include <iosfwd>+#include <iterator>++namespace souffle {++template <typename T, unsigned blockSize = 4096>+class Table {+    struct Block {+        Block* next;+        std::size_t used = 0;+        T data[blockSize];++        Block() : next(nullptr) {}++        bool isFull() const {+            return used == blockSize;+        }++        const T& append(const T& element) {+            const T& res = data[used];+            data[used] = element;+            used++;+            return res;+        }+    };++    Block* head;+    Block* tail;++    std::size_t count = 0;++public:+    class iterator : public std::iterator<std::forward_iterator_tag, T> {+        Block* block;+        unsigned pos;++    public:+        iterator(Block* block = nullptr, unsigned pos = 0) : block(block), pos(pos) {}++        iterator(const iterator&) = default;+        iterator(iterator&&) = default;+        iterator& operator=(const iterator&) = default;++        // the equality operator as required by the iterator concept+        bool operator==(const iterator& other) const {+            return (block == nullptr && other.block == nullptr) || (block == other.block && pos == other.pos);+        }++        // the not-equality operator as required by the iterator concept+        bool operator!=(const iterator& other) const {+            return !(*this == other);+        }++        // the deref operator as required by the iterator concept+        const T& operator*() const {+            return block->data[pos];+        }++        // the increment operator as required by the iterator concept+        iterator& operator++() {+            // move on in block+            if (++pos < block->used) {+                return *this;+            }+            // or to next block+            block = block->next;+            pos = 0;+            return *this;+        }+    };++    Table() : head(nullptr), tail(nullptr) {}++    ~Table() {+        clear();+    }++    bool empty() const {+        return (!head);+    }++    std::size_t size() const {+        return count;+    }++    const T& insert(const T& element) {+        // check whether the head is initialized+        if (!head) {+            head = new Block();+            tail = head;+        }++        // check whether tail is full+        if (tail->isFull()) {+            tail->next = new Block();+            tail = tail->next;+        }++        // increment counter+        count++;++        // add another element+        return tail->append(element);+    }++    iterator begin() const {+        return iterator(head);+    }++    iterator end() const {+        return iterator();+    }++    void clear() {+        while (head != nullptr) {+            auto cur = head;+            head = head->next;+            delete cur;+        }+        count = 0;+        head = nullptr;+        tail = nullptr;+    }+};++}  // end namespace souffle
+ cbits/souffle/UnionFind.h view
@@ -0,0 +1,356 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2017 The Souffle Developers. All Rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file UnionFind.h+ *+ * Defines a union-find data-structure+ *+ ***********************************************************************/++#pragma once++#include "LambdaBTree.h"+#include "PiggyList.h"+#include <atomic>+#include <cstddef>+#include <cstdint>+#include <functional>+#include <utility>++namespace souffle {++// branch predictor hacks+#define unlikely(x) __builtin_expect((x), 0)+#define likely(x) __builtin_expect((x), 1)++using rank_t = uint8_t;+/* technically uint56_t, but, doesn't exist. Just be careful about storing > 2^56 elements. */+using parent_t = uint64_t;++// number of bits that the rank is+constexpr uint8_t split_size = 8u;++// block_t stores parent in the upper half, rank in the lower half+using block_t = uint64_t;+// block_t & rank_mask extracts the rank+constexpr block_t rank_mask = (1ul << split_size) - 1;++/**+ * Structure that emulates a Disjoint Set, i.e. a data structure that supports efficient union-find operations+ */+class DisjointSet {+    template <typename TupleType>+    friend class EquivalenceRelation;++    PiggyList<std::atomic<block_t>> a_blocks;++public:+    DisjointSet() = default;++    // copy ctor+    DisjointSet(DisjointSet& other) = delete;+    // move ctor+    DisjointSet(DisjointSet&& other) = delete;++    // copy assign ctor+    DisjointSet& operator=(DisjointSet& ds) = delete;+    // move assign ctor+    DisjointSet& operator=(DisjointSet&& ds) = delete;++    /**+     * Return the number of elements in this disjoint set (not the number of pairs)+     */+    inline size_t size() {+        auto sz = a_blocks.size();+        return sz;+    };++    /**+     * Yield reference to the node by its node index+     * @param node node to be searched+     * @return the parent block of the specified node+     */+    inline std::atomic<block_t>& get(parent_t node) const {+        auto& ret = a_blocks.get(node);+        return ret;+    };++    /**+     * Equivalent to the find() function in union/find+     * Find the highest ancestor of the provided node - flattening as we go+     * @param x the node to find the parent of, whilst flattening its set-tree+     * @return The parent of x+     */+    parent_t findNode(parent_t x) {+        // while x's parent is not itself+        while (x != b2p(get(x))) {+            block_t xState = get(x);+            // yield x's parent's parent+            parent_t newParent = b2p(get(b2p(xState)));+            // construct block out of the original rank and the new parent+            block_t newState = pr2b(newParent, b2r(xState));++            this->get(x).compare_exchange_strong(xState, newState);++            x = newParent;+        }+        return x;+    }++private:+    /**+     * Update the root of the tree of which x is, to have y as the base instead+     * @param x : old root+     * @param oldrank : old root rank+     * @param y : new root+     * @param newrank : new root rank+     * @return Whether the update succeeded (fails if another root update/union has been perfomed in the+     * interim)+     */+    bool updateRoot(const parent_t x, const rank_t oldrank, const parent_t y, const rank_t newrank) {+        block_t oldState = get(x);+        parent_t nextN = b2p(oldState);+        rank_t rankN = b2r(oldState);++        if (nextN != x || rankN != oldrank) return false;+        // set the parent and rank of the new record+        block_t newVal = pr2b(y, newrank);++        return this->get(x).compare_exchange_strong(oldState, newVal);+    }++public:+    /**+     * Clears the DisjointSet of all nodes+     * Invalidates all iterators+     */+    void clear() {+        a_blocks.clear();+    }++    /**+     * Check whether the two indices are in the same set+     * @param x node to be checked+     * @param y node to be checked+     * @return where the two indices are in the same set+     */+    bool sameSet(parent_t x, parent_t y) {+        while (true) {+            x = findNode(x);+            y = findNode(y);+            if (x == y) return true;+            // if x's parent is itself, they are not the same set+            if (b2p(get(x)) == x) return false;+        }+    }++    /**+     * Union the two specified index nodes+     * @param x node to be unioned+     * @param y node to be unioned+     */+    void unionNodes(parent_t x, parent_t y) {+        while (true) {+            x = findNode(x);+            y = findNode(y);++            // no need to union if both already in same set+            if (x == y) return;++            rank_t xrank = b2r(get(x));+            rank_t yrank = b2r(get(y));++            // if x comes before y (better rank or earlier & equal node)+            if (xrank > yrank || ((xrank == yrank) && x > y)) {+                std::swap(x, y);+                std::swap(xrank, yrank);+            }+            // join the trees together+            // perhaps we can optimise the use of compare_exchange_strong here, as we're in a pessimistic loop+            if (!updateRoot(x, xrank, y, yrank)) {+                continue;+            }+            // make sure that the ranks are orderable+            if (xrank == yrank) {+                updateRoot(y, yrank, y, yrank + 1);+            }+            break;+        }+    }++    /**+     * Create a node with its parent as itself, rank 0+     * @return the newly created block+     */+    inline block_t makeNode() {+        // make node and find out where we've added it+        size_t nodeDetails = a_blocks.createNode();++        a_blocks.get(nodeDetails).store(pr2b(nodeDetails, 0));++        return a_blocks.get(nodeDetails).load();+    };++    /**+     * Extract parent from block+     * @param inblock the block to be masked+     * @return The parent_t contained in the upper half of block_t+     */+    static inline parent_t b2p(const block_t inblock) {+        return (parent_t)(inblock >> split_size);+    };++    /**+     * Extract rank from block+     * @param inblock the block to be masked+     * @return the rank_t contained in the lower half of block_t+     */+    static inline rank_t b2r(const block_t inblock) {+        return (rank_t)(inblock & rank_mask);+    };++    /**+     * Yield a block given parent and rank+     * @param parent the top half bits+     * @param rank the lower half bits+     * @return the resultant block after merge+     */+    static inline block_t pr2b(const parent_t parent, const rank_t rank) {+        return (((block_t)parent) << split_size) | rank;+    };+};++template <typename StorePair>+struct EqrelMapComparator {+    int operator()(const StorePair& a, const StorePair& b) {+        if (a.first < b.first) {+            return -1;+        } else if (b.first < a.first) {+            return 1;+        } else {+            return 0;+        }+    }++    bool less(const StorePair& a, const StorePair& b) {+        return operator()(a, b) < 0;+    }++    bool equal(const StorePair& a, const StorePair& b) {+        return operator()(a, b) == 0;+    }+};++template <typename SparseDomain>+class SparseDisjointSet {+    DisjointSet ds;++    template <typename TupleType>+    friend class EquivalenceRelation;++    using PairStore = std::pair<SparseDomain, parent_t>;+    using SparseMap =+            LambdaBTreeSet<PairStore, std::function<parent_t(PairStore&)>, EqrelMapComparator<PairStore>>;+    using DenseMap = RandomInsertPiggyList<SparseDomain>;++    typename SparseMap::operation_hints last_ins;++    SparseMap sparseToDenseMap;+    // mapping from union-find val to souffle, union-find encoded as index+    DenseMap denseToSparseMap;++public:+    /**+     * Retrieve dense encoding, adding it in if non-existent+     * @param in the sparse value+     * @return the corresponding dense value+     */+    parent_t toDense(const SparseDomain in) {+        // insert into the mapping - if the key doesn't exist (in), the function will be called+        // and a dense value will be created for it+        PairStore p = {in, -1};+        return sparseToDenseMap.insert(p, [&](PairStore& p) {+            parent_t c2 = DisjointSet::b2p(this->ds.makeNode());+            this->denseToSparseMap.insertAt(c2, p.first);+            p.second = c2;+            return c2;+        });+    }++public:+    SparseDisjointSet() = default;++    // copy ctor+    SparseDisjointSet(SparseDisjointSet& other) = delete;++    // move ctor+    SparseDisjointSet(SparseDisjointSet&& other) = delete;++    // copy assign ctor+    SparseDisjointSet& operator=(SparseDisjointSet& other) = delete;++    // move assign ctor+    SparseDisjointSet& operator=(SparseDisjointSet&& other) = delete;++    /**+     * For the given dense value, return the associated sparse value+     *   Undefined behaviour if dense value not in set+     * @param in the supplied dense value+     * @return the sparse value from the denseToSparseMap+     */+    inline const SparseDomain toSparse(const parent_t in) const {+        return denseToSparseMap.get(in);+    };++    /* a wrapper to enable checking in the sparse set - however also adds them if not already existing */+    inline bool sameSet(SparseDomain x, SparseDomain y) {+        return ds.sameSet(toDense(x), toDense(y));+    };+    /* finds the node in the underlying disjoint set, adding the node if non-existent */+    inline SparseDomain findNode(SparseDomain x) {+        return toSparse(ds.findNode(toDense(x)));+    };+    /* union the nodes, add if not existing */+    inline void unionNodes(SparseDomain x, SparseDomain y) {+        ds.unionNodes(toDense(x), toDense(y));+    };++    inline std::size_t size() {+        return ds.size();+    };++    /**+     * Remove all elements from this disjoint set+     */+    void clear() {+        ds.clear();+        sparseToDenseMap.clear();+        denseToSparseMap.clear();+    }++    /* wrapper for node creation */+    inline void makeNode(SparseDomain val) {+        // dense has the behaviour of creating if not exists.+        toDense(val);+    };++    /* whether we the supplied node exists */+    inline bool nodeExists(const SparseDomain val) const {+        return sparseToDenseMap.contains({val, -1});+    };++    inline bool contains(SparseDomain v1, SparseDomain v2) {+        if (nodeExists(v1) && nodeExists(v2)) {+            return sameSet(v1, v2);+        }+        return false;+    }+};+}  // namespace souffle
+ cbits/souffle/WriteStream.h view
@@ -0,0 +1,133 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file WriteStream.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "RecordTable.h"+#include "SerialisationStream.h"+#include "SymbolTable.h"+#include "json11.h"+#include "utility/MiscUtil.h"+#include <cassert>+#include <cstddef>+#include <map>+#include <memory>+#include <ostream>+#include <string>++namespace souffle {++using json11::Json;++class WriteStream : public SerialisationStream<true> {+public:+    WriteStream(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : SerialisationStream(symbolTable, recordTable, rwOperation),+              summary(rwOperation.at("IO") == "stdoutprintsize") {}++    template <typename T>+    void writeAll(const T& relation) {+        if (summary) {+            return writeSize(relation.size());+        }+        auto lease = symbolTable.acquireLock();+        (void)lease;  // silence "unused variable" warning+        if (arity == 0) {+            if (relation.begin() != relation.end()) {+                writeNullary();+            }+            return;+        }+        for (const auto& current : relation) {+            writeNext(current);+        }+    }++    template <typename T>+    void writeSize(const T& relation) {+        writeSize(relation.size());+    }++protected:+    const bool summary;++    virtual void writeNullary() = 0;+    virtual void writeNextTuple(const RamDomain* tuple) = 0;+    virtual void writeSize(std::size_t) {+        fatal("attempting to print size of a write operation");+    }++    template <typename Tuple>+    void writeNext(const Tuple tuple) {+        writeNextTuple(tuple.data);+    }++    void outputRecord(std::ostream& destination, const RamDomain value, const std::string& name) {+        auto&& recordInfo = types["records"][name];++        // Check if record type information are present+        assert(!recordInfo.is_null() && "Missing record type information");++        // Check for nil+        if (value == 0) {+            destination << "nil";+            return;+        }++        auto&& recordTypes = recordInfo["types"];+        const size_t recordArity = recordInfo["arity"].long_value();++        const RamDomain* tuplePtr = recordTable.unpack(value, recordArity);++        destination << "[";++        // print record's elements+        for (size_t i = 0; i < recordArity; ++i) {+            if (i > 0) {+                destination << ", ";+            }++            const std::string& recordType = recordTypes[i].string_value();+            const RamDomain recordValue = tuplePtr[i];++            switch (recordType[0]) {+                case 'i': destination << recordValue; break;+                case 'f': destination << ramBitCast<RamFloat>(recordValue); break;+                case 'u': destination << ramBitCast<RamUnsigned>(recordValue); break;+                case 's': destination << symbolTable.unsafeResolve(recordValue); break;+                case 'r': outputRecord(destination, recordValue, recordType); break;+                default: fatal("Unsupported type attribute: `%c`", recordType[0]);+            }+        }+        destination << "]";+    }+};++class WriteStreamFactory {+public:+    virtual std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable& symbolTable, const RecordTable& recordTable) = 0;++    virtual const std::string& getName() const = 0;+    virtual ~WriteStreamFactory() = default;+};++template <>+inline void WriteStream::writeNext(const RamDomain* tuple) {+    writeNextTuple(tuple);+}++} /* namespace souffle */
+ cbits/souffle/WriteStreamCSV.h view
@@ -0,0 +1,218 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file WriteStreamCSV.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "SymbolTable.h"+#include "WriteStream.h"+#include "utility/ContainerUtil.h"+#include "utility/MiscUtil.h"+#include "utility/ParallelUtil.h"+#ifdef USE_LIBZ+#include "gzfstream.h"+#endif++#include <cstddef>+#include <iostream>+#include <map>+#include <ostream>+#include <string>+#include <vector>++namespace souffle {++class RecordTable;++class WriteStreamCSV : public WriteStream {+protected:+    WriteStreamCSV(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : WriteStream(rwOperation, symbolTable, recordTable),+              delimiter(getOr(rwOperation, "delimiter", "\t")){};++    const std::string delimiter;++    void writeNextTupleCSV(std::ostream& destination, const RamDomain* tuple) {+        writeNextTupleElement(destination, typeAttributes.at(0), tuple[0]);++        for (size_t col = 1; col < arity; ++col) {+            destination << delimiter;+            writeNextTupleElement(destination, typeAttributes.at(col), tuple[col]);+        }++        destination << "\n";+    }++    void writeNextTupleElement(std::ostream& destination, const std::string& type, RamDomain value) {+        switch (type[0]) {+            case 's': destination << symbolTable.unsafeResolve(value); break;+            case 'i': destination << value; break;+            case 'u': destination << ramBitCast<RamUnsigned>(value); break;+            case 'f': destination << ramBitCast<RamFloat>(value); break;+            case 'r': outputRecord(destination, value, type); break;+            default: fatal("unsupported type attribute: `%c`", type[0]);+        }+    }+};++class WriteFileCSV : public WriteStreamCSV {+public:+    WriteFileCSV(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : WriteStreamCSV(rwOperation, symbolTable, recordTable),+              file(rwOperation.at("filename"), std::ios::out | std::ios::binary) {+        if (getOr(rwOperation, "headers", "false") == "true") {+            file << rwOperation.at("attributeNames") << std::endl;+        }+    }++    ~WriteFileCSV() override = default;++protected:+    std::ofstream file;++    void writeNullary() override {+        file << "()\n";+    }++    void writeNextTuple(const RamDomain* tuple) override {+        writeNextTupleCSV(file, tuple);+    }+};++#ifdef USE_LIBZ+class WriteGZipFileCSV : public WriteStreamCSV {+public:+    WriteGZipFileCSV(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : WriteStreamCSV(rwOperation, symbolTable, recordTable),+              file(rwOperation.at("filename"), std::ios::out | std::ios::binary) {+        if (getOr(rwOperation, "headers", "false") == "true") {+            file << rwOperation.at("attributeNames") << std::endl;+        }+    }++    ~WriteGZipFileCSV() override = default;++protected:+    void writeNullary() override {+        file << "()\n";+    }++    void writeNextTuple(const RamDomain* tuple) override {+        writeNextTupleCSV(file, tuple);+    }++    gzfstream::ogzfstream file;+};+#endif++class WriteCoutCSV : public WriteStreamCSV {+public:+    WriteCoutCSV(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : WriteStreamCSV(rwOperation, symbolTable, recordTable) {+        std::cout << "---------------\n" << rwOperation.at("name");+        if (getOr(rwOperation, "headers", "false") == "true") {+            std::cout << "\n" << rwOperation.at("attributeNames");+        }+        std::cout << "\n===============\n";+    }++    ~WriteCoutCSV() override {+        std::cout << "===============\n";+    }++protected:+    void writeNullary() override {+        std::cout << "()\n";+    }++    void writeNextTuple(const RamDomain* tuple) override {+        writeNextTupleCSV(std::cout, tuple);+    }+};++class WriteCoutPrintSize : public WriteStream {+public:+    explicit WriteCoutPrintSize(const std::map<std::string, std::string>& rwOperation)+            : WriteStream(rwOperation, {}, {}), lease(souffle::getOutputLock().acquire()) {+        std::cout << rwOperation.at("name") << "\t";+    }++    ~WriteCoutPrintSize() override = default;++protected:+    void writeNullary() override {+        fatal("attempting to iterate over a print size operation");+    }++    void writeNextTuple(const RamDomain* /* tuple */) override {+        fatal("attempting to iterate over a print size operation");+    }++    void writeSize(std::size_t size) override {+        std::cout << size << "\n";+    }++    Lock::Lease lease;+};++class WriteFileCSVFactory : public WriteStreamFactory {+public:+    std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable& symbolTable, const RecordTable& recordTable) override {+#ifdef USE_LIBZ+        if (contains(rwOperation, "compress")) {+            return std::make_unique<WriteGZipFileCSV>(rwOperation, symbolTable, recordTable);+        }+#endif+        return std::make_unique<WriteFileCSV>(rwOperation, symbolTable, recordTable);+    }+    const std::string& getName() const override {+        static const std::string name = "file";+        return name;+    }+    ~WriteFileCSVFactory() override = default;+};++class WriteCoutCSVFactory : public WriteStreamFactory {+public:+    std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable& symbolTable, const RecordTable& recordTable) override {+        return std::make_unique<WriteCoutCSV>(rwOperation, symbolTable, recordTable);+    }++    const std::string& getName() const override {+        static const std::string name = "stdout";+        return name;+    }+    ~WriteCoutCSVFactory() override = default;+};++class WriteCoutPrintSizeFactory : public WriteStreamFactory {+public:+    std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable&, const RecordTable&) override {+        return std::make_unique<WriteCoutPrintSize>(rwOperation);+    }+    const std::string& getName() const override {+        static const std::string name = "stdoutprintsize";+        return name;+    }+    ~WriteCoutPrintSizeFactory() override = default;+};++} /* namespace souffle */
+ cbits/souffle/WriteStreamSQLite.h view
@@ -0,0 +1,278 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file WriteStreamSQLite.h+ *+ ***********************************************************************/++#pragma once++#include "RamTypes.h"+#include "SymbolTable.h"+#include "WriteStream.h"+#include <cassert>+#include <cstddef>+#include <cstdint>+#include <map>+#include <memory>+#include <sstream>+#include <stdexcept>+#include <string>+#include <unordered_map>+#include <vector>+#include <sqlite3.h>++namespace souffle {++class RecordTable;++class WriteStreamSQLite : public WriteStream {+public:+    WriteStreamSQLite(const std::map<std::string, std::string>& rwOperation, const SymbolTable& symbolTable,+            const RecordTable& recordTable)+            : WriteStream(rwOperation, symbolTable, recordTable), dbFilename(rwOperation.at("filename")),+              relationName(rwOperation.at("name")) {+        openDB();+        createTables();+        prepareStatements();+        //        executeSQL("BEGIN TRANSACTION", db);+    }++    ~WriteStreamSQLite() override {+        sqlite3_finalize(insertStatement);+        sqlite3_finalize(symbolInsertStatement);+        sqlite3_finalize(symbolSelectStatement);+        sqlite3_close(db);+    }++protected:+    void writeNullary() override {}++    void writeNextTuple(const RamDomain* tuple) override {+        for (size_t i = 0; i < arity; i++) {+            RamDomain value = 0;  // Silence warning++            switch (typeAttributes.at(i)[0]) {+                case 's': value = getSymbolTableID(tuple[i]); break;+                default: value = tuple[i]; break;+            }++#if RAM_DOMAIN_SIZE == 64+            if (sqlite3_bind_int64(insertStatement, i + 1, value) != SQLITE_OK) {+#else+            if (sqlite3_bind_int(insertStatement, i + 1, value) != SQLITE_OK) {+#endif+                throwError("SQLite error in sqlite3_bind_text: ");+            }+        }+        if (sqlite3_step(insertStatement) != SQLITE_DONE) {+            throwError("SQLite error in sqlite3_step: ");+        }+        sqlite3_clear_bindings(insertStatement);+        sqlite3_reset(insertStatement);+    }++private:+    void executeSQL(const std::string& sql, sqlite3* db) {+        assert(db && "Database connection is closed");++        char* errorMessage = nullptr;+        /* Execute SQL statement */+        int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errorMessage);+        if (rc != SQLITE_OK) {+            std::stringstream error;+            error << "SQLite error in sqlite3_exec: " << sqlite3_errmsg(db) << "\n";+            error << "SQL error: " << errorMessage << "\n";+            error << "SQL: " << sql << "\n";+            sqlite3_free(errorMessage);+            throw std::invalid_argument(error.str());+        }+    }++    void throwError(const std::string& message) {+        std::stringstream error;+        error << message << sqlite3_errmsg(db) << "\n";+        throw std::invalid_argument(error.str());+    }++    uint64_t getSymbolTableIDFromDB(int index) {+        if (sqlite3_bind_text(symbolSelectStatement, 1, symbolTable.unsafeResolve(index).c_str(), -1,+                    SQLITE_TRANSIENT) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_bind_text: ");+        }+        if (sqlite3_step(symbolSelectStatement) != SQLITE_ROW) {+            throwError("SQLite error in sqlite3_step: ");+        }+        uint64_t rowid = sqlite3_column_int64(symbolSelectStatement, 0);+        sqlite3_clear_bindings(symbolSelectStatement);+        sqlite3_reset(symbolSelectStatement);+        return rowid;+    }+    uint64_t getSymbolTableID(int index) {+        if (dbSymbolTable.count(index) != 0) {+            return dbSymbolTable[index];+        }++        if (sqlite3_bind_text(symbolInsertStatement, 1, symbolTable.unsafeResolve(index).c_str(), -1,+                    SQLITE_TRANSIENT) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_bind_text: ");+        }+        // Either the insert succeeds and we have a new row id or it already exists and a select is needed.+        uint64_t rowid;+        if (sqlite3_step(symbolInsertStatement) != SQLITE_DONE) {+            // The symbol already exists so select it.+            rowid = getSymbolTableIDFromDB(index);+        } else {+            rowid = sqlite3_last_insert_rowid(db);+        }+        sqlite3_clear_bindings(symbolInsertStatement);+        sqlite3_reset(symbolInsertStatement);++        dbSymbolTable[index] = rowid;+        return rowid;+    }++    void openDB() {+        if (sqlite3_open(dbFilename.c_str(), &db) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_open");+        }+        sqlite3_extended_result_codes(db, 1);+        executeSQL("PRAGMA synchronous = OFF", db);+        executeSQL("PRAGMA journal_mode = MEMORY", db);+    }++    void prepareStatements() {+        prepareInsertStatement();+        prepareSymbolInsertStatement();+        prepareSymbolSelectStatement();+    }+    void prepareSymbolInsertStatement() {+        std::stringstream insertSQL;+        insertSQL << "INSERT INTO " << symbolTableName;+        insertSQL << " VALUES(null,@V0);";+        const char* tail = nullptr;+        if (sqlite3_prepare_v2(db, insertSQL.str().c_str(), -1, &symbolInsertStatement, &tail) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_prepare_v2: ");+        }+    }++    void prepareSymbolSelectStatement() {+        std::stringstream selectSQL;+        selectSQL << "SELECT id FROM " << symbolTableName;+        selectSQL << " WHERE symbol = @V0;";+        const char* tail = nullptr;+        if (sqlite3_prepare_v2(db, selectSQL.str().c_str(), -1, &symbolSelectStatement, &tail) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_prepare_v2: ");+        }+    }++    void prepareInsertStatement() {+        std::stringstream insertSQL;+        insertSQL << "INSERT INTO '_" << relationName << "' VALUES ";+        insertSQL << "(@V0";+        for (unsigned int i = 1; i < arity; i++) {+            insertSQL << ",@V" << i;+        }+        insertSQL << ");";+        const char* tail = nullptr;+        if (sqlite3_prepare_v2(db, insertSQL.str().c_str(), -1, &insertStatement, &tail) != SQLITE_OK) {+            throwError("SQLite error in sqlite3_prepare_v2: ");+        }+    }++    void createTables() {+        createRelationTable();+        createRelationView();+        createSymbolTable();+    }++    void createRelationTable() {+        std::stringstream createTableText;+        createTableText << "CREATE TABLE IF NOT EXISTS '_" << relationName << "' (";+        if (arity > 0) {+            createTableText << "'0' INTEGER";+            for (unsigned int i = 1; i < arity; i++) {+                createTableText << ",'" << std::to_string(i) << "' ";+                createTableText << "INTEGER";+            }+        }+        createTableText << ");";+        executeSQL(createTableText.str(), db);+        executeSQL("DELETE FROM '_" + relationName + "';", db);+    }++    void createRelationView() {+        // Create view with symbol strings resolved+        std::stringstream createViewText;+        createViewText << "CREATE VIEW IF NOT EXISTS '" << relationName << "' AS ";+        std::stringstream projectionClause;+        std::stringstream fromClause;+        fromClause << "'_" << relationName << "'";+        std::stringstream whereClause;+        bool firstWhere = true;+        for (unsigned int i = 0; i < arity; i++) {+            std::string columnName = std::to_string(i);+            if (i != 0) {+                projectionClause << ",";+            }+            if (typeAttributes.at(i)[0] == 's') {+                projectionClause << "'_symtab_" << columnName << "'.symbol AS '" << columnName << "'";+                fromClause << ",'" << symbolTableName << "' AS '_symtab_" << columnName << "'";+                if (!firstWhere) {+                    whereClause << " AND ";+                } else {+                    firstWhere = false;+                }+                whereClause << "'_" << relationName << "'.'" << columnName << "' = "+                            << "'_symtab_" << columnName << "'.id";+            } else {+                projectionClause << "'_" << relationName << "'.'" << columnName << "'";+            }+        }+        createViewText << "SELECT " << projectionClause.str() << " FROM " << fromClause.str();+        if (!firstWhere) {+            createViewText << " WHERE " << whereClause.str();+        }+        createViewText << ";";+        executeSQL(createViewText.str(), db);+    }+    void createSymbolTable() {+        std::stringstream createTableText;+        createTableText << "CREATE TABLE IF NOT EXISTS '" << symbolTableName << "' ";+        createTableText << "(id INTEGER PRIMARY KEY, symbol TEXT UNIQUE);";+        executeSQL(createTableText.str(), db);+    }++    const std::string& dbFilename;+    const std::string& relationName;+    const std::string symbolTableName = "__SymbolTable";++    std::unordered_map<uint64_t, uint64_t> dbSymbolTable;+    sqlite3_stmt* insertStatement = nullptr;+    sqlite3_stmt* symbolInsertStatement = nullptr;+    sqlite3_stmt* symbolSelectStatement = nullptr;+    sqlite3* db = nullptr;+};++class WriteSQLiteFactory : public WriteStreamFactory {+public:+    std::unique_ptr<WriteStream> getWriter(const std::map<std::string, std::string>& rwOperation,+            const SymbolTable& symbolTable, const RecordTable& recordTable) override {+        return std::make_unique<WriteStreamSQLite>(rwOperation, symbolTable, recordTable);+    }++    const std::string& getName() const override {+        static const std::string name = "sqlite";+        return name;+    }+    ~WriteSQLiteFactory() override = default;+};++} /* namespace souffle */
+ cbits/souffle/gzfstream.h view
@@ -0,0 +1,235 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2014, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file gzfstream.h+ * A simple zlib wrapper to provide gzip file streams.+ *+ ***********************************************************************/++#pragma once++#include <cstdio>+#include <cstring>+#include <iostream>+#include <string>+#include <zlib.h>++namespace souffle {++namespace gzfstream {++namespace internal {++class gzfstreambuf : public std::streambuf {+public:+    gzfstreambuf() {+        setp(buffer, buffer + (bufferSize - 1));+        setg(buffer + reserveSize, buffer + reserveSize, buffer + reserveSize);+    }++    gzfstreambuf(const gzfstreambuf&) = delete;++    gzfstreambuf(gzfstreambuf&& old) = default;++    gzfstreambuf* open(const std::string& filename, std::ios_base::openmode mode) {+        if (is_open()) {+            return nullptr;+        }+        if ((mode ^ std::ios::in ^ std::ios::out) == 0) {+            return nullptr;+        }++        this->mode = mode;+        std::string gzmode((mode & std::ios::in) != 0 ? "rb" : "wb");+        fileHandle = gzopen(filename.c_str(), gzmode.c_str());++        if (fileHandle == nullptr) {+            return nullptr;+        }+        isOpen = true;++        return this;+    }++    gzfstreambuf* close() {+        if (is_open()) {+            sync();+            isOpen = false;+            if (gzclose(fileHandle) == Z_OK) {+                return this;+            }+        }+        return nullptr;+    }++    bool is_open() const {+        return isOpen;+    }++    ~gzfstreambuf() override {+        try {+            close();+        } catch (...) {+            // Don't throw exceptions.+        }+    }++protected:+    int_type overflow(int c = EOF) override {+        if (((mode & std::ios::out) == 0) || !isOpen) {+            return EOF;+        }++        if (c != EOF) {+            *pptr() = c;+            pbump(1);+        }+        int toWrite = pptr() - pbase();+        if (gzwrite(fileHandle, pbase(), toWrite) != toWrite) {+            return EOF;+        }+        pbump(-toWrite);++        return c;+    }++    int_type underflow() override {+        if (((mode & std::ios::in) == 0) || !isOpen) {+            return EOF;+        }+        if ((gptr() != nullptr) && (gptr() < egptr())) {+            return traits_type::to_int_type(*gptr());+        }++        unsigned charsPutBack = gptr() - eback();+        if (charsPutBack > reserveSize) {+            charsPutBack = reserveSize;+        }+        memcpy(buffer + reserveSize - charsPutBack, gptr() - charsPutBack, charsPutBack);++        int charsRead = gzread(fileHandle, buffer + reserveSize, bufferSize - reserveSize);+        if (charsRead <= 0) {+            return EOF;+        }++        setg(buffer + reserveSize - charsPutBack, buffer + reserveSize, buffer + reserveSize + charsRead);++        return traits_type::to_int_type(*gptr());+    }++    int sync() override {+        if ((pptr() != nullptr) && pptr() > pbase()) {+            int toWrite = pptr() - pbase();+            if (gzwrite(fileHandle, pbase(), toWrite) != toWrite) {+                return -1;+            }+            pbump(-toWrite);+        }+        return 0;+    }++private:+    static constexpr unsigned int bufferSize = 65536;+    static constexpr unsigned int reserveSize = 16;++    char buffer[bufferSize] = {};+    gzFile fileHandle = {};+    bool isOpen = false;+    std::ios_base::openmode mode = std::ios_base::in;+};++class gzfstream : virtual public std::ios {+public:+    gzfstream() {+        init(&buf);+    }++    gzfstream(const std::string& filename, std::ios_base::openmode mode) {+        init(&buf);+        open(filename, mode);+    }++    gzfstream(const gzfstream&) = delete;++    gzfstream(gzfstream&&) = delete;++    ~gzfstream() override = default;++    void open(const std::string& filename, std::ios_base::openmode mode) {+        if (buf.open(filename, mode) == nullptr) {+            clear(rdstate() | std::ios::badbit);+        }+    }++    bool is_open() {+        return buf.is_open();+    }++    void close() {+        if (buf.is_open()) {+            if (buf.close() == nullptr) {+                clear(rdstate() | std::ios::badbit);+            }+        }+    }++    gzfstreambuf* rdbuf() const {+        return &buf;+    }++protected:+    mutable gzfstreambuf buf;+};++}  // namespace internal++class igzfstream : public internal::gzfstream, public std::istream {+public:+    igzfstream() : internal::gzfstream(), std::istream(&buf) {}++    explicit igzfstream(const std::string& filename, std::ios_base::openmode mode = std::ios::in)+            : internal::gzfstream(filename, mode), std::istream(&buf) {}++    igzfstream(const igzfstream&) = delete;++    igzfstream(igzfstream&&) = delete;++    internal::gzfstreambuf* rdbuf() const {+        return internal::gzfstream::rdbuf();+    }++    void open(const std::string& filename, std::ios_base::openmode mode = std::ios::in) {+        internal::gzfstream::open(filename, mode);+    }+};++class ogzfstream : public internal::gzfstream, public std::ostream {+public:+    ogzfstream() : std::ostream(&buf) {}++    explicit ogzfstream(const std::string& filename, std::ios_base::openmode mode = std::ios::out)+            : internal::gzfstream(filename, mode), std::ostream(&buf) {}++    ogzfstream(const ogzfstream&) = delete;++    ogzfstream(ogzfstream&&) = delete;++    internal::gzfstreambuf* rdbuf() const {+        return internal::gzfstream::rdbuf();+    }++    void open(const std::string& filename, std::ios_base::openmode mode = std::ios::out) {+        internal::gzfstream::open(filename, mode);+    }+};++} /* namespace gzfstream */++} /* namespace souffle */
+ cbits/souffle/json11.h view
@@ -0,0 +1,1107 @@+/* json11+ *+ * json11 is a tiny JSON library for C++11, providing JSON parsing and serialization.+ *+ * The core object provided by the library is json11::Json. A Json object represents any JSON+ * value: null, bool, number (int or double), string (std::string), array (std::vector), or+ * object (std::map).+ *+ * Json objects act like values: they can be assigned, copied, moved, compared for equality or+ * order, etc. There are also helper methods Json::dump, to serialize a Json to a string, and+ * Json::parse (static) to parse a std::string as a Json object.+ *+ * Internally, the various types of Json object are represented by the JsonValue class+ * hierarchy.+ *+ * A note on numbers - JSON specifies the syntax of number formatting but not its semantics,+ * so some JSON implementations distinguish between integers and floating-point numbers, while+ * some don't. In json11, we choose the latter. Because some JSON implementations (namely+ * Javascript itself) treat all numbers as the same type, distinguishing the two leads+ * to JSON that will be *silently* changed by a round-trip through those implementations.+ * Dangerous! To avoid that risk, json11 stores all numbers as double internally, but also+ * provides integer helpers.+ *+ * Fortunately, double-precision IEEE754 ('double') can precisely store any integer in the+ * range +/-2^53, which includes every 'int' on most systems. (Timestamps often use int64+ * or long long to avoid the Y2038K problem; a double storing microseconds since some epoch+ * will be exact for +/- 275 years.)+ */++/* Copyright (c) 2013 Dropbox, Inc.+ *+ * Permission is hereby granted, free of charge, to any person obtaining a copy+ * of this software and associated documentation files (the "Software"), to deal+ * in the Software without restriction, including without limitation the rights+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+ * copies of the Software, and to permit persons to whom the Software is+ * furnished to do so, subject to the following conditions:+ *+ * The above copyright notice and this permission notice shall be included in+ * all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN+ * THE SOFTWARE.+ */++#pragma once++#include <cassert>+#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstdlib>+#include <initializer_list>+#include <iosfwd>+#include <limits>+#include <map>+#include <memory>+#include <string>+#include <type_traits>+#include <utility>+#include <vector>++#ifdef _MSC_VER+#if _MSC_VER <= 1800  // VS 2013+#ifndef noexcept+#define noexcept throw()+#endif++#ifndef snprintf+#define snprintf _snprintf_s+#endif+#endif+#endif++namespace json11 {++enum JsonParse { STANDARD, COMMENTS };++class JsonValue;++class Json final {+public:+    // Types+    enum Type { NUL, NUMBER, BOOL, STRING, ARRAY, OBJECT };++    // Array and object typedefs+    using array = std::vector<Json>;+    using object = std::map<std::string, Json>;++    // Constructors for the various types of JSON value.+    Json() noexcept;                 // NUL+    Json(std::nullptr_t) noexcept;   // NUL+    Json(double value);              // NUMBER+    Json(long long value);           // NUMBER+    Json(bool value);                // BOOL+    Json(const std::string& value);  // STRING+    Json(std::string&& value);       // STRING+    Json(const char* value);         // STRING+    Json(const array& values);       // ARRAY+    Json(array&& values);            // ARRAY+    Json(const object& values);      // OBJECT+    Json(object&& values);           // OBJECT++    // Implicit constructor: anything with a to_json() function.+    template <class T, class = decltype(&T::to_json)>+    Json(const T& t) : Json(t.to_json()) {}++    // Implicit constructor: map-like objects (std::map, std::unordered_map, etc)+    template <class M,+            typename std::enable_if<+                    std::is_constructible<std::string, decltype(std::declval<M>().begin()->first)>::value &&+                            std::is_constructible<Json, decltype(std::declval<M>().begin()->second)>::value,+                    int>::type = 0>+    Json(const M& m) : Json(object(m.begin(), m.end())) {}++    // Implicit constructor: vector-like objects (std::list, std::vector, std::set, etc)+    template <class V,+            typename std::enable_if<std::is_constructible<Json, decltype(*std::declval<V>().begin())>::value,+                    int>::type = 0>+    Json(const V& v) : Json(array(v.begin(), v.end())) {}++    // This prevents Json(some_pointer) from accidentally producing a bool. Use+    // Json(bool(some_pointer)) if that behavior is desired.+    Json(void*) = delete;++    // Accessors+    Type type() const;++    bool is_null() const {+        return type() == NUL;+    }+    bool is_number() const {+        return type() == NUMBER;+    }+    bool is_bool() const {+        return type() == BOOL;+    }+    bool is_string() const {+        return type() == STRING;+    }+    bool is_array() const {+        return type() == ARRAY;+    }+    bool is_object() const {+        return type() == OBJECT;+    }++    // Return the enclosed value if this is a number, 0 otherwise. Note that json11 does not+    // distinguish between integer and non-integer numbers - number_value() and int_value()+    // can both be applied to a NUMBER-typed object.+    double number_value() const;+    int int_value() const;+    long long long_value() const;++    // Return the enclosed value if this is a boolean, false otherwise.+    bool bool_value() const;+    // Return the enclosed string if this is a string, "" otherwise.+    const std::string& string_value() const;+    // Return the enclosed std::vector if this is an array, or an empty vector otherwise.+    const array& array_items() const;+    // Return the enclosed std::map if this is an object, or an empty map otherwise.+    const object& object_items() const;++    // Return a reference to arr[i] if this is an array, Json() otherwise.+    const Json& operator[](size_t i) const;+    // Return a reference to obj[key] if this is an object, Json() otherwise.+    const Json& operator[](const std::string& key) const;++    // Serialize.+    void dump(std::string& out) const;+    std::string dump() const {+        std::string out;+        dump(out);+        return out;+    }++    // Parse. If parse fails, return Json() and assign an error message to err.+    static Json parse(const std::string& in, std::string& err, JsonParse strategy = JsonParse::STANDARD);++    static Json parse(const char* in, std::string& err, JsonParse strategy = JsonParse::STANDARD) {+        if (in == nullptr) {+            err = "null input";+            return nullptr;+        }+        return parse(std::string(in), err, strategy);+    }+    // Parse multiple objects, concatenated or separated by whitespace+    static std::vector<Json> parse_multi(const std::string& in, std::string::size_type& parser_stop_pos,+            std::string& err, JsonParse strategy = JsonParse::STANDARD);++    static inline std::vector<Json> parse_multi(+            const std::string& in, std::string& err, JsonParse strategy = JsonParse::STANDARD) {+        std::string::size_type parser_stop_pos;+        return parse_multi(in, parser_stop_pos, err, strategy);+    }++    bool operator==(const Json& rhs) const;+    bool operator<(const Json& rhs) const;+    bool operator!=(const Json& rhs) const {+        return !(*this == rhs);+    }+    bool operator<=(const Json& rhs) const {+        return !(rhs < *this);+    }+    bool operator>(const Json& rhs) const {+        return (rhs < *this);+    }+    bool operator>=(const Json& rhs) const {+        return !(*this < rhs);+    }++    /* has_shape(types, err)+     *+     * Return true if this is a JSON object and, for each item in types, has a field of+     * the given type. If not, return false and set err to a descriptive message.+     */+    using shape = std::initializer_list<std::pair<std::string, Type>>;+    bool has_shape(const shape& types, std::string& err) const {+        if (!is_object()) {+            err = "expected JSON object, got " + dump();+            return false;+        }++        for (auto& item : types) {+            if ((*this)[item.first].type() != item.second) {+                err = "bad type for " + item.first + " in " + dump();+                return false;+            }+        }++        return true;+    }++private:+    std::shared_ptr<JsonValue> m_ptr;+};++// Internal class hierarchy - JsonValue objects are not exposed to users of this API.+class JsonValue {+protected:+    friend class Json;+    friend class JsonInt;+    friend class JsonDouble;+    virtual Json::Type type() const = 0;+    virtual bool equals(const JsonValue* other) const = 0;+    virtual bool less(const JsonValue* other) const = 0;+    virtual void dump(std::string& out) const = 0;+    virtual double number_value() const;+    virtual int int_value() const;+    virtual long long long_value() const;+    virtual bool bool_value() const;+    virtual const std::string& string_value() const;+    virtual const Json::array& array_items() const;+    virtual const Json& operator[](size_t i) const;+    virtual const Json::object& object_items() const;+    virtual const Json& operator[](const std::string& key) const;+    virtual ~JsonValue() = default;+};++static const int max_depth = 200;++/* Helper for representing null - just a do-nothing struct, plus comparison+ * operators so the helpers in JsonValue work. We can't use nullptr_t because+ * it may not be orderable.+ */+struct NullStruct {+    bool operator==(NullStruct) const {+        return true;+    }+    bool operator<(NullStruct) const {+        return false;+    }+};++/* * * * * * * * * * * * * * * * * * * *+ * Serialization+ */++static void dump(NullStruct, std::string& out) {+    out += "null";+}++static void dump(double value, std::string& out) {+    if (std::isfinite(value)) {+        char buf[32];+        snprintf(buf, sizeof buf, "%.17g", value);+        out += buf;+    } else {+        out += "null";+    }+}++static void dump(long long value, std::string& out) {+    char buf[32];+    snprintf(buf, sizeof buf, "%lld", value);+    out += buf;+}++static void dump(bool value, std::string& out) {+    out += value ? "true" : "false";+}++static void dump(const std::string& value, std::string& out) {+    out += '"';+    for (size_t i = 0; i < value.length(); i++) {+        const char ch = value[i];+        if (ch == '\\') {+            out += "\\\\";+        } else if (ch == '"') {+            out += "\\\"";+        } else if (ch == '\b') {+            out += "\\b";+        } else if (ch == '\f') {+            out += "\\f";+        } else if (ch == '\n') {+            out += "\\n";+        } else if (ch == '\r') {+            out += "\\r";+        } else if (ch == '\t') {+            out += "\\t";+        } else if (static_cast<uint8_t>(ch) <= 0x1f) {+            char buf[8];+            snprintf(buf, sizeof buf, "\\u%04x", ch);+            out += buf;+        } else if (static_cast<uint8_t>(ch) == 0xe2 && static_cast<uint8_t>(value[i + 1]) == 0x80 &&+                   static_cast<uint8_t>(value[i + 2]) == 0xa8) {+            out += "\\u2028";+            i += 2;+        } else if (static_cast<uint8_t>(ch) == 0xe2 && static_cast<uint8_t>(value[i + 1]) == 0x80 &&+                   static_cast<uint8_t>(value[i + 2]) == 0xa9) {+            out += "\\u2029";+            i += 2;+        } else {+            out += ch;+        }+    }+    out += '"';+}++static void dump(const Json::array& values, std::string& out) {+    bool first = true;+    out += "[";+    for (const auto& value : values) {+        if (!first) out += ", ";+        value.dump(out);+        first = false;+    }+    out += "]";+}++static void dump(const Json::object& values, std::string& out) {+    bool first = true;+    out += "{";+    for (const auto& kv : values) {+        if (!first) out += ", ";+        dump(kv.first, out);+        out += ": ";+        kv.second.dump(out);+        first = false;+    }+    out += "}";+}++inline void Json::dump(std::string& out) const {+    m_ptr->dump(out);+}++/* * * * * * * * * * * * * * * * * * * *+ * Value wrappers+ */++template <Json::Type tag, typename T>+class Value : public JsonValue {+protected:+    // Constructors+    explicit Value(T value) : m_value(std::move(value)) {}++    // Get type tag+    Json::Type type() const override {+        return tag;+    }++    // Comparisons+    bool equals(const JsonValue* other) const override {+        return m_value == static_cast<const Value<tag, T>*>(other)->m_value;+    }+    bool less(const JsonValue* other) const override {+        return m_value < static_cast<const Value<tag, T>*>(other)->m_value;+    }++    const T m_value;+    void dump(std::string& out) const override {+        json11::dump(m_value, out);+    }+};++class JsonDouble final : public Value<Json::NUMBER, double> {+    double number_value() const override {+        return m_value;+    }+    int int_value() const override {+        return static_cast<int>(m_value);+    }+    long long long_value() const override {+        return static_cast<long long>(m_value);+    }+    bool equals(const JsonValue* other) const override {+        return m_value == other->number_value();+    }+    bool less(const JsonValue* other) const override {+        return m_value < other->number_value();+    }++public:+    explicit JsonDouble(double value) : Value(value) {}+};++class JsonInt final : public Value<Json::NUMBER, long long> {+    double number_value() const override {+        return m_value;+    }+    int int_value() const override {+        return m_value;+    }+    long long long_value() const override {+        return static_cast<long long>(m_value);+    }+    bool equals(const JsonValue* other) const override {+        return m_value == other->number_value();+    }+    bool less(const JsonValue* other) const override {+        return m_value < other->number_value();+    }++public:+    explicit JsonInt(int value) : Value(value) {}+};++class JsonBoolean final : public Value<Json::BOOL, bool> {+    bool bool_value() const override {+        return m_value;+    }++public:+    explicit JsonBoolean(bool value) : Value(value) {}+};++class JsonString final : public Value<Json::STRING, std::string> {+    const std::string& string_value() const override {+        return m_value;+    }++public:+    explicit JsonString(const std::string& value) : Value(value) {}+    explicit JsonString(std::string&& value) : Value(std::move(value)) {}+};++class JsonArray final : public Value<Json::ARRAY, Json::array> {+    const Json::array& array_items() const override {+        return m_value;+    }+    const Json& operator[](size_t i) const override;++public:+    explicit JsonArray(const Json::array& value) : Value(value) {}+    explicit JsonArray(Json::array&& value) : Value(std::move(value)) {}+};++class JsonObject final : public Value<Json::OBJECT, Json::object> {+    const Json::object& object_items() const override {+        return m_value;+    }+    const Json& operator[](const std::string& key) const override;++public:+    explicit JsonObject(const Json::object& value) : Value(value) {}+    explicit JsonObject(Json::object&& value) : Value(std::move(value)) {}+};++class JsonNull final : public Value<Json::NUL, NullStruct> {+public:+    JsonNull() : Value({}) {}+};++/* * * * * * * * * * * * * * * * * * * *+ * Static globals - static-init-safe+ */+struct Statics {+    const std::shared_ptr<JsonValue> null = std::make_shared<JsonNull>();+    const std::shared_ptr<JsonValue> t = std::make_shared<JsonBoolean>(true);+    const std::shared_ptr<JsonValue> f = std::make_shared<JsonBoolean>(false);+    const std::string empty_string{};+    const std::vector<Json> empty_vector{};+    const std::map<std::string, Json> empty_map{};+    Statics() = default;+};++static const Statics& statics() {+    static const Statics s{};+    return s;+}++static const Json& static_null() {+    // This has to be separate, not in Statics, because Json() accesses statics().null.+    static const Json json_null;+    return json_null;+}++/* * * * * * * * * * * * * * * * * * * *+ * Constructors+ */++inline Json::Json() noexcept : m_ptr(statics().null) {}+inline Json::Json(std::nullptr_t) noexcept : m_ptr(statics().null) {}+inline Json::Json(double value) : m_ptr(std::make_shared<JsonDouble>(value)) {}+inline Json::Json(long long value) : m_ptr(std::make_shared<JsonInt>(value)) {}+inline Json::Json(bool value) : m_ptr(value ? statics().t : statics().f) {}+inline Json::Json(const std::string& value) : m_ptr(std::make_shared<JsonString>(value)) {}+inline Json::Json(std::string&& value) : m_ptr(std::make_shared<JsonString>(std::move(value))) {}+inline Json::Json(const char* value) : m_ptr(std::make_shared<JsonString>(value)) {}+inline Json::Json(const Json::array& values) : m_ptr(std::make_shared<JsonArray>(values)) {}+inline Json::Json(Json::array&& values) : m_ptr(std::make_shared<JsonArray>(std::move(values))) {}+inline Json::Json(const Json::object& values) : m_ptr(std::make_shared<JsonObject>(values)) {}+inline Json::Json(Json::object&& values) : m_ptr(std::make_shared<JsonObject>(std::move(values))) {}++/* * * * * * * * * * * * * * * * * * * *+ * Accessors+ */++inline Json::Type Json::type() const {+    return m_ptr->type();+}+inline double Json::number_value() const {+    return m_ptr->number_value();+}+inline int Json::int_value() const {+    return m_ptr->int_value();+}+inline long long Json::long_value() const {+    return m_ptr->long_value();+}+inline bool Json::bool_value() const {+    return m_ptr->bool_value();+}+inline const std::string& Json::string_value() const {+    return m_ptr->string_value();+}+inline const std::vector<Json>& Json::array_items() const {+    return m_ptr->array_items();+}+inline const std::map<std::string, Json>& Json::object_items() const {+    return m_ptr->object_items();+}+inline const Json& Json::operator[](size_t i) const {+    return (*m_ptr)[i];+}+inline const Json& Json::operator[](const std::string& key) const {+    return (*m_ptr)[key];+}++inline double JsonValue::number_value() const {+    return 0;+}+inline int JsonValue::int_value() const {+    return 0;+}+inline long long JsonValue::long_value() const {+    return 0;+}+inline bool JsonValue::bool_value() const {+    return false;+}+inline const std::string& JsonValue::string_value() const {+    return statics().empty_string;+}+inline const std::vector<Json>& JsonValue::array_items() const {+    return statics().empty_vector;+}+inline const std::map<std::string, Json>& JsonValue::object_items() const {+    return statics().empty_map;+}+inline const Json& JsonValue::operator[](size_t) const {+    return static_null();+}+inline const Json& JsonValue::operator[](const std::string&) const {+    return static_null();+}++inline const Json& JsonObject::operator[](const std::string& key) const {+    auto iter = m_value.find(key);+    return (iter == m_value.end()) ? static_null() : iter->second;+}+inline const Json& JsonArray::operator[](size_t i) const {+    if (i >= m_value.size()) {+        return static_null();+    }+    return m_value[i];+}++/* * * * * * * * * * * * * * * * * * * *+ * Comparison+ */++inline bool Json::operator==(const Json& other) const {+    if (m_ptr == other.m_ptr) {+        return true;+    }+    if (m_ptr->type() != other.m_ptr->type()) {+        return false;+    }++    return m_ptr->equals(other.m_ptr.get());+}++inline bool Json::operator<(const Json& other) const {+    if (m_ptr == other.m_ptr) {+        return false;+    }+    if (m_ptr->type() != other.m_ptr->type()) {+        return m_ptr->type() < other.m_ptr->type();+    }++    return m_ptr->less(other.m_ptr.get());+}++/* * * * * * * * * * * * * * * * * * * *+ * Parsing+ */++/* esc(c)+ *+ * Format char c suitable for printing in an error message.+ */+static inline std::string esc(char c) {+    char buf[12];+    if (static_cast<uint8_t>(c) >= 0x20 && static_cast<uint8_t>(c) <= 0x7f) {+        snprintf(buf, sizeof buf, "'%c' (%d)", c, c);+    } else {+        snprintf(buf, sizeof buf, "(%d)", c);+    }+    return std::string(buf);+}++static inline bool in_range(long x, long lower, long upper) {+    return (x >= lower && x <= upper);+}++namespace {+/* JsonParser+ *+ * Object that tracks all state of an in-progress parse.+ */+struct JsonParser final {+    /* State+     */+    const std::string& str;+    size_t i;+    std::string& err;+    bool failed;+    const JsonParse strategy;++    /* fail(msg, err_ret = Json())+     *+     * Mark this parse as failed.+     */+    Json fail(std::string&& msg) {+        return fail(std::move(msg), Json());+    }++    template <typename T>+    T fail(std::string&& msg, T err_ret) {+        if (!failed) {+            err = std::move(msg);+        }+        failed = true;+        return err_ret;+    }++    /* consume_whitespace()+     *+     * Advance until the current character is non-whitespace.+     */+    void consume_whitespace() {+        while (str[i] == ' ' || str[i] == '\r' || str[i] == '\n' || str[i] == '\t') {+            i++;+        }+    }++    /* consume_comment()+     *+     * Advance comments (c-style inline and multiline).+     */+    bool consume_comment() {+        bool comment_found = false;+        if (str[i] == '/') {+            i++;+            if (i == str.size()) {+                return fail("unexpected end of input after start of comment", false);+            }+            if (str[i] == '/') {  // inline comment+                i++;+                // advance until next line, or end of input+                while (i < str.size() && str[i] != '\n') {+                    i++;+                }+                comment_found = true;+            } else if (str[i] == '*') {  // multiline comment+                i++;+                if (i > str.size() - 2)+                    return fail("unexpected end of input inside multi-line comment", false);+                // advance until closing tokens+                while (!(str[i] == '*' && str[i + 1] == '/')) {+                    i++;+                    if (i > str.size() - 2)+                        return fail("unexpected end of input inside multi-line comment", false);+                }+                i += 2;+                comment_found = true;+            } else {+                return fail("malformed comment", false);+            }+        }+        return comment_found;+    }++    /* consume_garbage()+     *+     * Advance until the current character is non-whitespace and non-comment.+     */+    void consume_garbage() {+        consume_whitespace();+        if (strategy == JsonParse::COMMENTS) {+            bool comment_found = false;+            do {+                comment_found = consume_comment();+                if (failed) {+                    return;+                }+                consume_whitespace();+            } while (comment_found);+        }+    }++    /* get_next_token()+     *+     * Return the next non-whitespace character. If the end of the input is reached,+     * flag an error and return 0.+     */+    char get_next_token() {+        consume_garbage();+        if (failed) {+            return static_cast<char>(0);+        }+        if (i == str.size()) {+            return fail("unexpected end of input", static_cast<char>(0));+        }++        return str[i++];+    }++    /* encode_utf8(pt, out)+     *+     * Encode pt as UTF-8 and add it to out.+     */+    void encode_utf8(long pt, std::string& out) {+        if (pt < 0) {+            return;+        }++        if (pt < 0x80) {+            out += static_cast<char>(pt);+        } else if (pt < 0x800) {+            out += static_cast<char>((pt >> 6) | 0xC0);+            out += static_cast<char>((pt & 0x3F) | 0x80);+        } else if (pt < 0x10000) {+            out += static_cast<char>((pt >> 12) | 0xE0);+            out += static_cast<char>(((pt >> 6) & 0x3F) | 0x80);+            out += static_cast<char>((pt & 0x3F) | 0x80);+        } else {+            out += static_cast<char>((pt >> 18) | 0xF0);+            out += static_cast<char>(((pt >> 12) & 0x3F) | 0x80);+            out += static_cast<char>(((pt >> 6) & 0x3F) | 0x80);+            out += static_cast<char>((pt & 0x3F) | 0x80);+        }+    }++    /* parse_string()+     *+     * Parse a std::string, starting at the current position.+     */+    std::string parse_string() {+        std::string out;+        long last_escaped_codepoint = -1;+        while (true) {+            if (i == str.size()) return fail("unexpected end of input in std::string", "");++            char ch = str[i++];++            if (ch == '"') {+                encode_utf8(last_escaped_codepoint, out);+                return out;+            }++            if (in_range(ch, 0, 0x1f)) {+                return fail("unescaped " + esc(ch) + " in std::string", "");+            }++            // The usual case: non-escaped characters+            if (ch != '\\') {+                encode_utf8(last_escaped_codepoint, out);+                last_escaped_codepoint = -1;+                out += ch;+                continue;+            }++            // Handle escapes+            if (i == str.size()) {+                return fail("unexpected end of input in std::string", "");+            }++            ch = str[i++];++            if (ch == 'u') {+                // Extract 4-byte escape sequence+                std::string esc = str.substr(i, 4);+                // Explicitly check length of the substring. The following loop+                // relies on std::string returning the terminating NUL when+                // accessing str[length]. Checking here reduces brittleness.+                if (esc.length() < 4) {+                    return fail("bad \\u escape: " + esc, "");+                }+                for (size_t j = 0; j < 4; j++) {+                    if (!in_range(esc[j], 'a', 'f') && !in_range(esc[j], 'A', 'F') &&+                            !in_range(esc[j], '0', '9'))+                        return fail("bad \\u escape: " + esc, "");+                }++                long codepoint = strtol(esc.data(), nullptr, 16);++                // JSON specifies that characters outside the BMP shall be encoded as a pair+                // of 4-hex-digit \u escapes encoding their surrogate pair components. Check+                // whether we're in the middle of such a beast: the previous codepoint was an+                // escaped lead (high) surrogate, and this is a trail (low) surrogate.+                if (in_range(last_escaped_codepoint, 0xD800, 0xDBFF) && in_range(codepoint, 0xDC00, 0xDFFF)) {+                    // Reassemble the two surrogate pairs into one astral-plane character, per+                    // the UTF-16 algorithm.+                    encode_utf8((((last_escaped_codepoint - 0xD800) << 10) | (codepoint - 0xDC00)) + 0x10000,+                            out);+                    last_escaped_codepoint = -1;+                } else {+                    encode_utf8(last_escaped_codepoint, out);+                    last_escaped_codepoint = codepoint;+                }++                i += 4;+                continue;+            }++            encode_utf8(last_escaped_codepoint, out);+            last_escaped_codepoint = -1;++            if (ch == 'b') {+                out += '\b';+            } else if (ch == 'f') {+                out += '\f';+            } else if (ch == 'n') {+                out += '\n';+            } else if (ch == 'r') {+                out += '\r';+            } else if (ch == 't') {+                out += '\t';+            } else if (ch == '"' || ch == '\\' || ch == '/') {+                out += ch;+            } else {+                return fail("invalid escape character " + esc(ch), "");+            }+        }+    }++    /* parse_number()+     *+     * Parse a double.+     */+    Json parse_number() {+        size_t start_pos = i;++        if (str[i] == '-') {+            i++;+        }++        // Integer part+        if (str[i] == '0') {+            i++;+            if (in_range(str[i], '0', '9')) {+                return fail("leading 0s not permitted in numbers");+            }+        } else if (in_range(str[i], '1', '9')) {+            i++;+            while (in_range(str[i], '0', '9')) {+                i++;+            }+        } else {+            return fail("invalid " + esc(str[i]) + " in number");+        }++        if (str[i] != '.' && str[i] != 'e' && str[i] != 'E' &&+                (i - start_pos) <= static_cast<size_t>(std::numeric_limits<int>::digits10)) {+            return std::atoll(str.c_str() + start_pos);+        }++        // Decimal part+        if (str[i] == '.') {+            i++;+            if (!in_range(str[i], '0', '9')) {+                return fail("at least one digit required in fractional part");+            }++            while (in_range(str[i], '0', '9')) {+                i++;+            }+        }++        // Exponent part+        if (str[i] == 'e' || str[i] == 'E') {+            i++;++            if (str[i] == '+' || str[i] == '-') {+                i++;+            }++            if (!in_range(str[i], '0', '9')) {+                return fail("at least one digit required in exponent");+            }++            while (in_range(str[i], '0', '9')) {+                i++;+            }+        }++        return std::strtod(str.c_str() + start_pos, nullptr);+    }++    /* expect(str, res)+     *+     * Expect that 'str' starts at the character that was just read. If it does, advance+     * the input and return res. If not, flag an error.+     */+    Json expect(const std::string& expected, Json res) {+        assert(i != 0);+        i--;+        if (str.compare(i, expected.length(), expected) == 0) {+            i += expected.length();+            return res;+        } else {+            return fail("parse error: expected " + expected + ", got " + str.substr(i, expected.length()));+        }+    }++    /* parse_json()+     *+     * Parse a JSON object.+     */+    Json parse_json(int depth) {+        if (depth > max_depth) {+            return fail("exceeded maximum nesting depth");+        }++        char ch = get_next_token();+        if (failed) {+            return Json();+        }++        if (ch == '-' || (ch >= '0' && ch <= '9')) {+            i--;+            return parse_number();+        }++        if (ch == 't') {+            return expect("true", true);+        }++        if (ch == 'f') {+            return expect("false", false);+        }++        if (ch == 'n') {+            return expect("null", Json());+        }++        if (ch == '"') {+            return parse_string();+        }++        if (ch == '{') {+            std::map<std::string, Json> data;+            ch = get_next_token();+            if (ch == '}') {+                return data;+            }++            while (true) {+                if (ch != '"') return fail("expected '\"' in object, got " + esc(ch));++                std::string key = parse_string();+                if (failed) {+                    return Json();+                }++                ch = get_next_token();+                if (ch != ':') {+                    return fail("expected ':' in object, got " + esc(ch));+                }++                data[std::move(key)] = parse_json(depth + 1);+                if (failed) {+                    return Json();+                }++                ch = get_next_token();+                if (ch == '}') {+                    break;+                }+                if (ch != ',') {+                    return fail("expected ',' in object, got " + esc(ch));+                }++                ch = get_next_token();+            }+            return data;+        }++        if (ch == '[') {+            std::vector<Json> data;+            ch = get_next_token();+            if (ch == ']') {+                return data;+            }++            while (true) {+                i--;+                data.push_back(parse_json(depth + 1));+                if (failed) {+                    return Json();+                }++                ch = get_next_token();+                if (ch == ']') {+                    break;+                }+                if (ch != ',') {+                    return fail("expected ',' in list, got " + esc(ch));+                }++                ch = get_next_token();+                (void)ch;+            }+            return data;+        }++        return fail("expected value, got " + esc(ch));+    }+};+}  // namespace++inline Json Json::parse(const std::string& in, std::string& err, JsonParse strategy) {+    JsonParser parser{in, 0, err, false, strategy};+    Json result = parser.parse_json(0);++    // Check for any trailing garbage+    parser.consume_garbage();+    if (parser.failed) {+        return Json();+    }+    if (parser.i != in.size()) {+        return parser.fail("unexpected trailing " + esc(in[parser.i]));+    }++    return result;+}++inline std::vector<Json> parse_multi(const std::string& in, std::string::size_type& parser_stop_pos,+        std::string& err, JsonParse strategy) {+    JsonParser parser{in, 0, err, false, strategy};+    parser_stop_pos = 0;+    std::vector<Json> json_vec;+    while (parser.i != in.size() && !parser.failed) {+        json_vec.push_back(parser.parse_json(0));+        if (parser.failed) {+            break;+        }++        // Check for another object+        parser.consume_garbage();+        if (parser.failed) {+            break;+        }+        parser_stop_pos = parser.i;+    }+    return json_vec;+}++}  // namespace json11
+ cbits/souffle/profile/CellInterface.h view
@@ -0,0 +1,33 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2016, The Souffle Developers. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++#pragma once++#include <chrono>+#include <string>++namespace souffle {+namespace profile {++class CellInterface {+public:+    virtual std::string toString(int precision) const = 0;++    virtual double getDoubleVal() const = 0;++    virtual long getLongVal() const = 0;++    virtual std::string getStringVal() const = 0;++    virtual std::chrono::microseconds getTimeVal() const = 0;++    virtual ~CellInterface() = default;+};++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/profile/DataComparator.h view
@@ -0,0 +1,66 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2016, The Souffle Developers. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++#pragma once++#include "CellInterface.h"+#include "Row.h"++#include <cmath>+#include <memory>+#include <vector>++namespace souffle {+namespace profile {++/*+ * Data comparison functions for sorting tables+ *+ * Will sort the values of only one column, in descending order+ *+ */+class DataComparator {+public:+    /** Sort by total time. */+    static bool TIME(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return a->cells[0]->getDoubleVal() > b->cells[0]->getDoubleVal();+    }++    /** Sort by non-recursive time. */+    static bool NR_T(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return a->cells[1]->getDoubleVal() > b->cells[1]->getDoubleVal();+    }++    /** Sort by recursive time. */+    static bool R_T(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return a->cells[2]->getDoubleVal() > b->cells[2]->getDoubleVal();+    }++    /** Sort by copy time. */+    static bool C_T(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return a->cells[3]->getDoubleVal() > b->cells[3]->getDoubleVal();+    }++    /** Sort by tuple count. */+    static bool TUP(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return b->cells[4]->getLongVal() < a->cells[4]->getLongVal();+    }++    /** Sort by name. */+    static bool NAME(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return b->cells[5]->getStringVal() > a->cells[5]->getStringVal();+    }++    /** Sort by ID. */+    static bool ID(const std::shared_ptr<Row>& a, const std::shared_ptr<Row>& b) {+        return b->cells[6]->getStringVal() > a->cells[6]->getStringVal();+    }+};++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/profile/Row.h view
@@ -0,0 +1,48 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2016, The Souffle Developers. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++#pragma once++#include "CellInterface.h"++#include <iostream>+#include <memory>+#include <string>+#include <vector>++namespace souffle {+namespace profile {++/*+ * Row class for Tables, holds a vector of cells.+ */+class Row {+public:+    std::vector<std::shared_ptr<CellInterface>> cells;++    Row(unsigned long size) : cells() {+        for (unsigned long i = 0; i < size; i++) {+            cells.emplace_back(std::shared_ptr<CellInterface>(nullptr));+        }+    }++    std::shared_ptr<CellInterface>& operator[](unsigned long i) {+        return cells.at(i);+    }++    //    void addCell(int location, std::shared_ptr<CellInterface> cell) {+    //        cells[location] = cell;+    //    }++    inline std::vector<std::shared_ptr<CellInterface>> getCells() {+        return cells;+    }+};++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/profile/Table.h view
@@ -0,0 +1,66 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2016, The Souffle Developers. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++#pragma once++#include "DataComparator.h"+#include "Row.h"+#include <algorithm>+#include <vector>++namespace souffle {+namespace profile {++/*+ * Table class for holding a vector of rows+ * And sorting the rows based on a datacomparator function+ */+class Table {+public:+    std::vector<std::shared_ptr<Row>> rows;++    Table() : rows() {}++    void addRow(std::shared_ptr<Row> row) {+        rows.push_back(row);+    }++    inline std::vector<std::shared_ptr<Row>> getRows() {+        return rows;+    }++    void sort(int col_num) {+        switch (col_num) {+            case 1:+                std::sort(rows.begin(), rows.end(), DataComparator::NR_T);+                break;+            case 2:+                std::sort(rows.begin(), rows.end(), DataComparator::R_T);+                break;+            case 3:+                std::sort(rows.begin(), rows.end(), DataComparator::C_T);+                break;+            case 4:+                std::sort(rows.begin(), rows.end(), DataComparator::TUP);+                break;+            case 5:+                std::sort(rows.begin(), rows.end(), DataComparator::ID);+                break;+            case 6:+                std::sort(rows.begin(), rows.end(), DataComparator::NAME);+                break;+            case 0:+            default:  // if the col_num isn't defined use TIME+                std::sort(rows.begin(), rows.end(), DataComparator::TIME);+                break;+        }+    }+};++}  // namespace profile+}  // namespace souffle
+ cbits/souffle/utility/CacheUtil.h view
@@ -0,0 +1,291 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file CacheUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include <array>+#include <fstream>++// -------------------------------------------------------------------------------+//                              Hint / Cache+// -------------------------------------------------------------------------------++namespace souffle {++/**+ * An Least-Recently-Used cache for arbitrary element types. Elements can be signaled+ * to be accessed and iterated through in their LRU order.+ */+template <typename T, unsigned size = 1>+class LRUCache {+    // the list of pointers maintained+    std::array<T, size> entries;++    // pointer to predecessor / successor in the entries list+    std::array<std::size_t, size> priv;  // < predecessor of element i+    std::array<std::size_t, size> post;  // < successor of element i++    std::size_t first{0};        // < index of the first element+    std::size_t last{size - 1};  // < index of the last element++public:+    // creates a new, empty cache+    LRUCache(const T& val = T()) {+        for (unsigned i = 0; i < size; i++) {+            entries[i] = val;+            priv[i] = i - 1;+            post[i] = i + 1;+        }+        priv[first] = last;+        post[last] = first;+    }++    // clears the content of this cache+    void clear(const T& val = T()) {+        for (auto& cur : entries) {+            cur = val;+        }+    }++    // registers an access to the given element+    void access(const T& val) {+        // test whether it is contained+        for (std::size_t i = 0; i < size; i++) {+            if (entries[i] != val) {+                continue;+            }++            // -- move this one to the front --++            // if it is the first, nothing to handle+            if (i == first) {+                return;+            }++            // if this is the last, just first and last need to change+            if (i == last) {+                auto tmp = last;+                last = priv[last];+                first = tmp;+                return;+            }++            // otherwise we need to update the linked list++            // remove from current position+            post[priv[i]] = post[i];+            priv[post[i]] = priv[i];++            // insert in first position+            post[i] = first;+            priv[i] = last;+            priv[first] = i;+            post[last] = i;++            // update first pointer+            first = i;+            return;+        }+        // not present => drop last, make it first+        entries[last] = val;+        auto tmp = last;+        last = priv[last];+        first = tmp;+    }++    /**+     * Iterates over the elements within this cache in LRU order.+     * The operator is applied on each element. If the operation+     * returns false, iteration is continued. If the operator return+     * true, iteration is stopped -- similar to the any operator.+     *+     * @param op the operator to be applied on every element+     * @return true if op returned true for any entry, false otherwise+     */+    template <typename Op>+    bool forEachInOrder(const Op& op) const {+        std::size_t i = first;+        while (i != last) {+            if (op(entries[i])) return true;+            i = post[i];+        }+        return op(entries[i]);+    }++    // equivalent to forEachInOrder+    template <typename Op>+    bool any(const Op& op) const {+        return forEachInOrder(op);+    }+};++template <typename T, unsigned size>+std::ostream& operator<<(std::ostream& out, const LRUCache<T, size>& cache) {+    bool first = true;+    cache.forEachInOrder([&](const T& val) {+        if (!first) {+            out << ",";+        }+        first = false;+        out << val;+        return false;+    });+    return out;+}++// a specialization for a single-entry cache+template <typename T>+class LRUCache<T, 1> {+    // the single entry in this cache+    T entry;++public:+    // creates a new, empty cache+    LRUCache() : entry() {}++    // creates a new, empty cache storing the given value+    LRUCache(const T& val) : entry(val) {}++    // clears the content of this cache+    void clear(const T& val = T()) {+        entry = val;+    }++    // registers an access to the given element+    void access(const T& val) {+        entry = val;+    }++    /**+     * See description in most general case.+     */+    template <typename Op>+    bool forEachInOrder(const Op& op) const {+        return op(entry);+    }++    // equivalent to forEachInOrder+    template <typename Op>+    bool any(const Op& op) const {+        return forEachInOrder(op);+    }++    // --- print support ---++    friend std::ostream& operator<<(std::ostream& out, const LRUCache& cache) {+        return out << cache.entry;+    }+};++// a specialization for no-entry caches.+template <typename T>+class LRUCache<T, 0> {+public:+    // creates a new, empty cache+    LRUCache(const T& = T()) {}++    // clears the content of this cache+    void clear(const T& = T()) {+        // nothing to do+    }++    // registers an access to the given element+    void access(const T&) {+        // nothing to do+    }++    /**+     * Always returns false.+     */+    template <typename Op>+    bool forEachInOrder(const Op&) const {+        return false;+    }++    // equivalent to forEachInOrder+    template <typename Op>+    bool any(const Op& op) const {+        return forEachInOrder(op);+    }++    // --- print support ---++    friend std::ostream& operator<<(std::ostream& out, const LRUCache& /* cache */) {+        return out << "-empty-";+    }+};++// -------------------------------------------------------------------------------+//                           Hint / Cache Profiling+// -------------------------------------------------------------------------------++/**+ * cache hits/misses.+ */+#ifdef _SOUFFLE_STATS++#include <atomic>++class CacheAccessCounter {+    std::atomic<std::size_t> hits;+    std::atomic<std::size_t> misses;++public:+    CacheAccessCounter() : hits(0), misses(0) {}+    CacheAccessCounter(const CacheAccessCounter& other) : hits(other.getHits()), misses(other.getMisses()) {}+    void addHit() {+        hits.fetch_add(1, std::memory_order_relaxed);+    }+    void addMiss() {+        misses.fetch_add(1, std::memory_order_relaxed);+    }+    std::size_t getHits() const {+        return hits;+    }+    std::size_t getMisses() const {+        return misses;+    }+    std::size_t getAccesses() const {+        return getHits() + getMisses();+    }+    void reset() {+        hits = 0;+        misses = 0;+    }+};++#else++class CacheAccessCounter {+public:+    CacheAccessCounter() = default;+    CacheAccessCounter(const CacheAccessCounter& /* other */) = default;+    inline void addHit() {}+    inline void addMiss() {}+    inline std::size_t getHits() {+        return 0;+    }+    inline std::size_t getMisses() {+        return 0;+    }+    inline std::size_t getAccesses() {+        return 0;+    }+    inline void reset() {}+};++#endif+}  // end namespace souffle
+ cbits/souffle/utility/ContainerUtil.h view
@@ -0,0 +1,407 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ContainerUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include <algorithm>+#include <functional>+#include <iterator>+#include <map>+#include <memory>+#include <set>+#include <type_traits>+#include <utility>+#include <vector>++namespace souffle {++// -------------------------------------------------------------------------------+//                           General Container Utilities+// -------------------------------------------------------------------------------++template <typename A>+using Own = std::unique_ptr<A>;++template <typename A>+using VecOwn = std::vector<Own<A>>;++template <typename A, typename B = A, typename... Args>+Own<A> mk(Args&&... xs) {+    return Own<A>(new B(std::forward<Args>(xs)...));+}++/**+ * Use to range-for iterate in reverse.+ * Assumes `std::rbegin` and `std::rend` are defined for type `A`.+ */+template <typename A>+struct reverse {+    reverse(A& iterable) : iterable(iterable) {}+    A& iterable;++    auto begin() {+        return std::rbegin(iterable);+    }++    auto end() {+        return std::rend(iterable);+    }+};++/**+ * A utility to check generically whether a given element is contained in a given+ * container.+ */+template <typename C>+bool contains(const C& container, const typename C::value_type& element) {+    return std::find(container.begin(), container.end(), element) != container.end();+}++// TODO: Detect and generalise to other set types?+template <typename A>+bool contains(const std::set<A>& container, const A& element) {+    return container.find(element) != container.end();+}++/**+ * Version of contains specialised for maps.+ *+ * This workaround is needed because of set container, for which value_type == key_type,+ * which is ambiguous in this context.+ */+template <typename C>+bool contains(const C& container, const typename C::value_type::first_type& element) {+    return container.find(element) != container.end();+}++/**+ * Returns the first element in a container that satisfies a given predicate,+ * nullptr otherwise.+ */+template <typename C>+typename C::value_type getIf(const C& container, std::function<bool(const typename C::value_type)> pred) {+    auto res = std::find_if(container.begin(), container.end(),+            [&](const typename C::value_type item) { return pred(item); });+    return res == container.end() ? nullptr : *res;+}++/**+ * Get value for a given key; if not found, return default value.+ */+template <typename C>+typename C::mapped_type const& getOr(+        const C& container, typename C::key_type key, const typename C::mapped_type& defaultValue) {+    auto it = container.find(key);++    if (it != container.end()) {+        return it->second;+    } else {+        return defaultValue;+    }+}++/**+ * A utility function enabling the creation of a vector with a fixed set of+ * elements within a single expression. This is the base case covering empty+ * vectors.+ */+template <typename T>+std::vector<T> toVector() {+    return std::vector<T>();+}++/**+ * A utility function enabling the creation of a vector with a fixed set of+ * elements within a single expression. This is the step case covering vectors+ * of arbitrary length.+ */+template <typename T, typename... R>+std::vector<T> toVector(const T& first, const R&... rest) {+    return {first, rest...};+}++/**+ * A utility function enabling the creation of a vector of pointers.+ */+template <typename T>+std::vector<T*> toPtrVector(const std::vector<std::unique_ptr<T>>& v) {+    std::vector<T*> res;+    for (auto& e : v) {+        res.push_back(e.get());+    }+    return res;+}++/**+ * Applies a function to each element of a vector and returns the results.+ */+template <typename A, typename F /* : A -> B */>+auto map(const std::vector<A>& xs, F&& f) {+    std::vector<decltype(f(xs[0]))> ys;+    ys.reserve(xs.size());+    for (auto&& x : xs) {+        ys.emplace_back(f(x));+    }+    return ys;+}++// -------------------------------------------------------------------------------+//                             Cloning Utilities+// -------------------------------------------------------------------------------++template <typename A>+auto clone(const std::vector<A>& xs) {+    std::vector<decltype(clone(xs[0]))> ys;+    ys.reserve(xs.size());+    for (auto&& x : xs) {+        ys.emplace_back(clone(x));+    }+    return ys;+}++// -------------------------------------------------------------+//                             Ranges+// -------------------------------------------------------------++/**+ * A utility class enabling representation of ranges by pairing+ * two iterator instances marking lower and upper boundaries.+ */+template <typename Iter>+struct range {+    // the lower and upper boundary+    Iter a, b;++    // a constructor accepting a lower and upper boundary+    range(Iter a, Iter b) : a(std::move(a)), b(std::move(b)) {}++    // default copy / move and assignment support+    range(const range&) = default;+    range(range&&) = default;+    range& operator=(const range&) = default;++    // get the lower boundary (for for-all loop)+    Iter& begin() {+        return a;+    }+    const Iter& begin() const {+        return a;+    }++    // get the upper boundary (for for-all loop)+    Iter& end() {+        return b;+    }+    const Iter& end() const {+        return b;+    }++    // emptiness check+    bool empty() const {+        return a == b;+    }++    // splits up this range into the given number of partitions+    std::vector<range> partition(int np = 100) {+        // obtain the size+        int n = 0;+        for (auto i = a; i != b; ++i) {+            n++;+        }++        // split it up+        auto s = n / np;+        auto r = n % np;+        std::vector<range> res;+        res.reserve(np);+        auto cur = a;+        auto last = cur;+        int i = 0;+        int p = 0;+        while (cur != b) {+            ++cur;+            i++;+            if (i >= (s + (p < r ? 1 : 0))) {+                res.push_back({last, cur});+                last = cur;+                p++;+                i = 0;+            }+        }+        if (cur != last) {+            res.push_back({last, cur});+        }+        return res;+    }+};++/**+ * A utility function enabling the construction of ranges+ * without explicitly specifying the iterator type.+ *+ * @tparam Iter .. the iterator type+ * @param a .. the lower boundary+ * @param b .. the upper boundary+ */+template <typename Iter>+range<Iter> make_range(const Iter& a, const Iter& b) {+    return range<Iter>(a, b);+}++// -------------------------------------------------------------------------------+//                             Equality Utilities+// -------------------------------------------------------------------------------++/**+ * Cast the values, from baseType to toType and compare using ==. (if casting fails -> return false.)+ *+ * @tparam baseType, initial Type of values+ * @tparam toType, type where equality comparison takes place.+ */+template <typename toType, typename baseType>+bool castEq(const baseType* left, const baseType* right) {+    if (auto castedLeft = dynamic_cast<const toType*>(left)) {+        if (auto castedRight = dynamic_cast<const toType*>(right)) {+            return castedLeft == castedRight;+        }+    }+    return false;+}++/**+ * A functor class supporting the values pointers are pointing to.+ */+template <typename T>+struct comp_deref {+    bool operator()(const T& a, const T& b) const {+        if (a == nullptr) {+            return false;+        }+        if (b == nullptr) {+            return false;+        }+        return *a == *b;+    }+};++/**+ * A function testing whether two containers are equal with the given Comparator.+ */+template <typename Container, typename Comparator>+bool equal_targets(const Container& a, const Container& b, const Comparator& comp) {+    // check reference+    if (&a == &b) {+        return true;+    }++    // check size+    if (a.size() != b.size()) {+        return false;+    }++    // check content+    return std::equal(a.begin(), a.end(), b.begin(), comp);+}++/**+ * A function testing whether two containers of pointers are referencing equivalent+ * targets.+ */+template <typename T, template <typename...> class Container>+bool equal_targets(const Container<T*>& a, const Container<T*>& b) {+    return equal_targets(a, b, comp_deref<T*>());+}++/**+ * A function testing whether two containers of unique pointers are referencing equivalent+ * targets.+ */+template <typename T, template <typename...> class Container>+bool equal_targets(const Container<std::unique_ptr<T>>& a, const Container<std::unique_ptr<T>>& b) {+    return equal_targets(a, b, comp_deref<std::unique_ptr<T>>());+}++/**+ * A function testing whether two maps of unique pointers are referencing to equivalent+ * targets.+ */+template <typename Key, typename Value>+bool equal_targets(+        const std::map<Key, std::unique_ptr<Value>>& a, const std::map<Key, std::unique_ptr<Value>>& b) {+    auto comp = comp_deref<std::unique_ptr<Value>>();+    return equal_targets(+            a, b, [&comp](auto& a, auto& b) { return a.first == b.first && comp(a.second, b.second); });+}++/**+ * Compares two values referenced by a pointer where the case where both+ * pointers are null is also considered equivalent.+ */+template <typename T>+bool equal_ptr(const T* a, const T* b) {+    if (a == nullptr && b == nullptr) {+        return true;+    }+    if (a != nullptr && b != nullptr) {+        return *a == *b;+    }+    return false;+}++/**+ * Compares two values referenced by a pointer where the case where both+ * pointers are null is also considered equivalent.+ */+template <typename T>+bool equal_ptr(const std::unique_ptr<T>& a, const std::unique_ptr<T>& b) {+    return equal_ptr(a.get(), b.get());+}++template <typename A, typename B>+using copy_const_t = std::conditional_t<std::is_const_v<A>, const B, B>;++/**+ * Helpers for `dynamic_cast`ing without having to specify redundant type qualifiers.+ * e.g. `as<AstLiteral>(p)` instead of `dynamic_cast<const AstLiteral*>(p.get())`.+ */+template <typename B, typename A>+auto as(A* x) {+    static_assert(std::is_base_of_v<A, B>,+            "`as<B, A>` does not allow cross-type dyn casts. "+            "(i.e. `as<B, A>` where `B <: A` is not true.) "+            "Such a cast is likely a mistake or typo.");+    return dynamic_cast<copy_const_t<A, B>*>(x);+}++template <typename B, typename A>+std::enable_if_t<std::is_base_of_v<A, B>, copy_const_t<A, B>*> as(A& x) {+    return as<B>(&x);+}++template <typename B, typename A>+B* as(const Own<A>& x) {+    return as<B>(x.get());+}++/**+ * Checks if the object of type Source can be casted to type Destination.+ */+template <typename Destination, typename Source>+bool isA(Source&& src) {+    return as<Destination>(std::forward<Source>(src)) != nullptr;+}++}  // namespace souffle
+ cbits/souffle/utility/EvaluatorUtil.h view
@@ -0,0 +1,100 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2020, The Souffle Developers. All rights reserved.+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file EvaluatorUtils.h+ *+ * Defines utility functions used by synthesised and interpreter code.+ *+ ***********************************************************************/++#pragma once++#include "../CompiledTuple.h"+#include "../RamTypes.h"+#include "tinyformat.h"++namespace souffle::evaluator {++template <typename A, typename F /* Tuple<RamDomain,1> -> void */>+void runRange(A from, A to, A step, F&& go) {+#define GO(x) go(Tuple<RamDomain, 1>{ramBitCast(x)})+    if (0 < step) {+        for (auto x = from; x < to; x += step) {+            GO(x);+        }+    } else if (step < 0) {+        for (auto x = from; to < x; x += step) {+            GO(x);+        }+    } else if (from != to) {+        // `step = 0` edge case, only if non-empty range+        GO(from);+    }+#undef GO+}++template <typename A, typename F /* Tuple<RamDomain,1> -> void */>+void runRange(A from, A to, F&& go) {+    return runRange(from, to, A(from <= to ? 1 : -1), std::forward<F>(go));+}++namespace details {+template <typename A>+A symbol2numeric(const std::string& s);++#define SYM_2_NUMERIC_OVERLOAD(ty)                \+    template <>                                   \+    ty symbol2numeric<ty>(const std::string& s) { \+        return ty##FromString(s);                 \+    }++SYM_2_NUMERIC_OVERLOAD(RamFloat)+SYM_2_NUMERIC_OVERLOAD(RamSigned)+SYM_2_NUMERIC_OVERLOAD(RamUnsigned)+#undef SYM_2_NUMERIC_OVERLOAD+}  // namespace details++template <typename A>+A symbol2numeric(const std::string& src) {+    try {+        return details::symbol2numeric<A>(src);+    } catch (...) {+        tfm::format(std::cerr, "error: wrong string provided by `to_number(\"%s\")` functor.\n", src);+        raise(SIGFPE);+        fatal("");  // UNREACHABLE: `raise` lacks a no-return attribute+    }+};++template <typename A>+bool lxor(A x, A y) {+    return (x || y) && (!x != !y);+}++// HACK:  C++ doesn't have an infix logical xor operator.+//        C++ doesn't allow defining new infix operators.+//        C++ isn't a very nice language, but C++ does allow overload-based war crimes.+//        This particular war crime allows a very verbose infix op for `lxor`.+//        It should only be used in macro dispatches.+struct lxor_infix {+    template <typename A>+    struct curry {+        A x;+        bool operator+(A y) const {+            return lxor(x, y);+        }+    };+};++template <typename A>+lxor_infix::curry<A> operator+(A x, lxor_infix) {+    return lxor_infix::curry<A>{x};+}++}  // namespace souffle::evaluator
+ cbits/souffle/utility/FileUtil.h view
@@ -0,0 +1,293 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file FileUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include <climits>+#include <cstdio>+#include <cstdlib>+#include <fstream>+#include <sstream>+#include <string>+#include <utility>+#include <sys/stat.h>++#ifndef _WIN32+#include <unistd.h>+#else+#include <fcntl.h>+#include <io.h>+#include <stdlib.h>+#include <windows.h>++// -------------------------------------------------------------------------------+//                               File Utils+// -------------------------------------------------------------------------------++#define X_OK 1 /* execute permission - unsupported in windows*/++#define PATH_MAX 260++/**+ * access and realpath are missing on windows, we use their windows equivalents+ * as work-arounds.+ */+#define access _access+inline char* realpath(const char* path, char* resolved_path) {+    return _fullpath(resolved_path, path, PATH_MAX);+}++#endif++namespace souffle {++/**+ *  Check whether a file exists in the file system+ */+inline bool existFile(const std::string& name) {+    struct stat buffer = {};+    if (stat(name.c_str(), &buffer) == 0) {+        if ((buffer.st_mode & S_IFMT) != 0) {+            return true;+        }+    }+    return false;+}++/**+ *  Check whether a directory exists in the file system+ */+inline bool existDir(const std::string& name) {+    struct stat buffer = {};+    if (stat(name.c_str(), &buffer) == 0) {+        if ((buffer.st_mode & S_IFDIR) != 0) {+            return true;+        }+    }+    return false;+}++/**+ * Check whether a given file exists and it is an executable+ */+inline bool isExecutable(const std::string& name) {+    return existFile(name) && (access(name.c_str(), X_OK) == 0);+}++/**+ * Simple implementation of a which tool+ */+inline std::string which(const std::string& name) {+    char buf[PATH_MAX];+    if ((::realpath(name.c_str(), buf) != nullptr) && isExecutable(buf)) {+        return buf;+    }+    const char* syspath = ::getenv("PATH");+    if (syspath == nullptr) {+        return "";+    }+    std::stringstream sstr;+    sstr << syspath;+    std::string sub;+    while (std::getline(sstr, sub, ':')) {+        std::string path = sub + "/" + name;+        if (isExecutable(path) && (realpath(path.c_str(), buf) != nullptr)) {+            return buf;+        }+    }+    return "";+}++/**+ *  C++-style dirname+ */+inline std::string dirName(const std::string& name) {+    if (name.empty()) {+        return ".";+    }+    size_t lastNotSlash = name.find_last_not_of('/');+    // All '/'+    if (lastNotSlash == std::string::npos) {+        return "/";+    }+    size_t leadingSlash = name.find_last_of('/', lastNotSlash);+    // No '/'+    if (leadingSlash == std::string::npos) {+        return ".";+    }+    // dirname is '/'+    if (leadingSlash == 0) {+        return "/";+    }+    return name.substr(0, leadingSlash);+}++/**+ *  C++-style realpath+ */+inline std::string absPath(const std::string& path) {+    char buf[PATH_MAX];+    char* res = realpath(path.c_str(), buf);+    return (res == nullptr) ? "" : std::string(buf);+}++/**+ *  Join two paths together; note that this does not resolve overlaps or relative paths.+ */+inline std::string pathJoin(const std::string& first, const std::string& second) {+    unsigned firstPos = static_cast<unsigned>(first.size()) - 1;+    while (first.at(firstPos) == '/') {+        firstPos--;+    }+    unsigned secondPos = 0;+    while (second.at(secondPos) == '/') {+        secondPos++;+    }+    return first.substr(0, firstPos + 1) + '/' + second.substr(secondPos);+}++/*+ * Find out if an executable given by @p tool exists in the path given @p path+ * relative to the directory given by @ base. A path here refers a+ * colon-separated list of directories.+ */+inline std::string findTool(const std::string& tool, const std::string& base, const std::string& path) {+    std::string dir = dirName(base);+    std::stringstream sstr(path);+    std::string sub;++    while (std::getline(sstr, sub, ':')) {+        std::string subpath = dir + "/" + sub + '/' + tool;+        if (isExecutable(subpath)) {+            return absPath(subpath);+        }+    }+    return "";+}++/*+ * Get the basename of a fully qualified filename+ */+inline std::string baseName(const std::string& filename) {+    if (filename.empty()) {+        return ".";+    }++    size_t lastNotSlash = filename.find_last_not_of('/');+    if (lastNotSlash == std::string::npos) {+        return "/";+    }++    size_t lastSlashBeforeBasename = filename.find_last_of('/', lastNotSlash - 1);+    if (lastSlashBeforeBasename == std::string::npos) {+        lastSlashBeforeBasename = static_cast<size_t>(-1);+    }+    return filename.substr(lastSlashBeforeBasename + 1, lastNotSlash - lastSlashBeforeBasename);+}++/**+ * File name, with extension removed.+ */+inline std::string simpleName(const std::string& path) {+    std::string name = baseName(path);+    const size_t lastDot = name.find_last_of('.');+    // file has no extension+    if (lastDot == std::string::npos) {+        return name;+    }+    const size_t lastSlash = name.find_last_of('/');+    // last slash occurs after last dot, so no extension+    if (lastSlash != std::string::npos && lastSlash > lastDot) {+        return name;+    }+    // last dot after last slash, or no slash+    return name.substr(0, lastDot);+}++/**+ * File extension, with all else removed.+ */+inline std::string fileExtension(const std::string& path) {+    std::string name = path;+    const size_t lastDot = name.find_last_of('.');+    // file has no extension+    if (lastDot == std::string::npos) {+        return std::string();+    }+    const size_t lastSlash = name.find_last_of('/');+    // last slash occurs after last dot, so no extension+    if (lastSlash != std::string::npos && lastSlash > lastDot) {+        return std::string();+    }+    // last dot after last slash, or no slash+    return name.substr(lastDot + 1);+}++/**+ * Generate temporary file.+ */+inline std::string tempFile() {+#ifdef _WIN32+    std::string templ;+    std::FILE* f = nullptr;+    while (f == nullptr) {+        templ = std::tmpnam(nullptr);+        f = fopen(templ.c_str(), "wx");+    }+    fclose(f);+    return templ;+#else+    char templ[40] = "./souffleXXXXXX";+    close(mkstemp(templ));+    return std::string(templ);+#endif+}++inline std::stringstream execStdOut(char const* cmd) {+    FILE* in = popen(cmd, "r");+    std::stringstream data;+    while (in != nullptr) {+        char c = fgetc(in);+        if (feof(in) != 0) {+            break;+        }+        data << c;+    }+    pclose(in);+    return data;+}++inline std::stringstream execStdOut(std::string const& cmd) {+    return execStdOut(cmd.c_str());+}++class TempFileStream : public std::fstream {+    std::string fileName;++public:+    TempFileStream(std::string fileName = tempFile())+            : std::fstream(fileName), fileName(std::move(fileName)) {}+    ~TempFileStream() override {+        close();+        remove(fileName.c_str());+    }++    std::string const& getFileName() const {+        return fileName;+    }+};++}  // namespace souffle
+ cbits/souffle/utility/FunctionalUtil.h view
@@ -0,0 +1,151 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file FunctionalUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include <algorithm>+#include <functional>++namespace souffle {++// -------------------------------------------------------------------------------+//                              Functional Utils+// -------------------------------------------------------------------------------++/**+ * A functor comparing the dereferenced value of a pointer type utilizing a+ * given comparator. Its main use case are sets of non-null pointers which should+ * be ordered according to the value addressed by the pointer.+ */+template <typename T, typename C = std::less<T>>+struct deref_less {+    bool operator()(const T* a, const T* b) const {+        return C()(*a, *b);+    }+};++// -------------------------------------------------------------------------------+//                               Lambda Utils+// -------------------------------------------------------------------------------++namespace detail {++template <typename T>+struct lambda_traits_helper;++template <typename R>+struct lambda_traits_helper<R()> {+    using result_type = R;+};++template <typename R, typename A0>+struct lambda_traits_helper<R(A0)> {+    using result_type = R;+    using arg0_type = A0;+};++template <typename R, typename A0, typename A1>+struct lambda_traits_helper<R(A0, A1)> {+    using result_type = R;+    using arg0_type = A0;+    using arg1_type = A1;+};++template <typename R, typename... Args>+struct lambda_traits_helper<R(Args...)> {+    using result_type = R;+};++template <typename R, typename C, typename... Args>+struct lambda_traits_helper<R (C::*)(Args...)> : public lambda_traits_helper<R(Args...)> {};++template <typename R, typename C, typename... Args>+struct lambda_traits_helper<R (C::*)(Args...) const> : public lambda_traits_helper<R (C::*)(Args...)> {};+}  // namespace detail++/**+ * A type trait enabling the deduction of type properties of lambdas.+ * Those include so far:+ *      - the result type (result_type)+ *      - the first argument type (arg0_type)+ */+template <typename Lambda>+struct lambda_traits : public detail::lambda_traits_helper<decltype(&Lambda::operator())> {};++// -------------------------------------------------------------------------------+//                              General Algorithms+// -------------------------------------------------------------------------------++/**+ * A generic test checking whether all elements within a container satisfy a+ * certain predicate.+ *+ * @param c the container+ * @param p the predicate+ * @return true if for all elements x in c the predicate p(x) is true, false+ *          otherwise; for empty containers the result is always true+ */+template <typename Container, typename UnaryPredicate>+bool all_of(const Container& c, UnaryPredicate p) {+    return std::all_of(c.begin(), c.end(), p);+}++/**+ * A generic test checking whether any elements within a container satisfy a+ * certain predicate.+ *+ * @param c the container+ * @param p the predicate+ * @return true if there is an element x in c such that predicate p(x) is true, false+ *          otherwise; for empty containers the result is always false+ */+template <typename Container, typename UnaryPredicate>+bool any_of(const Container& c, UnaryPredicate p) {+    return std::any_of(c.begin(), c.end(), p);+}++/**+ * A generic test checking whether all elements within a container satisfy a+ * certain predicate.+ *+ * @param c the container+ * @param p the predicate+ * @return true if for all elements x in c the predicate p(x) is true, false+ *          otherwise; for empty containers the result is always true+ */+template <typename Container, typename UnaryPredicate>+bool none_of(const Container& c, UnaryPredicate p) {+    return std::none_of(c.begin(), c.end(), p);+}++/**+ * Filter a vector to exclude certain elements.+ */+template <typename A, typename F>+std::vector<A> filterNot(std::vector<A> xs, F&& f) {+    xs.erase(std::remove_if(xs.begin(), xs.end(), std::forward<F>(f)), xs.end());+    return xs;+}++/**+ * Filter a vector to include certain elements.+ */+template <typename A, typename F>+std::vector<A> filter(std::vector<A> xs, F&& f) {+    return filterNot(std::move(xs), [&](auto&& x) { return !f(x); });+}++}  // namespace souffle
+ cbits/souffle/utility/MiscUtil.h view
@@ -0,0 +1,129 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file MiscUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include "tinyformat.h"+#include <cassert>+#include <chrono>+#include <cstdlib>+#include <iostream>+#include <memory>+#include <utility>++#ifdef _WIN32+#include <fcntl.h>+#include <io.h>+#include <stdlib.h>+#include <windows.h>++/**+ * Windows headers define these and they interfere with the standard library+ * functions.+ */+#undef min+#undef max++/**+ * On windows, the following gcc builtins are missing.+ *+ * In the case of popcountll, __popcnt64 is the windows equivalent.+ *+ * For ctz and ctzll, BitScanForward and BitScanForward64 are the respective+ * windows equivalents.  However ctz is used in a constexpr context, and we can't+ * use BitScanForward, so we implement it ourselves.+ */+#define __builtin_popcountll __popcnt64++constexpr unsigned long __builtin_ctz(unsigned long value) {+    unsigned long trailing_zeroes = 0;+    while ((value = value >> 1) ^ 1) {+        ++trailing_zeroes;+    }+    return trailing_zeroes;+}++inline unsigned long __builtin_ctzll(unsigned long long value) {+    unsigned long trailing_zero = 0;++    if (_BitScanForward64(&trailing_zero, value)) {+        return trailing_zero;+    } else {+        return 64;+    }+}+#endif++// -------------------------------------------------------------------------------+//                               Timing Utils+// -------------------------------------------------------------------------------++namespace souffle {++// a type def for a time point+using time_point = std::chrono::high_resolution_clock::time_point;+using std::chrono::microseconds;++// a shortcut for taking the current time+inline time_point now() {+    return std::chrono::high_resolution_clock::now();+}++// a shortcut for obtaining the time difference in milliseconds+inline long duration_in_us(const time_point& start, const time_point& end) {+    return static_cast<long>(std::chrono::duration_cast<std::chrono::microseconds>(end - start).count());+}++// a shortcut for obtaining the time difference in nanoseconds+inline long duration_in_ns(const time_point& start, const time_point& end) {+    return static_cast<long>(std::chrono::duration_cast<std::chrono::nanoseconds>(end - start).count());+}++// -------------------------------------------------------------------------------+//                             Cloning Utilities+// -------------------------------------------------------------------------------++template <typename A>+std::unique_ptr<A> clone(const A* node) {+    return node ? std::unique_ptr<A>(node->clone()) : nullptr;+}++template <typename A>+std::unique_ptr<A> clone(const std::unique_ptr<A>& node) {+    return node ? std::unique_ptr<A>(node->clone()) : nullptr;+}++template <typename A, typename B>+auto clone(const std::pair<A, B>& p) {+    return std::make_pair(clone(p.first), clone(p.second));+}++// -------------------------------------------------------------------------------+//                               Error Utilities+// -------------------------------------------------------------------------------++template <typename... Args>+[[noreturn]] void fatal(const char* format, const Args&... args) {+    tfm::format(std::cerr, format, args...);+    std::cerr << "\n";+    assert(false && "fatal error; see std err");+    abort();+}++// HACK:  Workaround to suppress spurious reachability warnings.+#define UNREACHABLE_BAD_CASE_ANALYSIS fatal("unhandled switch branch");++}  // namespace souffle
+ cbits/souffle/utility/ParallelUtil.h view
@@ -0,0 +1,573 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file ParallelUtil.h+ *+ * A set of utilities abstracting from the underlying parallel library.+ * Currently supported APIs: OpenMP and Cilk+ *+ ***********************************************************************/++#pragma once++#include <atomic>++#ifdef _OPENMP++/**+ * Implementation of parallel control flow constructs utilizing OpenMP+ */++#include <omp.h>++#ifdef __APPLE__+#define pthread_yield pthread_yield_np+#endif++// support for a parallel region+#define PARALLEL_START _Pragma("omp parallel") {+#define PARALLEL_END }++// support for parallel loops+#define pfor _Pragma("omp for schedule(dynamic)") for++// spawn and sync are processed sequentially (overhead to expensive)+#define task_spawn+#define task_sync++// section start / end => corresponding OpenMP pragmas+// NOTE: disabled since it causes performance losses+//#define SECTIONS_START _Pragma("omp parallel sections") {+// NOTE: we stick to flat-level parallelism since it is faster due to thread pooling+#define SECTIONS_START {+#define SECTIONS_END }++// the markers for a single section+//#define SECTION_START _Pragma("omp section") {+#define SECTION_START {+#define SECTION_END }++// a macro to create an operation context+#define CREATE_OP_CONTEXT(NAME, INIT) auto NAME = INIT;+#define READ_OP_CONTEXT(NAME) NAME++#else++// support for a parallel region => sequential execution+#define PARALLEL_START {+#define PARALLEL_END }++// support for parallel loops => simple sequential loop+#define pfor for++// spawn and sync not supported+#define task_spawn+#define task_sync++// sections are processed sequentially+#define SECTIONS_START {+#define SECTIONS_END }++// sections are inlined+#define SECTION_START {+#define SECTION_END }++// a macro to create an operation context+#define CREATE_OP_CONTEXT(NAME, INIT) auto NAME = INIT;+#define READ_OP_CONTEXT(NAME) NAME++// mark es sequential+#define IS_SEQUENTIAL++#endif++#ifndef IS_SEQUENTIAL+#define IS_PARALLEL+#endif++#ifdef IS_PARALLEL+#define MAX_THREADS (omp_get_max_threads())+#else+#define MAX_THREADS (1)+#endif++#ifdef IS_PARALLEL++#include <mutex>++namespace souffle {++/**+ * A small utility class for implementing simple locks.+ */+class Lock {+    // the underlying mutex+    std::mutex mux;++public:+    struct Lease {+        Lease(std::mutex& mux) : mux(&mux) {+            mux.lock();+        }+        Lease(Lease&& other) : mux(other.mux) {+            other.mux = nullptr;+        }+        Lease(const Lease& other) = delete;+        ~Lease() {+            if (mux != nullptr) {+                mux->unlock();+            }+        }++    protected:+        std::mutex* mux;+    };++    // acquired the lock for the live-cycle of the returned guard+    Lease acquire() {+        return Lease(mux);+    }++    void lock() {+        mux.lock();+    }++    bool try_lock() {+        return mux.try_lock();+    }++    void unlock() {+        mux.unlock();+    }+};++//    /* valuable source: http://locklessinc.com/articles/locks/ */++namespace detail {++/* Pause instruction to prevent excess processor bus usage */+#ifdef __x86_64__+#define cpu_relax() asm volatile("pause\n" : : : "memory")+#else+#define cpu_relax() asm volatile("" : : : "memory")+#endif++/**+ * A utility class managing waiting operations for spin locks.+ */+class Waiter {+    int i = 0;++public:+    Waiter() = default;++    /**+     * Conducts a wait operation.+     */+    void operator()() {+        ++i;+        if ((i % 1000) == 0) {+            // there was no progress => let others work+            pthread_yield();+        } else {+            // relax this CPU+            cpu_relax();+        }+    }+};+}  // namespace detail++/* compare: http://en.cppreference.com/w/cpp/atomic/atomic_flag */+class SpinLock {+    std::atomic<int> lck{0};++public:+    SpinLock() = default;++    void lock() {+        detail::Waiter wait;+        while (!try_lock()) {+            wait();+        }+    }++    bool try_lock() {+        int should = 0;+        return lck.compare_exchange_weak(should, 1, std::memory_order_acquire);+    }++    void unlock() {+        lck.store(0, std::memory_order_release);+    }+};++/**+ * A read/write lock for increased access performance on a+ * read-heavy use case.+ */+class ReadWriteLock {+    /**+     * Based on paper:+     *         Scalable Reader-Writer Synchronization+     *         for Shared-Memory Multiprocessors+     *+     * Layout of the lock:+     *      31        ...             2                    1                    0+     *      +-------------------------+--------------------+--------------------++     *      | interested reader count |   waiting writer   | active writer flag |+     *      +-------------------------+--------------------+--------------------++     */++    std::atomic<int> lck{0};++public:+    ReadWriteLock() = default;++    void start_read() {+        // add reader+        auto r = lck.fetch_add(4, std::memory_order_acquire);++        // wait until there is no writer any more+        detail::Waiter wait;+        while (r & 0x3) {+            // release reader+            end_read();++            // wait a bit+            wait();++            // apply as a reader again+            r = lck.fetch_add(4, std::memory_order_acquire);++        }  // while there is a writer => spin+    }++    void end_read() {+        lck.fetch_sub(4, std::memory_order_release);+    }++    void start_write() {+        detail::Waiter wait;++        // set wait-for-write bit+        auto stat = lck.fetch_or(2, std::memory_order_acquire);+        while (stat & 0x2) {+            wait();+            stat = lck.fetch_or(2, std::memory_order_acquire);+        }++        // the caller may starve here ...+        int should = 2;+        while (!lck.compare_exchange_strong(+                should, 1, std::memory_order_acquire, std::memory_order_relaxed)) {+            wait();+            should = 2;+        }+    }++    bool try_write() {+        int should = 0;+        return lck.compare_exchange_strong(should, 1, std::memory_order_acquire, std::memory_order_relaxed);+    }++    void end_write() {+        lck.fetch_sub(1, std::memory_order_release);+    }++    bool try_upgrade_to_write() {+        int should = 4;+        return lck.compare_exchange_strong(should, 1, std::memory_order_acquire, std::memory_order_relaxed);+    }++    void downgrade_to_read() {+        // delete write bit + set num readers to 1+        lck.fetch_add(3, std::memory_order_release);+    }+};++/**+ * An implementation of an optimistic r/w lock.+ */+class OptimisticReadWriteLock {+    /**+     * The version number utilized for the synchronization.+     *+     * Usage:+     *      - even version numbers are stable versions, not being updated+     *      - odd version numbers are temporary versions, currently being updated+     */+    std::atomic<int> version{0};++public:+    /**+     * The lease utilized to link start and end of read phases.+     */+    class Lease {+        friend class OptimisticReadWriteLock;+        int version;++    public:+        Lease(int version = 0) : version(version) {}+        Lease(const Lease& lease) = default;+        Lease& operator=(const Lease& other) = default;+        Lease& operator=(Lease&& other) = default;+    };++    /**+     * A default constructor initializing the lock.+     */+    OptimisticReadWriteLock() = default;++    /**+     * Starts a read phase, making sure that there is currently no+     * active concurrent modification going on. The resulting lease+     * enables the invoking process to later-on verify that no+     * concurrent modifications took place.+     */+    Lease start_read() {+        detail::Waiter wait;++        // get a snapshot of the lease version+        auto v = version.load(std::memory_order_acquire);++        // spin while there is a write in progress+        while ((v & 0x1) == 1) {+            // wait for a moment+            wait();+            // get an updated version+            v = version.load(std::memory_order_acquire);+        }++        // done+        return Lease(v);+    }++    /**+     * Tests whether there have been concurrent modifications since+     * the given lease has been issued.+     *+     * @return true if no updates have been conducted, false otherwise+     */+    bool validate(const Lease& lease) {+        // check whether version number has changed in the mean-while+        std::atomic_thread_fence(std::memory_order_acquire);+        return lease.version == version.load(std::memory_order_relaxed);+    }++    /**+     * Ends a read phase by validating the given lease.+     *+     * @return true if no updates have been conducted since the+     *         issuing of the lease, false otherwise+     */+    bool end_read(const Lease& lease) {+        // check lease in the end+        return validate(lease);+    }++    /**+     * Starts a write phase on this lock be ensuring exclusive access+     * and invalidating any existing read lease.+     */+    void start_write() {+        detail::Waiter wait;++        // set last bit => make it odd+        auto v = version.fetch_or(0x1, std::memory_order_acquire);++        // check for concurrent writes+        while ((v & 0x1) == 1) {+            // wait for a moment+            wait();+            // get an updated version+            v = version.fetch_or(0x1, std::memory_order_acquire);+        }++        // done+    }++    /**+     * Tries to start a write phase unless there is a currently ongoing+     * write operation. In this case no write permission will be obtained.+     *+     * @return true if write permission has been granted, false otherwise.+     */+    bool try_start_write() {+        auto v = version.fetch_or(0x1, std::memory_order_acquire);+        return !(v & 0x1);+    }++    /**+     * Updates a read-lease to a write permission by a) validating that the+     * given lease is still valid and b) making sure that there is no currently+     * ongoing write operation.+     *+     * @return true if the lease was still valid and write permissions could+     *      be granted, false otherwise.+     */+    bool try_upgrade_to_write(const Lease& lease) {+        auto v = version.fetch_or(0x1, std::memory_order_acquire);++        // check whether write privileges have been gained+        if (v & 0x1) return false;  // there is another writer already++        // check whether there was no write since the gain of the read lock+        if (lease.version == v) return true;++        // if there was, undo write update+        abort_write();++        // operation failed+        return false;+    }++    /**+     * Aborts a write operation by reverting to the version number before+     * starting the ongoing write, thereby re-validating existing leases.+     */+    void abort_write() {+        // reset version number+        version.fetch_sub(1, std::memory_order_release);+    }++    /**+     * Ends a write operation by giving up the associated exclusive access+     * to the protected data and abandoning the provided write permission.+     */+    void end_write() {+        // update version number another time+        version.fetch_add(1, std::memory_order_release);+    }++    /**+     * Tests whether currently write permissions have been granted to any+     * client by this lock.+     *+     * @return true if so, false otherwise+     */+    bool is_write_locked() const {+        return version & 0x1;+    }+};++#else++namespace souffle {++/**+ * A small utility class for implementing simple locks.+ */+struct Lock {+    class Lease {};++    // no locking if there is no parallel execution+    Lease acquire() {+        return Lease();+    }++    void lock() {}++    bool try_lock() {+        return true;+    }++    void unlock() {}+};++/**+ * A 'sequential' non-locking implementation for a spin lock.+ */+class SpinLock {+public:+    SpinLock() = default;++    void lock() {}++    bool try_lock() {+        return true;+    }++    void unlock() {}+};++class ReadWriteLock {+public:+    ReadWriteLock() = default;++    void start_read() {}++    void end_read() {}++    void start_write() {}++    bool try_write() {+        return true;+    }++    void end_write() {}++    bool try_upgrade_to_write() {+        return true;+    }++    void downgrade_to_read() {}+};++/**+ * A 'sequential' non-locking implementation for an optimistic r/w lock.+ */+class OptimisticReadWriteLock {+public:+    class Lease {};++    OptimisticReadWriteLock() = default;++    Lease start_read() {+        return Lease();+    }++    bool validate(const Lease& /*lease*/) {+        return true;+    }++    bool end_read(const Lease& /*lease*/) {+        return true;+    }++    void start_write() {}++    bool try_start_write() {+        return true;+    }++    bool try_upgrade_to_write(const Lease& /*lease*/) {+        return true;+    }++    void abort_write() {}++    void end_write() {}++    bool is_write_locked() const {+        return true;+    }+};++#endif++/**+ * Obtains a reference to the lock synchronizing output operations.+ */+inline Lock& getOutputLock() {+    static Lock outputLock;+    return outputLock;+}++}  // end of namespace souffle
+ cbits/souffle/utility/StreamUtil.h view
@@ -0,0 +1,277 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file StreamUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include <map>+#include <memory>+#include <ostream>+#include <set>+#include <string>+#include <type_traits>+#include <utility>+#include <vector>++// -------------------------------------------------------------------------------+//                           General Print Utilities+// -------------------------------------------------------------------------------++namespace souffle {++template <typename A>+struct IsPtrLike : std::is_pointer<A> {};+template <typename A>+struct IsPtrLike<std::unique_ptr<A>> : std::true_type {};+template <typename A>+struct IsPtrLike<std::shared_ptr<A>> : std::true_type {};+template <typename A>+struct IsPtrLike<std::weak_ptr<A>> : std::true_type {};++namespace detail {++/**+ * A auxiliary class to be returned by the join function aggregating the information+ * required to print a list of elements as well as the implementation of the printing+ * itself.+ */+template <typename Iter, typename Printer>+class joined_sequence {+    /** The begin of the range to be printed */+    Iter begin;++    /** The end of the range to be printed */+    Iter end;++    /** The seperator to be utilized between elements */+    std::string sep;++    /** A functor printing an element */+    Printer p;++public:+    /** A constructor setting up all fields of this class */+    joined_sequence(const Iter& a, const Iter& b, std::string sep, Printer p)+            : begin(a), end(b), sep(std::move(sep)), p(std::move(p)) {}++    /** The actual print method */+    friend std::ostream& operator<<(std::ostream& out, const joined_sequence& s) {+        auto cur = s.begin;+        if (cur == s.end) {+            return out;+        }++        s.p(out, *cur);+        ++cur;+        for (; cur != s.end; ++cur) {+            out << s.sep;+            s.p(out, *cur);+        }+        return out;+    }+};++/**+ * A generic element printer.+ *+ * @tparam Extractor a functor preparing a given value before being printed.+ */+template <typename Extractor>+struct print {+    template <typename T>+    void operator()(std::ostream& out, const T& value) const {+        // extract element to be printed from the given value and print it+        Extractor ext;+        out << ext(value);+    }+};+}  // namespace detail++/**+ * A functor representing the identity function for a generic type T.+ *+ * @tparam T some arbitrary type+ */+template <typename T>+struct id {+    T& operator()(T& t) const {+        return t;+    }+    const T& operator()(const T& t) const {+        return t;+    }+};++/**+ * A functor dereferencing a given type+ *+ * @tparam T some arbitrary type with an overloaded * operator (deref)+ */+template <typename T>+struct deref {+    auto operator()(T& t) const -> decltype(*t) {+        return *t;+    }+    auto operator()(const T& t) const -> decltype(*t) {+        return *t;+    }+};++/**+ * A functor printing elements after dereferencing it. This functor+ * is mainly intended to be utilized when printing sequences of elements+ * of a pointer type when using the join function below.+ */+template <typename T>+struct print_deref : public detail::print<deref<T>> {};++/**+ * Creates an object to be forwarded to some output stream for printing+ * sequences of elements interspersed by a given separator.+ *+ * For use cases see the test case {util_test.cpp}.+ */+template <typename Iter, typename Printer>+detail::joined_sequence<Iter, Printer> join(+        const Iter& a, const Iter& b, const std::string& sep, const Printer& p) {+    return souffle::detail::joined_sequence<Iter, Printer>(a, b, sep, p);+}++/**+ * Creates an object to be forwarded to some output stream for printing+ * sequences of elements interspersed by a given separator.+ *+ * For use cases see the test case {util_test.cpp}.+ */+template <typename Iter, typename T = typename Iter::value_type>+detail::joined_sequence<Iter, detail::print<id<T>>> join(+        const Iter& a, const Iter& b, const std::string& sep = ",") {+    return join(a, b, sep, detail::print<id<T>>());+}++/**+ * Creates an object to be forwarded to some output stream for printing+ * the content of containers interspersed by a given separator.+ *+ * For use cases see the test case {util_test.cpp}.+ */+template <typename Container, typename Printer, typename Iter = typename Container::const_iterator>+detail::joined_sequence<Iter, Printer> join(const Container& c, const std::string& sep, const Printer& p) {+    return join(c.begin(), c.end(), sep, p);+}++// Decide if the sane default is to deref-then-print or just print.+// Right now, deref anything deref-able *except* for a `const char*` (which handled as a C-string).+template <typename A>+constexpr bool JoinShouldDeref = IsPtrLike<A>::value && !std::is_same_v<A, char const*>;++/**+ * Creates an object to be forwarded to some output stream for printing+ * the content of containers interspersed by a given separator.+ *+ * For use cases see the test case {util_test.cpp}.+ */+template <typename Container, typename Iter = typename Container::const_iterator,+        typename T = typename Iter::value_type>+std::enable_if_t<!JoinShouldDeref<T>, detail::joined_sequence<Iter, detail::print<id<T>>>> join(+        const Container& c, const std::string& sep = ",") {+    return join(c.begin(), c.end(), sep, detail::print<id<T>>());+}++template <typename Container, typename Iter = typename Container::const_iterator,+        typename T = typename Iter::value_type>+std::enable_if_t<JoinShouldDeref<T>, detail::joined_sequence<Iter, detail::print<deref<T>>>> join(+        const Container& c, const std::string& sep = ",") {+    return join(c.begin(), c.end(), sep, detail::print<deref<T>>());+}++}  // end namespace souffle++#ifndef __EMBEDDED_SOUFFLE__++namespace std {++/**+ * Introduces support for printing pairs as long as their components can be printed.+ */+template <typename A, typename B>+ostream& operator<<(ostream& out, const pair<A, B>& p) {+    return out << "(" << p.first << "," << p.second << ")";+}++/**+ * Enables the generic printing of vectors assuming their element types+ * are printable.+ */+template <typename T, typename A>+ostream& operator<<(ostream& out, const vector<T, A>& v) {+    return out << "[" << souffle::join(v) << "]";+}++/**+ * Enables the generic printing of sets assuming their element types+ * are printable.+ */+template <typename K, typename C, typename A>+ostream& operator<<(ostream& out, const set<K, C, A>& s) {+    return out << "{" << souffle::join(s) << "}";+}++/**+ * Enables the generic printing of maps assuming their element types+ * are printable.+ */+template <typename K, typename T, typename C, typename A>+ostream& operator<<(ostream& out, const map<K, T, C, A>& m) {+    return out << "{" << souffle::join(m, ",", [](ostream& out, const pair<K, T>& cur) {+        out << cur.first << "->" << cur.second;+    }) << "}";+}++}  // end namespace std++#endif++namespace souffle {++namespace detail {++/**+ * A utility class required for the implementation of the times function.+ */+template <typename T>+struct multiplying_printer {+    const T& value;+    unsigned times;+    multiplying_printer(const T& value, unsigned times) : value(value), times(times) {}++    friend std::ostream& operator<<(std::ostream& out, const multiplying_printer& printer) {+        for (unsigned i = 0; i < printer.times; i++) {+            out << printer.value;+        }+        return out;+    }+};+}  // namespace detail++/**+ * A utility printing a given value multiple times.+ */+template <typename T>+detail::multiplying_printer<T> times(const T& value, unsigned num) {+    return detail::multiplying_printer<T>(value, num);+}++}  // namespace souffle
+ cbits/souffle/utility/StringUtil.h view
@@ -0,0 +1,430 @@+/*+ * Souffle - A Datalog Compiler+ * Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved+ * Licensed under the Universal Permissive License v 1.0 as shown at:+ * - https://opensource.org/licenses/UPL+ * - <souffle root>/licenses/SOUFFLE-UPL.txt+ */++/************************************************************************+ *+ * @file StringUtil.h+ *+ * @brief Datalog project utilities+ *+ ***********************************************************************/++#pragma once++#include "../RamTypes.h"+#include <algorithm>+#include <cctype>+#include <cstdlib>+#include <fstream>+#include <limits>+#include <sstream>+#include <stdexcept>+#include <string>+#include <type_traits>+#include <typeinfo>+#include <vector>++namespace souffle {++// Forward declaration+inline bool isPrefix(const std::string& prefix, const std::string& element);++/**+ * Converts a string to a RamSigned+ *+ * This procedure has similar behaviour to std::stoi/stoll.+ *+ * The procedure accepts prefixes 0b (if base = 2) and 0x (if base = 16)+ * If base = 0, the procedure will try to infer the base from the prefix, if present.+ */+inline RamSigned RamSignedFromString(+        const std::string& str, std::size_t* position = nullptr, const int base = 10) {+    RamSigned val;++    if (base == 0) {+        if (isPrefix("-0b", str) || isPrefix("0b", str)) {+            return RamSignedFromString(str, position, 2);+        } else if (isPrefix("-0x", str) || isPrefix("0x", str)) {+            return RamSignedFromString(str, position, 16);+        } else {+            return RamSignedFromString(str, position);+        }+    }+    std::string binaryNumber;+    bool parsingBinary = base == 2;++    // stoi/stoll can't handle base 2 prefix by default.+    if (parsingBinary) {+        if (isPrefix("-0b", str)) {+            binaryNumber = "-" + str.substr(3);+        } else if (isPrefix("0b", str)) {+            binaryNumber = str.substr(2);+        }+    }+    const std::string& tmp = parsingBinary ? binaryNumber : str;++#if RAM_DOMAIN_SIZE == 64+    val = std::stoll(tmp, position, base);+#else+    val = std::stoi(tmp, position, base);+#endif++    if (parsingBinary && position != nullptr) {+        *position += 2;+    }++    return val;+}++/**+ * Converts a string to a RamFloat+ */+inline RamFloat RamFloatFromString(const std::string& str, std::size_t* position = nullptr) {+    RamFloat val;+#if RAM_DOMAIN_SIZE == 64+    val = std::stod(str, position);+#else+    val = std::stof(str, position);+#endif+    return static_cast<RamFloat>(val);+}+/**+ * Converts a string to a RamUnsigned+ *+ * This procedure has similar behaviour to std::stoul/stoull.+ *+ * The procedure accepts prefixes 0b (if base = 2) and 0x (if base = 16)+ * If base = 0, the procedure will try to infer the base from the prefix, if present.+ */+inline RamUnsigned RamUnsignedFromString(+        const std::string& str, std::size_t* position = nullptr, const int base = 10) {+    // Be default C++ (stoul) allows unsigned numbers starting with "-".+    if (isPrefix("-", str)) {+        throw std::invalid_argument("Unsigned number can't start with minus.");+    }++    if (base == 0) {+        if (isPrefix("0b", str)) {+            return RamUnsignedFromString(str, position, 2);+        } else if (isPrefix("0x", str)) {+            return RamUnsignedFromString(str, position, 16);+        } else {+            return RamUnsignedFromString(str, position);+        }+    }++    // stoul/stoull can't handle binary prefix by default.+    std::string binaryNumber;+    bool parsingBinary = false;+    if (base == 2 && isPrefix("0b", str)) {+        binaryNumber = str.substr(2);+        parsingBinary = true;+    }+    const std::string& tmp = parsingBinary ? binaryNumber : str;++    RamUnsigned val;+#if RAM_DOMAIN_SIZE == 64+    val = std::stoull(tmp, position, base);+#else+    val = std::stoul(tmp, position, base);+#endif++    if (parsingBinary && position != nullptr) {+        *position += 2;+    }++    // check if it's safe to cast (stoul returns unsigned long)+    if (val > std::numeric_limits<RamUnsigned>::max()) {+        throw std::invalid_argument("Unsigned number of of bounds");+    }++    return static_cast<RamUnsigned>(val);+}++/**+ * Can a string be parsed as RamSigned.+ *+ * Souffle (parser, not fact file readers) accepts: hex, binary and base 10.+ * Integer can be negative, in all 3 formats this means that it+ * starts with minus (c++ default semantics).+ */+inline bool canBeParsedAsRamSigned(const std::string& string) {+    size_t charactersRead = 0;++    try {+        RamSignedFromString(string, &charactersRead, 0);+    } catch (...) {+        return false;+    }++    return charactersRead == string.size();+}++/**+ * Can a string be parsed as RamUnsigned.+ *+ * Souffle accepts: hex, binary and base 10.+ */+inline bool canBeParsedAsRamUnsigned(const std::string& string) {+    size_t charactersRead = 0;+    try {+        RamUnsignedFromString(string, &charactersRead, 0);+    } catch (...) {+        return false;+    }+    return charactersRead == string.size();+}++/**+ * Can a string be parsed as RamFloat.+ */+inline bool canBeParsedAsRamFloat(const std::string& string) {+    size_t charactersRead = 0;+    try {+        RamFloatFromString(string, &charactersRead);+    } catch (...) {+        return false;+    }+    return charactersRead == string.size();+}++#if RAM_DOMAIN_SIZE == 64+inline RamDomain stord(const std::string& str, std::size_t* pos = nullptr, int base = 10) {+    return static_cast<RamDomain>(std::stoull(str, pos, base));+}+#elif RAM_DOMAIN_SIZE == 32+inline RamDomain stord(const std::string& str, std::size_t* pos = nullptr, int base = 10) {+    return static_cast<RamDomain>(std::stoul(str, pos, base));+}+#else+#error RAM Domain is neither 32bit nor 64bit+#endif++/**+ * Check whether a string is a sequence of digits+ */+inline bool isNumber(const char* str) {+    if (str == nullptr) {+        return false;+    }++    while (*str != 0) {+        if (isdigit(*str) == 0) {+            return false;+        }+        str++;+    }+    return true;+}++/**+ * A generic function converting strings into strings (trivial case).+ */+inline const std::string& toString(const std::string& str) {+    return str;+}++namespace detail {++/**+ * A type trait to check whether a given type is printable.+ * In this general case, nothing is printable.+ */+template <typename T, typename filter = void>+struct is_printable : public std::false_type {};++/**+ * A type trait to check whether a given type is printable.+ * This specialization makes types with an output operator printable.+ */+template <typename T>+struct is_printable<T, typename std::conditional<false,+                               decltype(std::declval<std::ostream&>() << std::declval<T>()), void>::type>+        : public std::true_type {};+}  // namespace detail++/**+ * A generic function converting arbitrary objects to strings by utilizing+ * their print capability.+ *+ * This function is mainly intended for implementing test cases and debugging+ * operations.+ */+template <typename T>+typename std::enable_if<detail::is_printable<T>::value, std::string>::type toString(const T& value) {+    // write value into stream and return result+    std::stringstream ss;+    ss << value;+    return ss.str();+}++/**+ * A fallback for the to-string function in case an unprintable object is supposed+ * to be printed.+ */+template <typename T>+typename std::enable_if<!detail::is_printable<T>::value, std::string>::type toString(const T&) {+    std::stringstream ss;+    ss << "(print for type ";+    ss << typeid(T).name();+    ss << " not supported)";+    return ss.str();+}++// -------------------------------------------------------------------------------+//                              String Utils+// -------------------------------------------------------------------------------++/**+ * Determine if one string is a prefix of another+ */+inline bool isPrefix(const std::string& prefix, const std::string& element) {+    auto itPrefix = prefix.begin();+    auto itElement = element.begin();++    while (itPrefix != prefix.end() && itElement != element.end()) {+        if (*itPrefix != *itElement) {+            break;+        }+        ++itPrefix;+        ++itElement;+    }++    return itPrefix == prefix.end();+}++/**+ * Determines whether the given value string ends with the given+ * end string.+ */+inline bool endsWith(const std::string& value, const std::string& ending) {+    if (value.size() < ending.size()) {+        return false;+    }+    return std::equal(ending.rbegin(), ending.rend(), value.rbegin());+}++/**+ * Splits a string given a delimiter+ */+inline std::vector<std::string> splitString(const std::string& str, char delimiter) {+    std::vector<std::string> parts;+    std::stringstream strstr(str);+    std::string token;+    while (std::getline(strstr, token, delimiter)) {+        parts.push_back(token);+    }+    return parts;+}++/**+ * Stringify a string using escapes for escape, newline, tab, double-quotes and semicolons+ */+inline std::string stringify(const std::string& input) {+    std::string str(input);++    // replace escapes with double escape sequence+    size_t start_pos = 0;+    while ((start_pos = str.find('\\', start_pos)) != std::string::npos) {+        str.replace(start_pos, 1, "\\\\");+        start_pos += 2;+    }+    // replace semicolons with escape sequence+    start_pos = 0;+    while ((start_pos = str.find(';', start_pos)) != std::string::npos) {+        str.replace(start_pos, 1, "\\;");+        start_pos += 2;+    }+    // replace double-quotes with escape sequence+    start_pos = 0;+    while ((start_pos = str.find('"', start_pos)) != std::string::npos) {+        str.replace(start_pos, 1, "\\\"");+        start_pos += 2;+    }+    // replace newline with escape sequence+    start_pos = 0;+    while ((start_pos = str.find('\n', start_pos)) != std::string::npos) {+        str.replace(start_pos, 1, "\\n");+        start_pos += 2;+    }+    // replace tab with escape sequence+    start_pos = 0;+    while ((start_pos = str.find('\t', start_pos)) != std::string::npos) {+        str.replace(start_pos, 1, "\\t");+        start_pos += 2;+    }+    return str;+}++/**+ * Escape JSON string.+ */+inline std::string escapeJSONstring(const std::string& JSONstr) {+    std::ostringstream destination;++    // Iterate over all characters except first and last+    for (char c : JSONstr) {+        if (c == '\"') {+            destination << "\\";+        }+        destination << c;+    }+    return destination.str();+}++/** Valid C++ identifier, note that this does not ensure the uniqueness of identifiers returned. */+inline std::string identifier(std::string id) {+    for (size_t i = 0; i < id.length(); i++) {+        if (((isalpha(id[i]) == 0) && i == 0) || ((isalnum(id[i]) == 0) && id[i] != '_')) {+            id[i] = '_';+        }+    }+    return id;+}++// TODO (b-scholz): tidy up unescape/escape functions++inline std::string unescape(+        const std::string& inputString, const std::string& needle, const std::string& replacement) {+    std::string result = inputString;+    size_t pos = 0;+    while ((pos = result.find(needle, pos)) != std::string::npos) {+        result = result.replace(pos, needle.length(), replacement);+        pos += replacement.length();+    }+    return result;+}++inline std::string unescape(const std::string& inputString) {+    std::string unescaped = unescape(inputString, "\\\"", "\"");+    unescaped = unescape(unescaped, "\\t", "\t");+    unescaped = unescape(unescaped, "\\r", "\r");+    unescaped = unescape(unescaped, "\\n", "\n");+    return unescaped;+}++inline std::string escape(+        const std::string& inputString, const std::string& needle, const std::string& replacement) {+    std::string result = inputString;+    size_t pos = 0;+    while ((pos = result.find(needle, pos)) != std::string::npos) {+        result = result.replace(pos, needle.length(), replacement);+        pos += replacement.length();+    }+    return result;+}++inline std::string escape(const std::string& inputString) {+    std::string escaped = escape(inputString, "\"", "\\\"");+    escaped = escape(escaped, "\t", "\\t");+    escaped = escape(escaped, "\r", "\\r");+    escaped = escape(escaped, "\n", "\\n");+    return escaped;+}++}  // end namespace souffle
+ cbits/souffle/utility/tinyformat.h view
@@ -0,0 +1,1147 @@+// clang-format off+// Release: 2.3.0++// tinyformat.h+// Copyright (C) 2011, Chris Foster [chris42f (at) gmail (d0t) com]+//+// Boost Software License - Version 1.0+//+// Permission is hereby granted, free of charge, to any person or organization+// obtaining a copy of the software and accompanying documentation covered by+// this license (the "Software") to use, reproduce, display, distribute,+// execute, and transmit the Software, and to prepare derivative works of the+// Software, and to permit third-parties to whom the Software is furnished to+// do so, all subject to the following:+//+// The copyright notices in the Software and this entire statement, including+// the above license grant, this restriction and the following disclaimer,+// must be included in all copies of the Software, in whole or in part, and+// all derivative works of the Software, unless such copies or derivative+// works are solely in the form of machine-executable object code generated by+// a source language processor.+//+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+// FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT+// SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE+// FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,+// ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER+// DEALINGS IN THE SOFTWARE.++//------------------------------------------------------------------------------+// Tinyformat: A minimal type safe printf replacement+//+// tinyformat.h is a type safe printf replacement library in a single C+++// header file.  Design goals include:+//+// * Type safety and extensibility for user defined types.+// * C99 printf() compatibility, to the extent possible using std::ostream+// * POSIX extension for positional arguments+// * Simplicity and minimalism.  A single header file to include and distribute+//   with your projects.+// * Augment rather than replace the standard stream formatting mechanism+// * C++98 support, with optional C++11 niceties+//+//+// Main interface example usage+// ----------------------------+//+// To print a date to std::cout for American usage:+//+//   std::string weekday = "Wednesday";+//   const char* month = "July";+//   size_t day = 27;+//   long hour = 14;+//   int min = 44;+//+//   tfm::printf("%s, %s %d, %.2d:%.2d\n", weekday, month, day, hour, min);+//+// POSIX extension for positional arguments is available.+// The ability to rearrange formatting arguments is an important feature+// for localization because the word order may vary in different languages.+//+// Previous example for German usage. Arguments are reordered:+//+//   tfm::printf("%1$s, %3$d. %2$s, %4$d:%5$.2d\n", weekday, month, day, hour, min);+//+// The strange types here emphasize the type safety of the interface; it is+// possible to print a std::string using the "%s" conversion, and a+// size_t using the "%d" conversion.  A similar result could be achieved+// using either of the tfm::format() functions.  One prints on a user provided+// stream:+//+//   tfm::format(std::cerr, "%s, %s %d, %.2d:%.2d\n",+//               weekday, month, day, hour, min);+//+// The other returns a std::string:+//+//   std::string date = tfm::format("%s, %s %d, %.2d:%.2d\n",+//                                  weekday, month, day, hour, min);+//   std::cout << date;+//+// These are the three primary interface functions.  There is also a+// convenience function printfln() which appends a newline to the usual result+// of printf() for super simple logging.+//+//+// User defined format functions+// -----------------------------+//+// Simulating variadic templates in C++98 is pretty painful since it requires+// writing out the same function for each desired number of arguments.  To make+// this bearable tinyformat comes with a set of macros which are used+// internally to generate the API, but which may also be used in user code.+//+// The three macros TINYFORMAT_ARGTYPES(n), TINYFORMAT_VARARGS(n) and+// TINYFORMAT_PASSARGS(n) will generate a list of n argument types,+// type/name pairs and argument names respectively when called with an integer+// n between 1 and 16.  We can use these to define a macro which generates the+// desired user defined function with n arguments.  To generate all 16 user+// defined function bodies, use the macro TINYFORMAT_FOREACH_ARGNUM.  For an+// example, see the implementation of printf() at the end of the source file.+//+// Sometimes it's useful to be able to pass a list of format arguments through+// to a non-template function.  The FormatList class is provided as a way to do+// this by storing the argument list in a type-opaque way.  Continuing the+// example from above, we construct a FormatList using makeFormatList():+//+//   FormatListRef formatList = tfm::makeFormatList(weekday, month, day, hour, min);+//+// The format list can now be passed into any non-template function and used+// via a call to the vformat() function:+//+//   tfm::vformat(std::cout, "%s, %s %d, %.2d:%.2d\n", formatList);+//+//+// Additional API information+// --------------------------+//+// Error handling: Define TINYFORMAT_ERROR to customize the error handling for+// format strings which are unsupported or have the wrong number of format+// specifiers (calls assert() by default).+//+// User defined types: Uses operator<< for user defined types by default.+// Overload formatValue() for more control.+++#ifndef TINYFORMAT_H_INCLUDED+#define TINYFORMAT_H_INCLUDED++namespace tinyformat {}+//------------------------------------------------------------------------------+// Config section.  Customize to your liking!++// Namespace alias to encourage brevity+namespace tfm = tinyformat;++// Error handling; calls assert() by default.+// #define TINYFORMAT_ERROR(reasonString) your_error_handler(reasonString)++// Define for C++11 variadic templates which make the code shorter & more+// general.  If you don't define this, C++11 support is autodetected below.+// #define TINYFORMAT_USE_VARIADIC_TEMPLATES+++//------------------------------------------------------------------------------+// Implementation details.+#include <algorithm>+#include <iostream>+#include <sstream>+#include <string>++#ifndef TINYFORMAT_ASSERT+#   include <cassert>+#   define TINYFORMAT_ASSERT(cond) assert(cond)+#endif++#ifndef TINYFORMAT_ERROR+#   include <cassert>+#   define TINYFORMAT_ERROR(reason) assert(0 && reason)+#endif++#if !defined(TINYFORMAT_USE_VARIADIC_TEMPLATES) && !defined(TINYFORMAT_NO_VARIADIC_TEMPLATES)+#   ifdef __GXX_EXPERIMENTAL_CXX0X__+#       define TINYFORMAT_USE_VARIADIC_TEMPLATES+#   endif+#endif++#if defined(__GLIBCXX__) && __GLIBCXX__ < 20080201+//  std::showpos is broken on old libstdc++ as provided with macOS.  See+//  http://gcc.gnu.org/ml/libstdc++/2007-11/msg00075.html+#   define TINYFORMAT_OLD_LIBSTDCPLUSPLUS_WORKAROUND+#endif++#ifdef __APPLE__+// Workaround macOS linker warning: Xcode uses different default symbol+// visibilities for static libs vs executables (see issue #25)+#   define TINYFORMAT_HIDDEN __attribute__((visibility("hidden")))+#else+#   define TINYFORMAT_HIDDEN+#endif++namespace tinyformat {++//------------------------------------------------------------------------------+namespace detail {++// Test whether type T1 is convertible to type T2+template <typename T1, typename T2>+struct is_convertible+{+    private:+        // two types of different size+        struct fail { char dummy[2]; };+        struct succeed { char dummy; };+        // Try to convert a T1 to a T2 by plugging into tryConvert+        static fail tryConvert(...);+        static succeed tryConvert(const T2&);+        static const T1& makeT1();+    public:+#       ifdef _MSC_VER+        // Disable spurious loss of precision warnings in tryConvert(makeT1())+#       pragma warning(push)+#       pragma warning(disable:4244)+#       pragma warning(disable:4267)+#       endif+        // Standard trick: the (...) version of tryConvert will be chosen from+        // the overload set only if the version taking a T2 doesn't match.+        // Then we compare the sizes of the return types to check which+        // function matched.  Very neat, in a disgusting kind of way :)+        static const bool value =+            sizeof(tryConvert(makeT1())) == sizeof(succeed);+#       ifdef _MSC_VER+#       pragma warning(pop)+#       endif+};+++// Detect when a type is not a wchar_t string+template<typename T> struct is_wchar { using tinyformat_wchar_is_not_supported = int; };+template<> struct is_wchar<wchar_t*> {};+template<> struct is_wchar<const wchar_t*> {};+template<int n> struct is_wchar<const wchar_t[n]> {};+template<int n> struct is_wchar<wchar_t[n]> {};+++// Format the value by casting to type fmtT.  This default implementation+// should never be called.+template<typename T, typename fmtT, bool convertible = is_convertible<T, fmtT>::value>+struct formatValueAsType+{+    static void invoke(std::ostream& /*out*/, const T& /*value*/) { TINYFORMAT_ASSERT(0); }+};+// Specialized version for types that can actually be converted to fmtT, as+// indicated by the "convertible" template parameter.+template<typename T, typename fmtT>+struct formatValueAsType<T,fmtT,true>+{+    static void invoke(std::ostream& out, const T& value)+        { out << static_cast<fmtT>(value); }+};++#ifdef TINYFORMAT_OLD_LIBSTDCPLUSPLUS_WORKAROUND+template<typename T, bool convertible = is_convertible<T, int>::value>+struct formatZeroIntegerWorkaround+{+    static bool invoke(std::ostream& /**/, const T& /**/) { return false; }+};+template<typename T>+struct formatZeroIntegerWorkaround<T,true>+{+    static bool invoke(std::ostream& out, const T& value)+    {+        if (static_cast<int>(value) == 0 && out.flags() & std::ios::showpos) {+            out << "+0";+            return true;+        }+        return false;+    }+};+#endif // TINYFORMAT_OLD_LIBSTDCPLUSPLUS_WORKAROUND++// Convert an arbitrary type to integer.  The version with convertible=false+// throws an error.+template<typename T, bool convertible = is_convertible<T,int>::value>+struct convertToInt+{+    static int invoke(const T& /*value*/)+    {+        TINYFORMAT_ERROR("tinyformat: Cannot convert from argument type to "+                         "integer for use as variable width or precision");+        return 0;+    }+};+// Specialization for convertToInt when conversion is possible+template<typename T>+struct convertToInt<T,true>+{+    static int invoke(const T& value) { return static_cast<int>(value); }+};++// Format at most ntrunc characters to the given stream.+template<typename T>+inline void formatTruncated(std::ostream& out, const T& value, int ntrunc)+{+    std::ostringstream tmp;+    tmp << value;+    std::string result = tmp.str();+    out.write(result.c_str(), (std::min)(ntrunc, static_cast<int>(result.size())));+}+#define TINYFORMAT_DEFINE_FORMAT_TRUNCATED_CSTR(type)       \+inline void formatTruncated(std::ostream& out, type* value, int ntrunc) \+{                                                           \+    std::streamsize len = 0;                                \+    while (len < ntrunc && value[len] != 0)                 \+        ++len;                                              \+    out.write(value, len);                                  \+}+// Overload for const char* and char*.  Could overload for signed & unsigned+// char too, but these are technically unneeded for printf compatibility.+TINYFORMAT_DEFINE_FORMAT_TRUNCATED_CSTR(const char)+TINYFORMAT_DEFINE_FORMAT_TRUNCATED_CSTR(char)+#undef TINYFORMAT_DEFINE_FORMAT_TRUNCATED_CSTR++} // namespace detail+++//------------------------------------------------------------------------------+// Variable formatting functions.  May be overridden for user-defined types if+// desired.+++/// Format a value into a stream, delegating to operator<< by default.+///+/// Users may override this for their own types.  When this function is called,+/// the stream flags will have been modified according to the format string.+/// The format specification is provided in the range [fmtBegin, fmtEnd).  For+/// truncating conversions, ntrunc is set to the desired maximum number of+/// characters, for example "%.7s" calls formatValue with ntrunc = 7.+///+/// By default, formatValue() uses the usual stream insertion operator+/// operator<< to format the type T, with special cases for the %c and %p+/// conversions.+template<typename T>+inline void formatValue(std::ostream& out, const char* /*fmtBegin*/,+                        const char* fmtEnd, int ntrunc, const T& value)+{+#ifndef TINYFORMAT_ALLOW_WCHAR_STRINGS+    // Since we don't support printing of wchar_t using "%ls", make it fail at+    // compile time in preference to printing as a void* at runtime.+    using DummyType = typename detail::is_wchar<T>::tinyformat_wchar_is_not_supported;+    (void) DummyType(); // avoid unused type warning with gcc-4.8+#endif+    // The mess here is to support the %c and %p conversions: if these+    // conversions are active we try to convert the type to a char or const+    // void* respectively and format that instead of the value itself.  For the+    // %p conversion it's important to avoid dereferencing the pointer, which+    // could otherwise lead to a crash when printing a dangling (const char*).+    const bool canConvertToChar = detail::is_convertible<T,char>::value;+    const bool canConvertToVoidPtr = detail::is_convertible<T, const void*>::value;+    if (canConvertToChar && *(fmtEnd-1) == 'c')+        detail::formatValueAsType<T, char>::invoke(out, value);+    else if (canConvertToVoidPtr && *(fmtEnd-1) == 'p')+        detail::formatValueAsType<T, const void*>::invoke(out, value);+#ifdef TINYFORMAT_OLD_LIBSTDCPLUSPLUS_WORKAROUND+    else if (detail::formatZeroIntegerWorkaround<T>::invoke(out, value)) /**/;+#endif+    else if (ntrunc >= 0) {+        // Take care not to overread C strings in truncating conversions like+        // "%.4s" where at most 4 characters may be read.+        detail::formatTruncated(out, value, ntrunc);+    }+    else+        out << value;+}+++// Overloaded version for char types to support printing as an integer+#define TINYFORMAT_DEFINE_FORMATVALUE_CHAR(charType)                  \+inline void formatValue(std::ostream& out, const char* /*fmtBegin*/,  \+                        const char* fmtEnd, int /**/, charType value) \+{                                                                     \+    switch (*(fmtEnd-1)) {                                            \+        case 'u': case 'd': case 'i': case 'o': case 'X': case 'x':   \+            out << static_cast<int>(value); break;                    \+        default:                                                      \+            out << value;                   break;                    \+    }                                                                 \+}+// per 3.9.1: char, signed char and unsigned char are all distinct types+TINYFORMAT_DEFINE_FORMATVALUE_CHAR(char)+TINYFORMAT_DEFINE_FORMATVALUE_CHAR(signed char)+TINYFORMAT_DEFINE_FORMATVALUE_CHAR(unsigned char)+#undef TINYFORMAT_DEFINE_FORMATVALUE_CHAR+++//------------------------------------------------------------------------------+// Tools for emulating variadic templates in C++98.  The basic idea here is+// stolen from the boost preprocessor metaprogramming library and cut down to+// be just general enough for what we need.++#define TINYFORMAT_ARGTYPES(n) TINYFORMAT_ARGTYPES_ ## n+#define TINYFORMAT_VARARGS(n) TINYFORMAT_VARARGS_ ## n+#define TINYFORMAT_PASSARGS(n) TINYFORMAT_PASSARGS_ ## n+#define TINYFORMAT_PASSARGS_TAIL(n) TINYFORMAT_PASSARGS_TAIL_ ## n++// To keep it as transparent as possible, the macros below have been generated+// using python via the excellent cog.py code generation script.  This avoids+// the need for a bunch of complex (but more general) preprocessor tricks as+// used in boost.preprocessor.+//+// To rerun the code generation in place, use `cog.py -r tinyformat.h`+// (see http://nedbatchelder.com/code/cog).  Alternatively you can just create+// extra versions by hand.++/*[[[cog+maxParams = 16++def makeCommaSepLists(lineTemplate, elemTemplate, startInd=1):+    for j in range(startInd,maxParams+1):+        list = ', '.join([elemTemplate % {'i':i} for i in range(startInd,j+1)])+        cog.outl(lineTemplate % {'j':j, 'list':list})++makeCommaSepLists('#define TINYFORMAT_ARGTYPES_%(j)d %(list)s',+                  'class T%(i)d')++cog.outl()+makeCommaSepLists('#define TINYFORMAT_VARARGS_%(j)d %(list)s',+                  'const T%(i)d& v%(i)d')++cog.outl()+makeCommaSepLists('#define TINYFORMAT_PASSARGS_%(j)d %(list)s', 'v%(i)d')++cog.outl()+cog.outl('#define TINYFORMAT_PASSARGS_TAIL_1')+makeCommaSepLists('#define TINYFORMAT_PASSARGS_TAIL_%(j)d , %(list)s',+                  'v%(i)d', startInd = 2)++cog.outl()+cog.outl('#define TINYFORMAT_FOREACH_ARGNUM(m) \\\n    ' ++         ' '.join(['m(%d)' % (j,) for j in range(1,maxParams+1)]))+]]]*/+#define TINYFORMAT_ARGTYPES_1 class T1+#define TINYFORMAT_ARGTYPES_2 class T1, class T2+#define TINYFORMAT_ARGTYPES_3 class T1, class T2, class T3+#define TINYFORMAT_ARGTYPES_4 class T1, class T2, class T3, class T4+#define TINYFORMAT_ARGTYPES_5 class T1, class T2, class T3, class T4, class T5+#define TINYFORMAT_ARGTYPES_6 class T1, class T2, class T3, class T4, class T5, class T6+#define TINYFORMAT_ARGTYPES_7 class T1, class T2, class T3, class T4, class T5, class T6, class T7+#define TINYFORMAT_ARGTYPES_8 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8+#define TINYFORMAT_ARGTYPES_9 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9+#define TINYFORMAT_ARGTYPES_10 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10+#define TINYFORMAT_ARGTYPES_11 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11+#define TINYFORMAT_ARGTYPES_12 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11, class T12+#define TINYFORMAT_ARGTYPES_13 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11, class T12, class T13+#define TINYFORMAT_ARGTYPES_14 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11, class T12, class T13, class T14+#define TINYFORMAT_ARGTYPES_15 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11, class T12, class T13, class T14, class T15+#define TINYFORMAT_ARGTYPES_16 class T1, class T2, class T3, class T4, class T5, class T6, class T7, class T8, class T9, class T10, class T11, class T12, class T13, class T14, class T15, class T16++#define TINYFORMAT_VARARGS_1 const T1& v1+#define TINYFORMAT_VARARGS_2 const T1& v1, const T2& v2+#define TINYFORMAT_VARARGS_3 const T1& v1, const T2& v2, const T3& v3+#define TINYFORMAT_VARARGS_4 const T1& v1, const T2& v2, const T3& v3, const T4& v4+#define TINYFORMAT_VARARGS_5 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5+#define TINYFORMAT_VARARGS_6 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6+#define TINYFORMAT_VARARGS_7 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7+#define TINYFORMAT_VARARGS_8 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8+#define TINYFORMAT_VARARGS_9 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9+#define TINYFORMAT_VARARGS_10 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10+#define TINYFORMAT_VARARGS_11 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11+#define TINYFORMAT_VARARGS_12 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11, const T12& v12+#define TINYFORMAT_VARARGS_13 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11, const T12& v12, const T13& v13+#define TINYFORMAT_VARARGS_14 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11, const T12& v12, const T13& v13, const T14& v14+#define TINYFORMAT_VARARGS_15 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11, const T12& v12, const T13& v13, const T14& v14, const T15& v15+#define TINYFORMAT_VARARGS_16 const T1& v1, const T2& v2, const T3& v3, const T4& v4, const T5& v5, const T6& v6, const T7& v7, const T8& v8, const T9& v9, const T10& v10, const T11& v11, const T12& v12, const T13& v13, const T14& v14, const T15& v15, const T16& v16++#define TINYFORMAT_PASSARGS_1 v1+#define TINYFORMAT_PASSARGS_2 v1, v2+#define TINYFORMAT_PASSARGS_3 v1, v2, v3+#define TINYFORMAT_PASSARGS_4 v1, v2, v3, v4+#define TINYFORMAT_PASSARGS_5 v1, v2, v3, v4, v5+#define TINYFORMAT_PASSARGS_6 v1, v2, v3, v4, v5, v6+#define TINYFORMAT_PASSARGS_7 v1, v2, v3, v4, v5, v6, v7+#define TINYFORMAT_PASSARGS_8 v1, v2, v3, v4, v5, v6, v7, v8+#define TINYFORMAT_PASSARGS_9 v1, v2, v3, v4, v5, v6, v7, v8, v9+#define TINYFORMAT_PASSARGS_10 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10+#define TINYFORMAT_PASSARGS_11 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11+#define TINYFORMAT_PASSARGS_12 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12+#define TINYFORMAT_PASSARGS_13 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13+#define TINYFORMAT_PASSARGS_14 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14+#define TINYFORMAT_PASSARGS_15 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15+#define TINYFORMAT_PASSARGS_16 v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16++#define TINYFORMAT_PASSARGS_TAIL_1+#define TINYFORMAT_PASSARGS_TAIL_2 , v2+#define TINYFORMAT_PASSARGS_TAIL_3 , v2, v3+#define TINYFORMAT_PASSARGS_TAIL_4 , v2, v3, v4+#define TINYFORMAT_PASSARGS_TAIL_5 , v2, v3, v4, v5+#define TINYFORMAT_PASSARGS_TAIL_6 , v2, v3, v4, v5, v6+#define TINYFORMAT_PASSARGS_TAIL_7 , v2, v3, v4, v5, v6, v7+#define TINYFORMAT_PASSARGS_TAIL_8 , v2, v3, v4, v5, v6, v7, v8+#define TINYFORMAT_PASSARGS_TAIL_9 , v2, v3, v4, v5, v6, v7, v8, v9+#define TINYFORMAT_PASSARGS_TAIL_10 , v2, v3, v4, v5, v6, v7, v8, v9, v10+#define TINYFORMAT_PASSARGS_TAIL_11 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11+#define TINYFORMAT_PASSARGS_TAIL_12 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12+#define TINYFORMAT_PASSARGS_TAIL_13 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13+#define TINYFORMAT_PASSARGS_TAIL_14 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14+#define TINYFORMAT_PASSARGS_TAIL_15 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15+#define TINYFORMAT_PASSARGS_TAIL_16 , v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16++#define TINYFORMAT_FOREACH_ARGNUM(m) \+    m(1) m(2) m(3) m(4) m(5) m(6) m(7) m(8) m(9) m(10) m(11) m(12) m(13) m(14) m(15) m(16)+//[[[end]]]++++namespace detail {++// Type-opaque holder for an argument to format(), with associated actions on+// the type held as explicit function pointers.  This allows FormatArg's for+// each argument to be allocated as a homogeneous array inside FormatList+// whereas a naive implementation based on inheritance does not.+class FormatArg+{+    public:+        FormatArg() = default;++        template<typename T>+        FormatArg(const T& value)+            : m_value(static_cast<const void*>(&value)),+            m_formatImpl(&formatImpl<T>),+            m_toIntImpl(&toIntImpl<T>)+        { }++        void format(std::ostream& out, const char* fmtBegin,+                    const char* fmtEnd, int ntrunc) const+        {+            TINYFORMAT_ASSERT(m_value);+            TINYFORMAT_ASSERT(m_formatImpl);+            m_formatImpl(out, fmtBegin, fmtEnd, ntrunc, m_value);+        }++        int toInt() const+        {+            TINYFORMAT_ASSERT(m_value);+            TINYFORMAT_ASSERT(m_toIntImpl);+            return m_toIntImpl(m_value);+        }++    private:+        template<typename T>+        TINYFORMAT_HIDDEN static void formatImpl(std::ostream& out, const char* fmtBegin,+                        const char* fmtEnd, int ntrunc, const void* value)+        {+            formatValue(out, fmtBegin, fmtEnd, ntrunc, *static_cast<const T*>(value));+        }++        template<typename T>+        TINYFORMAT_HIDDEN static int toIntImpl(const void* value)+        {+            return convertToInt<T>::invoke(*static_cast<const T*>(value));+        }++        const void* m_value = nullptr;+        void (*m_formatImpl)(std::ostream& out, const char* fmtBegin,+                             const char* fmtEnd, int ntrunc, const void* value) = nullptr;+        int (*m_toIntImpl)(const void* value) = nullptr;+};+++// Parse and return an integer from the string c, as atoi()+// On return, c is set to one past the end of the integer.+inline int parseIntAndAdvance(const char*& c)+{+    int i = 0;+    for (;*c >= '0' && *c <= '9'; ++c)+        i = 10*i + (*c - '0');+    return i;+}++// Parse width or precision `n` from format string pointer `c`, and advance it+// to the next character. If an indirection is requested with `*`, the argument+// is read from `args[argIndex]` and `argIndex` is incremented (or read+// from `args[n]` in positional mode). Returns true if one or more+// characters were read.+inline bool parseWidthOrPrecision(int& n, const char*& c, bool positionalMode,+                                  const detail::FormatArg* args,+                                  int& argIndex, int numArgs)+{+    if (*c >= '0' && *c <= '9') {+        n = parseIntAndAdvance(c);+    }+    else if (*c == '*') {+        ++c;+        n = 0;+        if (positionalMode) {+            int pos = parseIntAndAdvance(c) - 1;+            if (*c != '$')+                TINYFORMAT_ERROR("tinyformat: Non-positional argument used after a positional one");+            if (pos >= 0 && pos < numArgs)+                n = args[pos].toInt();+            else+                TINYFORMAT_ERROR("tinyformat: Positional argument out of range");+            ++c;+        }+        else {+            if (argIndex < numArgs)+                n = args[argIndex++].toInt();+            else+                TINYFORMAT_ERROR("tinyformat: Not enough arguments to read variable width or precision");+        }+    }+    else {+        return false;+    }+    return true;+}++// Print literal part of format string and return next format spec position.+//+// Skips over any occurrences of '%%', printing a literal '%' to the output.+// The position of the first % character of the next nontrivial format spec is+// returned, or the end of string.+inline const char* printFormatStringLiteral(std::ostream& out, const char* fmt)+{+    const char* c = fmt;+    for (;; ++c) {+        if (*c == '\0') {+            out.write(fmt, c - fmt);+            return c;+        }+        else if (*c == '%') {+            out.write(fmt, c - fmt);+            if (*(c+1) != '%')+                return c;+            // for "%%", tack trailing % onto next literal section.+            fmt = ++c;+        }+    }+}+++// Parse a format string and set the stream state accordingly.+//+// The format mini-language recognized here is meant to be the one from C99,+// with the form "%[flags][width][.precision][length]type" with POSIX+// positional arguments extension.+//+// POSIX positional arguments extension:+// Conversions can be applied to the nth argument after the format in+// the argument list, rather than to the next unused argument. In this case,+// the conversion specifier character % (see below) is replaced by the sequence+// "%n$", where n is a decimal integer in the range [1,{NL_ARGMAX}],+// giving the position of the argument in the argument list. This feature+// provides for the definition of format strings that select arguments+// in an order appropriate to specific languages.+//+// The format can contain either numbered argument conversion specifications+// (that is, "%n$" and "*m$"), or unnumbered argument conversion specifications+// (that is, % and * ), but not both. The only exception to this is that %%+// can be mixed with the "%n$" form. The results of mixing numbered and+// unnumbered argument specifications in a format string are undefined.+// When numbered argument specifications are used, specifying the Nth argument+// requires that all the leading arguments, from the first to the (N-1)th,+// are specified in the format string.+//+// In format strings containing the "%n$" form of conversion specification,+// numbered arguments in the argument list can be referenced from the format+// string as many times as required.+//+// Formatting options which can't be natively represented using the ostream+// state are returned in spacePadPositive (for space padded positive numbers)+// and ntrunc (for truncating conversions).  argIndex is incremented if+// necessary to pull out variable width and precision.  The function returns a+// pointer to the character after the end of the current format spec.+inline const char* streamStateFromFormat(std::ostream& out, bool& positionalMode,+                                         bool& spacePadPositive,+                                         int& ntrunc, const char* fmtStart,+                                         const detail::FormatArg* args,+                                         int& argIndex, int numArgs)+{+    TINYFORMAT_ASSERT(*fmtStart == '%');+    // Reset stream state to defaults.+    out.width(0);+    out.precision(6);+    out.fill(' ');+    // Reset most flags; ignore irrelevant unitbuf & skipws.+    out.unsetf(std::ios::adjustfield | std::ios::basefield |+               std::ios::floatfield | std::ios::showbase | std::ios::boolalpha |+               std::ios::showpoint | std::ios::showpos | std::ios::uppercase);+    bool precisionSet = false;+    bool widthSet = false;+    int widthExtra = 0;+    const char* c = fmtStart + 1;++    // 1) Parse an argument index (if followed by '$') or a width possibly+    // preceded with '0' flag.+    if (*c >= '0' && *c <= '9') {+        const char tmpc = *c;+        int value = parseIntAndAdvance(c);+        if (*c == '$') {+            // value is an argument index+            if (value > 0 && value <= numArgs)+                argIndex = value - 1;+            else+                TINYFORMAT_ERROR("tinyformat: Positional argument out of range");+            ++c;+            positionalMode = true;+        }+        else if (positionalMode) {+            TINYFORMAT_ERROR("tinyformat: Non-positional argument used after a positional one");+        }+        else {+            if (tmpc == '0') {+                // Use internal padding so that numeric values are+                // formatted correctly, eg -00010 rather than 000-10+                out.fill('0');+                out.setf(std::ios::internal, std::ios::adjustfield);+            }+            if (value != 0) {+                // Nonzero value means that we parsed width.+                widthSet = true;+                out.width(value);+            }+        }+    }+    else if (positionalMode) {+        TINYFORMAT_ERROR("tinyformat: Non-positional argument used after a positional one");+    }+    // 2) Parse flags and width if we did not do it in previous step.+    if (!widthSet) {+        // Parse flags+        for (;; ++c) {+            switch (*c) {+                case '#':+                    out.setf(std::ios::showpoint | std::ios::showbase);+                    continue;+                case '0':+                    // overridden by left alignment ('-' flag)+                    if (!(out.flags() & std::ios::left)) {+                        // Use internal padding so that numeric values are+                        // formatted correctly, eg -00010 rather than 000-10+                        out.fill('0');+                        out.setf(std::ios::internal, std::ios::adjustfield);+                    }+                    continue;+                case '-':+                    out.fill(' ');+                    out.setf(std::ios::left, std::ios::adjustfield);+                    continue;+                case ' ':+                    // overridden by show positive sign, '+' flag.+                    if (!(out.flags() & std::ios::showpos))+                        spacePadPositive = true;+                    continue;+                case '+':+                    out.setf(std::ios::showpos);+                    spacePadPositive = false;+                    widthExtra = 1;+                    continue;+                default:+                    break;+            }+            break;+        }+        // Parse width+        int width = 0;+        widthSet = parseWidthOrPrecision(width, c, positionalMode,+                                         args, argIndex, numArgs);+        if (widthSet) {+            if (width < 0) {+                // negative widths correspond to '-' flag set+                out.fill(' ');+                out.setf(std::ios::left, std::ios::adjustfield);+                width = -width;+            }+            out.width(width);+        }+    }+    // 3) Parse precision+    if (*c == '.') {+        ++c;+        int precision = 0;+        parseWidthOrPrecision(precision, c, positionalMode,+                              args, argIndex, numArgs);+        // Presence of `.` indicates precision set, unless the inferred value+        // was negative in which case the default is used.+        precisionSet = precision >= 0;+        if (precisionSet)+            out.precision(precision);+    }+    // 4) Ignore any C99 length modifier+    while (*c == 'l' || *c == 'h' || *c == 'L' ||+           *c == 'j' || *c == 'z' || *c == 't') {+        ++c;+    }+    // 5) We're up to the conversion specifier character.+    // Set stream flags based on conversion specifier (thanks to the+    // boost::format class for forging the way here).+    bool intConversion = false;+    switch (*c) {+        case 'u': case 'd': case 'i':+            out.setf(std::ios::dec, std::ios::basefield);+            intConversion = true;+            break;+        case 'o':+            out.setf(std::ios::oct, std::ios::basefield);+            intConversion = true;+            break;+        case 'X':+            out.setf(std::ios::uppercase);+            // Falls through+        case 'x': case 'p':+            out.setf(std::ios::hex, std::ios::basefield);+            intConversion = true;+            break;+        case 'E':+            out.setf(std::ios::uppercase);+            // Falls through+        case 'e':+            out.setf(std::ios::scientific, std::ios::floatfield);+            out.setf(std::ios::dec, std::ios::basefield);+            break;+        case 'F':+            out.setf(std::ios::uppercase);+            // Falls through+        case 'f':+            out.setf(std::ios::fixed, std::ios::floatfield);+            break;+        case 'A':+            out.setf(std::ios::uppercase);+            // Falls through+        case 'a':+#           ifdef _MSC_VER+            // Workaround https://developercommunity.visualstudio.com/content/problem/520472/hexfloat-stream-output-does-not-ignore-precision-a.html+            // by always setting maximum precision on MSVC to avoid precision+            // loss for doubles.+            out.precision(13);+#           endif+            out.setf(std::ios::fixed | std::ios::scientific, std::ios::floatfield);+            break;+        case 'G':+            out.setf(std::ios::uppercase);+            // Falls through+        case 'g':+            out.setf(std::ios::dec, std::ios::basefield);+            // As in boost::format, let stream decide float format.+            out.flags(out.flags() & ~std::ios::floatfield);+            break;+        case 'c':+            // Handled as special case inside formatValue()+            break;+        case 's':+            if (precisionSet)+                ntrunc = static_cast<int>(out.precision());+            // Make %s print Booleans as "true" and "false"+            out.setf(std::ios::boolalpha);+            break;+        case 'n':+            // Not supported - will cause problems!+            TINYFORMAT_ERROR("tinyformat: %n conversion spec not supported");+            break;+        case '\0':+            TINYFORMAT_ERROR("tinyformat: Conversion spec incorrectly "+                             "terminated by end of string");+            return c;+        default:+            break;+    }+    if (intConversion && precisionSet && !widthSet) {+        // "precision" for integers gives the minimum number of digits (to be+        // padded with zeros on the left).  This isn't really supported by the+        // iostreams, but we can approximately simulate it with the width if+        // the width isn't otherwise used.+        out.width(out.precision() + widthExtra);+        out.setf(std::ios::internal, std::ios::adjustfield);+        out.fill('0');+    }+    return c+1;+}+++//------------------------------------------------------------------------------+inline void formatImpl(std::ostream& out, const char* fmt,+                       const detail::FormatArg* args,+                       int numArgs)+{+    // Saved stream state+    std::streamsize origWidth = out.width();+    std::streamsize origPrecision = out.precision();+    std::ios::fmtflags origFlags = out.flags();+    char origFill = out.fill();++    // "Positional mode" means all format specs should be of the form "%n$..."+    // with `n` an integer. We detect this in `streamStateFromFormat`.+    bool positionalMode = false;+    int argIndex = 0;+    while (true) {+        fmt = printFormatStringLiteral(out, fmt);+        if (*fmt == '\0') {+            if (!positionalMode && argIndex < numArgs) {+                TINYFORMAT_ERROR("tinyformat: Not enough conversion specifiers in format string");+            }+            break;+        }+        bool spacePadPositive = false;+        int ntrunc = -1;+        const char* fmtEnd = streamStateFromFormat(out, positionalMode, spacePadPositive, ntrunc, fmt,+                                                   args, argIndex, numArgs);+        // NB: argIndex may be incremented by reading variable width/precision+        // in `streamStateFromFormat`, so do the bounds check here.+        if (argIndex >= numArgs) {+            TINYFORMAT_ERROR("tinyformat: Too many conversion specifiers in format string");+            return;+        }+        const FormatArg& arg = args[argIndex];+        // Format the arg into the stream.+        if (!spacePadPositive) {+            arg.format(out, fmt, fmtEnd, ntrunc);+        }+        else {+            // The following is a special case with no direct correspondence+            // between stream formatting and the printf() behaviour.  Simulate+            // it crudely by formatting into a temporary string stream and+            // munging the resulting string.+            std::ostringstream tmpStream;+            tmpStream.copyfmt(out);+            tmpStream.setf(std::ios::showpos);+            arg.format(tmpStream, fmt, fmtEnd, ntrunc);+            std::string result = tmpStream.str(); // allocates... yuck.+            for (char & i : result) {+                if (i == '+')+                    i = ' ';+            }+            out << result;+        }+        if (!positionalMode)+            ++argIndex;+        fmt = fmtEnd;+    }++    // Restore stream state+    out.width(origWidth);+    out.precision(origPrecision);+    out.flags(origFlags);+    out.fill(origFill);+}++} // namespace detail+++/// List of template arguments format(), held in a type-opaque way.+///+/// A const reference to FormatList (typedef'd as FormatListRef) may be+/// conveniently used to pass arguments to non-template functions: All type+/// information has been stripped from the arguments, leaving just enough of a+/// common interface to perform formatting as required.+class FormatList+{+    public:+        FormatList(detail::FormatArg* args, int N)+            : m_args(args), m_N(N) { }++        friend void vformat(std::ostream& out, const char* fmt,+                            const FormatList& list);++    private:+        const detail::FormatArg* m_args;+        int m_N;+};++/// Reference to type-opaque format list for passing to vformat()+using FormatListRef = const FormatList &;+++namespace detail {++// Format list subclass with fixed storage to avoid dynamic allocation+template<int N>+class FormatListN : public FormatList+{+    public:+#ifdef TINYFORMAT_USE_VARIADIC_TEMPLATES+        template<typename... Args>+        FormatListN(const Args&... args)+            : FormatList(&m_formatterStore[0], N),+            m_formatterStore { FormatArg(args)... }+        { static_assert(sizeof...(args) == N, "Number of args must be N"); }+#else // C++98 version+        void init(int) {}+#       define TINYFORMAT_MAKE_FORMATLIST_CONSTRUCTOR(n)                \+                                                                        \+        template<TINYFORMAT_ARGTYPES(n)>                                \+        FormatListN(TINYFORMAT_VARARGS(n))                              \+            : FormatList(&m_formatterStore[0], n)                       \+        { TINYFORMAT_ASSERT(n == N); init(0, TINYFORMAT_PASSARGS(n)); } \+                                                                        \+        template<TINYFORMAT_ARGTYPES(n)>                                \+        void init(int i, TINYFORMAT_VARARGS(n))                         \+        {                                                               \+            m_formatterStore[i] = FormatArg(v1);                        \+            init(i+1 TINYFORMAT_PASSARGS_TAIL(n));                      \+        }++        TINYFORMAT_FOREACH_ARGNUM(TINYFORMAT_MAKE_FORMATLIST_CONSTRUCTOR)+#       undef TINYFORMAT_MAKE_FORMATLIST_CONSTRUCTOR+#endif+        FormatListN(const FormatListN& other)+            : FormatList(&m_formatterStore[0], N)+        { std::copy(&other.m_formatterStore[0], &other.m_formatterStore[N],+                    &m_formatterStore[0]); }++    private:+        FormatArg m_formatterStore[N];+};++// Special 0-arg version - MSVC says zero-sized C array in struct is nonstandard+template<> class FormatListN<0> : public FormatList+{+    public: FormatListN() : FormatList(nullptr, 0) {}+};++} // namespace detail+++//------------------------------------------------------------------------------+// Primary API functions++#ifdef TINYFORMAT_USE_VARIADIC_TEMPLATES++/// Make type-agnostic format list from list of template arguments.+///+/// The exact return type of this function is an implementation detail and+/// shouldn't be relied upon.  Instead it should be stored as a FormatListRef:+///+///   FormatListRef formatList = makeFormatList( /*...*/ );+template<typename... Args>+detail::FormatListN<sizeof...(Args)> makeFormatList(const Args&... args)+{+    return detail::FormatListN<sizeof...(args)>(args...);+}++#else // C++98 version++inline detail::FormatListN<0> makeFormatList()+{+    return detail::FormatListN<0>();+}+#define TINYFORMAT_MAKE_MAKEFORMATLIST(n)                     \+template<TINYFORMAT_ARGTYPES(n)>                              \+detail::FormatListN<n> makeFormatList(TINYFORMAT_VARARGS(n))  \+{                                                             \+    return detail::FormatListN<n>(TINYFORMAT_PASSARGS(n));    \+}+TINYFORMAT_FOREACH_ARGNUM(TINYFORMAT_MAKE_MAKEFORMATLIST)+#undef TINYFORMAT_MAKE_MAKEFORMATLIST++#endif++/// Format list of arguments to the stream according to the given format string.+///+/// The name vformat() is chosen for the semantic similarity to vprintf(): the+/// list of format arguments is held in a single function argument.+inline void vformat(std::ostream& out, const char* fmt, FormatListRef list)+{+    detail::formatImpl(out, fmt, list.m_args, list.m_N);+}+++#ifdef TINYFORMAT_USE_VARIADIC_TEMPLATES++/// Format list of arguments to the stream according to given format string.+template<typename... Args>+void format(std::ostream& out, const char* fmt, const Args&... args)+{+    vformat(out, fmt, makeFormatList(args...));+}++/// Format list of arguments according to the given format string and return+/// the result as a string.+template<typename... Args>+std::string format(const char* fmt, const Args&... args)+{+    std::ostringstream oss;+    format(oss, fmt, args...);+    return oss.str();+}++/// Format list of arguments to std::cout, according to the given format string+template<typename... Args>+void printf(const char* fmt, const Args&... args)+{+    format(std::cout, fmt, args...);+}++template<typename... Args>+void printfln(const char* fmt, const Args&... args)+{+    format(std::cout, fmt, args...);+    std::cout << '\n';+}+++#else // C++98 version++inline void format(std::ostream& out, const char* fmt)+{+    vformat(out, fmt, makeFormatList());+}++inline std::string format(const char* fmt)+{+    std::ostringstream oss;+    format(oss, fmt);+    return oss.str();+}++inline void printf(const char* fmt)+{+    format(std::cout, fmt);+}++inline void printfln(const char* fmt)+{+    format(std::cout, fmt);+    std::cout << '\n';+}++#define TINYFORMAT_MAKE_FORMAT_FUNCS(n)                                   \+                                                                          \+template<TINYFORMAT_ARGTYPES(n)>                                          \+void format(std::ostream& out, const char* fmt, TINYFORMAT_VARARGS(n))    \+{                                                                         \+    vformat(out, fmt, makeFormatList(TINYFORMAT_PASSARGS(n)));            \+}                                                                         \+                                                                          \+template<TINYFORMAT_ARGTYPES(n)>                                          \+std::string format(const char* fmt, TINYFORMAT_VARARGS(n))                \+{                                                                         \+    std::ostringstream oss;                                               \+    format(oss, fmt, TINYFORMAT_PASSARGS(n));                             \+    return oss.str();                                                     \+}                                                                         \+                                                                          \+template<TINYFORMAT_ARGTYPES(n)>                                          \+void printf(const char* fmt, TINYFORMAT_VARARGS(n))                       \+{                                                                         \+    format(std::cout, fmt, TINYFORMAT_PASSARGS(n));                       \+}                                                                         \+                                                                          \+template<TINYFORMAT_ARGTYPES(n)>                                          \+void printfln(const char* fmt, TINYFORMAT_VARARGS(n))                     \+{                                                                         \+    format(std::cout, fmt, TINYFORMAT_PASSARGS(n));                       \+    std::cout << '\n';                                                    \+}++TINYFORMAT_FOREACH_ARGNUM(TINYFORMAT_MAKE_FORMAT_FUNCS)+#undef TINYFORMAT_MAKE_FORMAT_FUNCS++#endif+++} // namespace tinyformat++#endif // TINYFORMAT_H_INCLUDED++// clang-format on
lib/Language/Souffle/Class.hs view
@@ -62,7 +62,7 @@ -- -- instance Program Path where --   type ProgramFacts Path = '[Edge, Reachable]---   factName = const "path"+--   programName = const "path" -- @ class Program a where   -- | A type level list of facts that belong to this program.@@ -253,36 +253,36 @@   -- | Load all facts from files in a certain directory.   loadFiles :: Handler m prog -> FilePath -> m () -  -- | Write out all facts of the program to CSV files+  -- | Write out all facts of the program to CSV files in a certain directory   --   (as defined in the Souffle program).-  writeFiles :: Handler m prog -> m ()+  writeFiles :: Handler m prog -> FilePath -> m ()  instance MonadSouffleFileIO m => MonadSouffleFileIO (ReaderT r m) where   loadFiles prog = lift . loadFiles prog   {-# INLINABLE loadFiles #-}-  writeFiles = lift . writeFiles+  writeFiles prog = lift . writeFiles prog   {-# INLINABLE writeFiles #-}  instance (Monoid w, MonadSouffleFileIO m) => MonadSouffleFileIO (WriterT w m) where   loadFiles prog = lift . loadFiles prog   {-# INLINABLE loadFiles #-}-  writeFiles = lift . writeFiles+  writeFiles prog = lift . writeFiles prog   {-# INLINABLE writeFiles #-}  instance MonadSouffleFileIO m => MonadSouffleFileIO (StateT s m) where   loadFiles prog = lift . loadFiles prog   {-# INLINABLE loadFiles #-}-  writeFiles = lift . writeFiles+  writeFiles prog = lift . writeFiles prog   {-# INLINABLE writeFiles #-}  instance (MonadSouffleFileIO m, Monoid w) => MonadSouffleFileIO (RWST r w s m) where   loadFiles prog = lift . loadFiles prog   {-# INLINABLE loadFiles #-}-  writeFiles = lift . writeFiles+  writeFiles prog = lift . writeFiles prog   {-# INLINABLE writeFiles #-}  instance MonadSouffleFileIO m => MonadSouffleFileIO (ExceptT s m) where   loadFiles prog = lift . loadFiles prog   {-# INLINABLE loadFiles #-}-  writeFiles = lift . writeFiles+  writeFiles prog = lift . writeFiles prog   {-# INLINABLE writeFiles #-}
lib/Language/Souffle/Compiled.hs view
@@ -177,6 +177,6 @@   loadFiles (Handle prog) = SouffleM . Internal.loadAll prog   {-# INLINABLE loadFiles #-} -  writeFiles (Handle prog) = SouffleM $ Internal.printAll prog+  writeFiles (Handle prog) = SouffleM . Internal.printAll prog   {-# INLINABLE writeFiles #-}
lib/Language/Souffle/Internal.hs view
@@ -83,13 +83,13 @@  -- | Load all facts from files in a certain directory. loadAll :: ForeignPtr Souffle -> FilePath -> IO ()-loadAll prog str = withForeignPtr prog $ withCString str . Bindings.loadAll+loadAll prog inputDir = withForeignPtr prog $ withCString inputDir . Bindings.loadAll {-# INLINABLE loadAll #-} --- | Write out all facts of the program to CSV files+-- | Write out all facts of the program to CSV files in a certain directory --   (as defined in the Souffle program).-printAll :: ForeignPtr Souffle -> IO ()-printAll prog = withForeignPtr prog Bindings.printAll+printAll :: ForeignPtr Souffle -> FilePath -> IO ()+printAll prog outputDir = withForeignPtr prog $ withCString outputDir . Bindings.printAll {-# INLINABLE printAll #-}  {-| Lookup a relation by name in the Souffle program.
lib/Language/Souffle/Internal/Bindings.hs view
@@ -104,14 +104,14 @@ foreign import ccall unsafe "souffle_load_all" loadAll   :: Ptr Souffle -> CString -> IO () -{-| Write out all facts of the program to CSV files+{-| Write out all facts of the program to CSV files in a given directory     (as defined in the Souffle program).      You need to check if the pointer is not equal to 'nullPtr' before passing it     to this function. Not doing so results in undefined behavior (in C++). -} foreign import ccall unsafe "souffle_print_all" printAll-  :: Ptr Souffle -> IO ()+  :: Ptr Souffle -> CString -> IO ()  {-| Lookup a relation in the Souffle program.
lib/Language/Souffle/Interpreted.hs view
@@ -21,11 +21,14 @@   , runSouffleWith   , defaultConfig   , cleanup+  , souffleStdOut+  , souffleStdErr   ) where  import Prelude hiding (init)  import Control.DeepSeq (deepseq)+import Control.Exception (ErrorCall(..), throwIO) import Control.Monad.State.Strict import Control.Monad.Reader import Data.IORef@@ -34,6 +37,7 @@ import Data.Semigroup (Last(..)) import Data.Maybe (fromMaybe) import Data.Proxy+import qualified Data.Text as T import qualified Data.Vector as V import Data.Word import Language.Souffle.Class@@ -42,7 +46,7 @@ import System.Environment import System.Exit import System.FilePath-import System.IO (hGetContents)+import System.IO (hGetContents, hClose) import System.IO.Temp import System.Process import Text.Printf@@ -61,10 +65,17 @@ --   - __cfgSouffleBin__: The name of the souffle binary. Has to be available in --   \$PATH or an absolute path needs to be provided. Note: Passing in `Nothing` --   will fail to start up the interpreter in the `MonadSouffle.init` function.+--   - __cfgFactDir__: The directory where the initial input fact file(s) can be found+--   if present. If Nothing, then a temporary directory will be used, during the+--   souffle session.+--   - __cfgOutputDir__: The directory where the output facts file(s) are created.+--   If Nothing, it will be part of the temporary directory. data Config   = Config-  { cfgDatalogDir :: FilePath-  , cfgSouffleBin :: Maybe FilePath+  { cfgDatalogDir   :: FilePath+  , cfgSouffleBin   :: Maybe FilePath+  , cfgFactDir      :: Maybe FilePath+  , cfgOutputDir    :: Maybe FilePath   } deriving Show  -- | Retrieves the default config for the interpreter. These settings can@@ -77,13 +88,15 @@ --   - __cfgSouffleBin__: Looks at environment variable \$SOUFFLE_BIN, --   or tries to locate the souffle binary using the which shell command --   if the variable is not set.+--   - __cfgFactDir__: Will make use of a temporary directory.+--   - __cfgOutputDir__: Will make use of a temporary directory. defaultConfig :: MonadIO m => m Config defaultConfig = liftIO $ do   dlDir <- lookupEnv "DATALOG_DIR"   envSouffleBin <- fmap Last <$> lookupEnv "SOUFFLE_BIN"   locatedBin <- fmap Last <$> locateSouffle   let souffleBin = getLast <$> locatedBin <> envSouffleBin-  pure $ Config (fromMaybe "." dlDir) souffleBin+  pure $ Config (fromMaybe "." dlDir) souffleBin Nothing Nothing {-# INLINABLE defaultConfig #-}  -- | Returns an IO action that will run the Souffle interpreter with@@ -103,7 +116,11 @@ -- | A datatype representing a handle to a datalog program. --   The type parameter is used for keeping track of which program --   type the handle belongs to for additional type safety.-newtype Handle prog = Handle (IORef HandleData)+data Handle prog = Handle+  { handleData   :: IORef HandleData+  , stdoutResult :: IORef (Maybe T.Text)+  , stderrResult :: IORef (Maybe T.Text)+  }  -- | The data needed for the interpreter is the path where the souffle --   executable can be found, and a template directory where the program@@ -172,48 +189,69 @@       souffleTempDir <- liftIO $ do         tmpDir <- getCanonicalTemporaryDirectory         createTempDirectory tmpDir "souffle-haskell"-      let factDir = souffleTempDir </> "fact"-          outDir  = souffleTempDir </> "out"++      factDir <- fromMaybe (souffleTempDir </> "fact") <$> asks cfgFactDir+      outDir <- fromMaybe (souffleTempDir </> "out") <$> asks cfgOutputDir       liftIO $ do         createDirectoryIfMissing True factDir         createDirectoryIfMissing True outDir       mSouffleBin <- asks cfgSouffleBin       liftIO $ forM mSouffleBin $ \souffleBin ->-        fmap Handle $ newIORef $ HandleData-          { soufflePath = souffleBin-          , basePath    = souffleTempDir-          , factPath    = factDir-          , outputPath  = outDir-          , datalogExec = datalogExecutable-          , noOfThreads = 1-          }+        Handle+          <$> (newIORef $ HandleData+                { soufflePath = souffleBin+                , basePath    = souffleTempDir+                , factPath    = factDir+                , outputPath  = outDir+                , datalogExec = datalogExecutable+                , noOfThreads = 1+                })+          <*> newIORef Nothing+          <*> newIORef Nothing   {-# INLINABLE init #-} -  run (Handle ref) = liftIO $ do-    handle <- readIORef ref+  run (Handle refHandleData refHandleStdOut refHandleStdErr) = liftIO $ do+    handle <- readIORef refHandleData     -- Invoke the souffle binary using parameters, supposing that the facts     -- are placed in the factPath, rendering the output into the outputPath.-    callCommand $-      printf "%s -F%s -D%s -j%d %s"-        (soufflePath handle)-        (factPath handle)-        (outputPath handle)-        (noOfThreads handle)-        (datalogExec handle)+    let processToRun =+          (shell+            (printf "%s -F%s -D%s -j%d %s"+              (soufflePath handle)+              (factPath handle)+              (outputPath handle)+              (noOfThreads handle)+              (datalogExec handle)))+            { std_in  = NoStream+            , std_out = CreatePipe+            , std_err = CreatePipe+            }+    (_, mStdOutHandle, mStdErrHandle, processHandle) <- createProcess_ "souffle-haskell" processToRun+    waitForProcess processHandle >>= \case+      ExitSuccess   -> pure ()+      ExitFailure c -> throwIO $ ErrorCall $ "Souffle exited with: " ++ show c+    forM_ mStdOutHandle $ \stdoutHandle -> do+      stdout <- T.pack <$> hGetContents stdoutHandle+      writeIORef refHandleStdOut $! Just $! stdout+      hClose stdoutHandle+    forM_ mStdErrHandle $ \stderrHandle -> do+      stderr <- T.pack <$> hGetContents stderrHandle+      writeIORef refHandleStdErr $! Just $! stderr+      hClose stderrHandle   {-# INLINABLE run #-} -  setNumThreads (Handle ref) n = liftIO $-    modifyIORef' ref (\h -> h { noOfThreads = n })+  setNumThreads handle n = liftIO $+    modifyIORef' (handleData handle) (\h -> h { noOfThreads = n })   {-# INLINABLE setNumThreads #-} -  getNumThreads (Handle ref) = liftIO $-    noOfThreads <$> readIORef ref+  getNumThreads handle = liftIO $+    noOfThreads <$> readIORef (handleData handle)   {-# INLINABLE getNumThreads #-}    getFacts :: forall a c prog. (Marshal a, Fact a, ContainsFact prog a, Collect c)            => Handle prog -> SouffleM (c a)-  getFacts (Handle ref) = liftIO $ do-    handle <- readIORef ref+  getFacts h = liftIO $ do+    handle <- readIORef $ handleData h     let relationName = factName (Proxy :: Proxy a)     let factFile = outputPath handle </> relationName <.> "csv"     facts <- collect factFile@@ -229,8 +267,8 @@    addFact :: forall a prog. (Fact a, ContainsFact prog a, Marshal a)           => Handle prog -> a -> SouffleM ()-  addFact (Handle ref) fact = liftIO $ do-    handle <- readIORef ref+  addFact h fact = liftIO $ do+    handle <- readIORef $ handleData h     let relationName = factName (Proxy :: Proxy a)     let factFile = factPath handle </> relationName <.> "facts"     let line = pushMarshalT (push fact)@@ -239,8 +277,8 @@    addFacts :: forall a prog f. (Fact a, ContainsFact prog a, Marshal a, Foldable f)            => Handle prog -> f a -> SouffleM ()-  addFacts (Handle ref) facts = SouffleM $ liftIO $ do-    handle <- readIORef ref+  addFacts h facts = liftIO $ do+    handle <- readIORef $ handleData h     let relationName = factName (Proxy :: Proxy a)     let factFile = factPath handle </> relationName <.> "facts"     let factLines = map (pushMarshalT . push) (foldMap pure facts)@@ -281,15 +319,22 @@ --   This functionality is only provided for the interpreted version since the --   compiled version directly (de-)serializes data via the C++ API. cleanup :: forall prog. Program prog => Handle prog -> SouffleM ()-cleanup (Handle ref) = liftIO $ do-  handle <- readIORef ref+cleanup h  = liftIO $ do+  handle <- readIORef $ handleData h   traverse_ removeDirectoryRecursive [factPath handle, outputPath handle, basePath handle] {-# INLINABLE cleanup #-} +-- | Returns the handle of stdout from the souffle interpreter.+souffleStdOut :: forall prog. Program prog => Handle prog -> SouffleM (Maybe T.Text)+souffleStdOut = liftIO . readIORef . stdoutResult++-- | Returns the content of stderr from the souffle interpreter.+souffleStdErr :: forall prog. Program prog => Handle prog -> SouffleM (Maybe T.Text)+souffleStdErr = liftIO . readIORef . stderrResult+ splitOn :: Char -> String -> [String] splitOn c s =   let (x, rest) = break (== c) s       rest' = drop 1 rest    in x : splitOn c rest' {-# INLINABLE splitOn #-}-
− lib/Language/Souffle/TH.hs 
@@ -1,25 +0,0 @@---- | Helper module for easier integration of generated embedded souffle programs---   with Haskell code. Without these helper functions, it becomes much harder---   to link the C++ code directly to Haskell due to the way how Souffle---   generates the code.-module Language.Souffle.TH ( embedProgram ) where--import Language.Haskell.TH.Syntax----- | Helper function for embedding a Souffle program in Haskell.---   Requires the use of the TemplateHaskell language extension.------   The passed in String should be a path relative from the root of the---   project where the .cpp file is located.------   Example usage:------ @--- module Main where--- import Language.Haskell.TH.Syntax as Souffle--- Souffle.embedProgram "path\/to\/file.cpp"  -- NOTE: call directly on top level!--- @-embedProgram :: String -> Q [Dec]-embedProgram path = [] <$ qAddForeignFilePath LangCxx path
+ scripts/import_souffle_headers.hs view
@@ -0,0 +1,144 @@+{-# LANGUAGE DataKinds, TypeFamilies, DeriveGeneric, DeriveAnyClass #-}++module Main ( main ) where++-- NOTE: This is a helper script for importing all Souffle headers into+-- this repository. This is done in order to make Haskell libraries that+-- use souffle-haskell "self-contained", meaning users of the packages+-- using those libraries are not required to have souffle (headers) installed.++import System.Directory+import System.FilePath+import System.Process+import Control.Monad+import Control.Monad.Extra+import Control.Applicative+import Data.List+import Data.Maybe+import Data.Void+import GHC.Generics+import qualified Text.Megaparsec as P+import qualified Text.Megaparsec.Char as P+import qualified Language.Souffle.Interpreted as Souffle+++data Includes = Includes FilePath FilePath+  deriving (Eq, Show, Generic, Souffle.Marshal)++newtype TopLevelInclude = TopLevelInclude FilePath+  deriving (Eq, Show, Generic, Souffle.Marshal)++newtype RequiredInclude = RequiredInclude FilePath+  deriving (Eq, Show, Generic, Souffle.Marshal)++data Handle = Handle++instance Souffle.Program Handle where+  type ProgramFacts Handle = [TopLevelInclude, Includes, RequiredInclude]+  programName = const "required_include"++instance Souffle.Fact Includes where+  factName = const "includes"++instance Souffle.Fact TopLevelInclude where+  factName = const "top_level_include"++instance Souffle.Fact RequiredInclude where+  factName = const "required_include"+++run :: String -> IO ()+run = callCommand++runWithResult :: String -> IO String+runWithResult s = case words s of+  (program:args) -> readProcess program args ""+  _ -> error "Passed empty string to 'runWithResult'. Aborting."++headerDir :: FilePath+headerDir = "cbits/souffle/"++main :: IO ()+main = do+  pwd <- getCurrentDirectory+  gitRoot <- getGitRootDirectory+  when (pwd /= gitRoot) $+    putStrLn "You need to run this script in the root directory of this repo! Aborting."++  run "git submodule update --init --recursive"+  run $ "rm -rf " <> headerDir <> "*.h"+  run $ "cp souffle/LICENSE " <> headerDir+  files <- copyHeaders+  putStrLn "Replace 'install-includes' in your package.yaml with the following:"+  putStrLn . unlines $ map ("  - souffle/" <>) files++getGitRootDirectory :: IO FilePath+getGitRootDirectory =+  filter (/= '\n') <$> runWithResult "git rev-parse --show-toplevel"++parseIncludes :: String -> [FilePath]+parseIncludes s = either (const []) catMaybes $ P.runParser parser "" s where+  parser :: P.Parsec Void String [Maybe FilePath]+  parser = many (includeParser <|> skipRestOfLine) <* P.eof+  includeParser = do+    P.chunk "#include" *> P.space1+    P.lookAhead (P.char '"' <|> P.char '<') >>= \case+      '"' -> do+        include <- P.between quotes quotes $ P.takeWhile1P Nothing (/= '"')+        void skipRestOfLine+        pure $ Just include+      _ -> pure Nothing+  quotes = P.char '"'+  skipRestOfLine = Nothing <$ (P.takeWhileP Nothing (/= '\n') *> P.newline)++parseIncludesInHeader :: FilePath -> IO [Includes]+parseIncludesInHeader file = f <$> readFile file where+  f = map ((file `Includes`) . normalizeFilePath dir) . parseIncludes+  dir = takeDirectory file++normalizeFilePath :: FilePath -> FilePath -> FilePath+normalizeFilePath dir file = normalize $ dir </> file' where+  file' = if "souffle/" `isPrefixOf` file then file \\ "souffle/" else file+  normalize = withExplodedPath (reverse . removeParentDirRefs . reverse)+  removeParentDirRefs = \case+    ("../":_:xs) -> removeParentDirRefs xs+    (x:xs) -> x:removeParentDirRefs xs+    [] -> []++copyHeaders :: IO [FilePath]+copyHeaders = do+  headers <- filter (".h" `isSuffixOf`) . lines+          <$> runWithResult "find souffle -type f"+  includes <- concatMapM parseIncludesInHeader headers+  traverse copyHeader =<< computeRequiredIncludes includes++computeRequiredIncludes :: [Includes] -> IO [FilePath]+computeRequiredIncludes includes = do+  cfg <- Souffle.defaultConfig+  let config = cfg { Souffle.cfgDatalogDir = "./scripts" }+  requiredIncludes <- Souffle.runSouffleWith config $+    Souffle.init Handle >>= \case+      Nothing -> error "Failed to load Souffle program. Aborting."+      Just prog -> do+        Souffle.addFacts prog [ TopLevelInclude "souffle/src/SouffleInterface.h"+                              , TopLevelInclude "souffle/src/CompiledSouffle.h"+                              ]+        Souffle.addFacts prog includes+        Souffle.run prog+        Souffle.getFacts prog+  pure $ map (\(RequiredInclude include) -> include) requiredIncludes++copyHeader :: FilePath -> IO FilePath+copyHeader file = do+  header <- head . filter (file `isSuffixOf`) . lines+        <$> runWithResult "find souffle/ -type f"+  let header' = withExplodedPath (drop 2) header+      dir = headerDir </> takeDirectory header'+      destination = replaceDirectory header' dir+  createDirectoryIfMissing True dir+  copyFile header destination+  pure header'++withExplodedPath :: ([FilePath] -> [FilePath]) -> FilePath -> FilePath+withExplodedPath f = joinPath . f . splitPath+
souffle-haskell.cabal view
@@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack ----- hash: edee24fd0c26174211bf0f36205df0d361d3a2d6d55987259a09e4b4a80bd62d+-- hash: e178bba53f0bfba290c3747f4551e6a6300a1f9a2fd84f691701b456b2001cc8  name:           souffle-haskell-version:        0.2.3+version:        1.0.0 synopsis:       Souffle Datalog bindings for Haskell description:    Souffle Datalog bindings for Haskell. category:       Logic Programming, Foreign Binding, Bindings@@ -24,6 +24,51 @@     CHANGELOG.md     LICENSE     cbits/souffle.h+    cbits/souffle/Brie.h+    cbits/souffle/BTree.h+    cbits/souffle/CompiledOptions.h+    cbits/souffle/CompiledSouffle.h+    cbits/souffle/CompiledTuple.h+    cbits/souffle/EquivalenceRelation.h+    cbits/souffle/EventProcessor.h+    cbits/souffle/gzfstream.h+    cbits/souffle/IOSystem.h+    cbits/souffle/IterUtils.h+    cbits/souffle/json11.h+    cbits/souffle/LambdaBTree.h+    cbits/souffle/Logger.h+    cbits/souffle/PiggyList.h+    cbits/souffle/profile/CellInterface.h+    cbits/souffle/profile/DataComparator.h+    cbits/souffle/profile/Row.h+    cbits/souffle/profile/Table.h+    cbits/souffle/ProfileDatabase.h+    cbits/souffle/ProfileEvent.h+    cbits/souffle/RamTypes.h+    cbits/souffle/ReadStream.h+    cbits/souffle/ReadStreamCSV.h+    cbits/souffle/ReadStreamSQLite.h+    cbits/souffle/RecordTable.h+    cbits/souffle/SerialisationStream.h+    cbits/souffle/SignalHandler.h+    cbits/souffle/SouffleInterface.h+    cbits/souffle/SymbolTable.h+    cbits/souffle/Table.h+    cbits/souffle/UnionFind.h+    cbits/souffle/utility/CacheUtil.h+    cbits/souffle/utility/ContainerUtil.h+    cbits/souffle/utility/EvaluatorUtil.h+    cbits/souffle/utility/FileUtil.h+    cbits/souffle/utility/FunctionalUtil.h+    cbits/souffle/utility/MiscUtil.h+    cbits/souffle/utility/ParallelUtil.h+    cbits/souffle/utility/StreamUtil.h+    cbits/souffle/utility/StringUtil.h+    cbits/souffle/utility/tinyformat.h+    cbits/souffle/WriteStream.h+    cbits/souffle/WriteStreamCSV.h+    cbits/souffle/WriteStreamSQLite.h+    cbits/souffle/LICENSE  source-repository head   type: git@@ -39,7 +84,6 @@       Language.Souffle.Internal.Constraints       Language.Souffle.Interpreted       Language.Souffle.Marshal-      Language.Souffle.TH   other-modules:       Paths_souffle_haskell   autogen-modules:@@ -48,8 +92,51 @@       lib   default-extensions: OverloadedStrings LambdaCase ScopedTypeVariables   ghc-options: -Wall -Weverything -Wno-safe -Wno-unsafe -Wno-implicit-prelude -Wno-missed-specializations -Wno-all-missed-specializations -Wno-missing-import-lists -Wno-type-defaults -Wno-missing-local-signatures -Wno-monomorphism-restriction -Wno-missing-deriving-strategies -optP-Wno-nonportable-include-path -fhide-source-paths -fno-show-valid-hole-fits -fno-sort-valid-hole-fits-  cpp-options: -std=c++17-  cxx-options: -Wall+  cxx-options: -std=c++17 -Wall+  include-dirs:+      cbits+      cbits/souffle+  install-includes:+      souffle/RamTypes.h+      souffle/utility/MiscUtil.h+      souffle/SymbolTable.h+      souffle/json11.h+      souffle/utility/FunctionalUtil.h+      souffle/utility/StreamUtil.h+      souffle/utility/StringUtil.h+      souffle/ReadStreamCSV.h+      souffle/ReadStream.h+      souffle/utility/ContainerUtil.h+      souffle/utility/FileUtil.h+      souffle/gzfstream.h+      souffle/CompiledTuple.h+      souffle/RecordTable.h+      souffle/Brie.h+      souffle/utility/CacheUtil.h+      souffle/SouffleInterface.h+      souffle/EventProcessor.h+      souffle/ProfileDatabase.h+      souffle/WriteStreamSQLite.h+      souffle/WriteStream.h+      souffle/LambdaBTree.h+      souffle/BTree.h+      souffle/utility/ParallelUtil.h+      souffle/WriteStreamCSV.h+      souffle/utility/tinyformat.h+      souffle/IOSystem.h+      souffle/ReadStreamSQLite.h+      souffle/UnionFind.h+      souffle/PiggyList.h+      souffle/ProfileEvent.h+      souffle/SerialisationStream.h+      souffle/CompiledSouffle.h+      souffle/EquivalenceRelation.h+      souffle/IterUtils.h+      souffle/SignalHandler.h+      souffle/Table.h+      souffle/utility/EvaluatorUtil.h+      souffle/CompiledOptions.h+      souffle/Logger.h   cxx-sources:       cbits/souffle.cpp   build-depends:@@ -69,6 +156,77 @@         stdc++   default-language: Haskell2010 +executable import-souffle-headers+  main-is: import_souffle_headers.hs+  other-modules:+      Paths_souffle_haskell+  hs-source-dirs:+      scripts+  default-extensions: OverloadedStrings LambdaCase ScopedTypeVariables+  ghc-options: -Wall -Weverything -Wno-safe -Wno-unsafe -Wno-implicit-prelude -Wno-missed-specializations -Wno-all-missed-specializations -Wno-missing-import-lists -Wno-type-defaults -Wno-missing-local-signatures -Wno-monomorphism-restriction -Wno-missing-deriving-strategies -optP-Wno-nonportable-include-path -fhide-source-paths -fno-show-valid-hole-fits -fno-sort-valid-hole-fits+  cxx-options: -std=c++17+  include-dirs:+      cbits+      cbits/souffle+  install-includes:+      souffle/RamTypes.h+      souffle/utility/MiscUtil.h+      souffle/SymbolTable.h+      souffle/json11.h+      souffle/utility/FunctionalUtil.h+      souffle/utility/StreamUtil.h+      souffle/utility/StringUtil.h+      souffle/ReadStreamCSV.h+      souffle/ReadStream.h+      souffle/utility/ContainerUtil.h+      souffle/utility/FileUtil.h+      souffle/gzfstream.h+      souffle/CompiledTuple.h+      souffle/RecordTable.h+      souffle/Brie.h+      souffle/utility/CacheUtil.h+      souffle/SouffleInterface.h+      souffle/EventProcessor.h+      souffle/ProfileDatabase.h+      souffle/WriteStreamSQLite.h+      souffle/WriteStream.h+      souffle/LambdaBTree.h+      souffle/BTree.h+      souffle/utility/ParallelUtil.h+      souffle/WriteStreamCSV.h+      souffle/utility/tinyformat.h+      souffle/IOSystem.h+      souffle/ReadStreamSQLite.h+      souffle/UnionFind.h+      souffle/PiggyList.h+      souffle/ProfileEvent.h+      souffle/SerialisationStream.h+      souffle/CompiledSouffle.h+      souffle/EquivalenceRelation.h+      souffle/IterUtils.h+      souffle/SignalHandler.h+      souffle/Table.h+      souffle/utility/EvaluatorUtil.h+      souffle/CompiledOptions.h+      souffle/Logger.h+  build-depends:+      base >=4.12 && <5+    , containers >=0.6.2.1 && <1+    , deepseq >=1.4.4 && <2+    , directory >=1.3.3 && <2+    , extra >=1.6.18 && <2+    , filepath >=1.4.2 && <2+    , megaparsec >=7.0.5 && <8+    , mtl >=2.0 && <3+    , process >=1.6 && <2+    , souffle-haskell+    , template-haskell >=2 && <3+    , temporary >=1.3 && <2+    , text >=1.0 && <2+    , type-errors-pretty >=0.0.1.0 && <1+    , vector <=1.0+  default-language: Haskell2010+ test-suite souffle-haskell-test   type: exitcode-stdio-1.0   main-is: test.hs@@ -81,7 +239,53 @@       tests   default-extensions: OverloadedStrings LambdaCase ScopedTypeVariables   ghc-options: -Wall -Weverything -Wno-safe -Wno-unsafe -Wno-implicit-prelude -Wno-missed-specializations -Wno-all-missed-specializations -Wno-missing-import-lists -Wno-type-defaults -Wno-missing-local-signatures -Wno-monomorphism-restriction -Wno-missing-deriving-strategies -optP-Wno-nonportable-include-path -fhide-source-paths -fno-show-valid-hole-fits -fno-sort-valid-hole-fits-  cpp-options: -std=c++17 -D__EMBEDDED_SOUFFLE__+  cxx-options: -std=c++17 -D__EMBEDDED_SOUFFLE__+  include-dirs:+      cbits+      cbits/souffle+  install-includes:+      souffle/RamTypes.h+      souffle/utility/MiscUtil.h+      souffle/SymbolTable.h+      souffle/json11.h+      souffle/utility/FunctionalUtil.h+      souffle/utility/StreamUtil.h+      souffle/utility/StringUtil.h+      souffle/ReadStreamCSV.h+      souffle/ReadStream.h+      souffle/utility/ContainerUtil.h+      souffle/utility/FileUtil.h+      souffle/gzfstream.h+      souffle/CompiledTuple.h+      souffle/RecordTable.h+      souffle/Brie.h+      souffle/utility/CacheUtil.h+      souffle/SouffleInterface.h+      souffle/EventProcessor.h+      souffle/ProfileDatabase.h+      souffle/WriteStreamSQLite.h+      souffle/WriteStream.h+      souffle/LambdaBTree.h+      souffle/BTree.h+      souffle/utility/ParallelUtil.h+      souffle/WriteStreamCSV.h+      souffle/utility/tinyformat.h+      souffle/IOSystem.h+      souffle/ReadStreamSQLite.h+      souffle/UnionFind.h+      souffle/PiggyList.h+      souffle/ProfileEvent.h+      souffle/SerialisationStream.h+      souffle/CompiledSouffle.h+      souffle/EquivalenceRelation.h+      souffle/IterUtils.h+      souffle/SignalHandler.h+      souffle/Table.h+      souffle/utility/EvaluatorUtil.h+      souffle/CompiledOptions.h+      souffle/Logger.h+  cxx-sources:+      tests/fixtures/path.cpp   build-depends:       base >=4.12 && <5     , deepseq >=1.4.4 && <2
tests/Test/Language/Souffle/CompiledSpec.hs view
@@ -1,5 +1,4 @@-{-# LANGUAGE TemplateHaskell, ScopedTypeVariables, DataKinds #-}-{-# LANGUAGE TypeFamilies, DeriveGeneric #-}+{-# LANGUAGE ScopedTypeVariables, DataKinds, TypeFamilies, DeriveGeneric #-}  module Test.Language.Souffle.CompiledSpec   ( module Test.Language.Souffle.CompiledSpec@@ -9,10 +8,7 @@ import GHC.Generics import Data.Maybe import qualified Data.Vector as V-import qualified Language.Souffle.TH as Souffle import qualified Language.Souffle as Souffle--Souffle.embedProgram "tests/fixtures/path.cpp"  data Path = Path
tests/Test/Language/Souffle/InterpretedSpec.hs view
@@ -7,7 +7,11 @@  import Test.Hspec import GHC.Generics+import Control.Monad (join) import Data.Maybe+import Control.Monad.IO.Class (liftIO)+import System.Directory+import System.IO.Temp import qualified Data.Vector as V import qualified Language.Souffle.Interpreted as Souffle @@ -45,6 +49,10 @@   type ProgramFacts BadPath = [Edge, Reachable]   programName = const "bad_path" +getTestTemporaryDirectory :: IO FilePath+getTestTemporaryDirectory = do+  tmpDir <- getCanonicalTemporaryDirectory+  createTempDirectory tmpDir "souffle-haskell-test"  spec :: Spec spec = describe "Souffle API" $ parallel $ do@@ -91,13 +99,30 @@         pure results       edges `shouldBe` ([] :: [Edge]) +    it "can retrieve facts from custom output directory" $ do+      cfg <- Souffle.defaultConfig+      tmp <- getTestTemporaryDirectory+      (edges, reachables) <- Souffle.runSouffleWith (cfg { Souffle.cfgOutputDir = Just tmp }) $ do+        prog <- fromJust <$> Souffle.init PathNoInput+        Souffle.run prog+        es <- Souffle.getFacts prog+        rs <- Souffle.getFacts prog+        Souffle.cleanup prog+        pure (es , rs)+      edges `shouldBe` V.fromList [Edge "a" "b", Edge "b" "c"]+      reachables `shouldBe` V.fromList [Reachable "a" "b", Reachable "a" "c", Reachable "b" "c"]+      outputDirExist <- doesDirectoryExist tmp+      outputDirExist `shouldBe` False+   describe "addFact" $ parallel $ do     it "adds a fact" $ do       edges <- Souffle.runSouffle $ do         prog <- fromJust <$> Souffle.init Path         Souffle.addFact prog $ Edge "e" "f"         Souffle.run prog-        Souffle.getFacts prog+        edges <- Souffle.getFacts prog+        Souffle.cleanup prog+        pure edges       edges `shouldBe` [Edge "a" "b", Edge "b" "c", Edge "e" "f"]      -- NOTE: this is different compared to compiled version (bug in Souffle?)@@ -106,10 +131,26 @@         prog <- fromJust <$> Souffle.init PathNoInput         Souffle.addFact prog $ Reachable "e" "f"         Souffle.run prog-        Souffle.getFacts prog+        reachables <- Souffle.getFacts prog+        Souffle.cleanup prog+        pure reachables       reachables `shouldBe`         [ Reachable "a" "b", Reachable "a" "c", Reachable "b" "c" ] +    it "adds a fact to a custom input directory" $ do+      cfg <- Souffle.defaultConfig+      tmp <- getTestTemporaryDirectory+      join $ Souffle.runSouffleWith (cfg { Souffle.cfgFactDir = Just tmp }) $ do -- Souffle+        prog <- fromJust <$> Souffle.init Path+        Souffle.addFact prog $ Edge "e" "f"+        Souffle.run prog+        edges <- Souffle.getFacts prog+        entries <- liftIO $ listDirectory tmp+        Souffle.cleanup prog+        pure $ do -- IO+          edges `shouldBe` [Edge "a" "b", Edge "b" "c", Edge "e" "f"]+          length entries `shouldNotBe` 0+   describe "addFacts" $ parallel $     it "can add multiple facts at once" $ do       edges <- Souffle.runSouffle $ do@@ -148,6 +189,17 @@         [ Reachable "a" "b", Reachable "a" "c", Reachable "a" "d"         , Reachable "a" "e", Reachable "b" "c", Reachable "b" "d"         , Reachable "b" "e", Reachable "c" "d" ]++    it "saves stdout and stderr output after run" $ do+      (stdout, stderr) <- Souffle.runSouffle $ do+        prog <- fromJust <$> Souffle.init PathNoInput+        Souffle.run prog+        out <- Souffle.souffleStdOut prog+        err <- Souffle.souffleStdErr prog+        Souffle.cleanup prog+        pure (out, err)+      stdout `shouldBe` Just ""+      stderr `shouldBe` Just ""    describe "configuring number of cores" $ parallel $     it "is possible to configure number of cores" $ do
+ tests/fixtures/path.cpp view
@@ -0,0 +1,478 @@++#include "souffle/CompiledSouffle.h"++extern "C" {+}++namespace souffle {+static const RamDomain RAM_BIT_SHIFT_MASK = RAM_DOMAIN_SIZE - 1;+struct t_btree_2__0_1__01__11 {+using t_tuple = Tuple<RamDomain, 2>;+struct t_comparator_0{+ int operator()(const t_tuple& a, const t_tuple& b) const {+  return (a[0] < b[0]) ? -1 : ((a[0] > b[0]) ? 1 :((a[1] < b[1]) ? -1 : ((a[1] > b[1]) ? 1 :(0))));+ }+bool less(const t_tuple& a, const t_tuple& b) const {+  return  a[0] < b[0]|| (a[0] == b[0] && ( a[1] < b[1]));+ }+bool equal(const t_tuple& a, const t_tuple& b) const {+return a[0] == b[0]&&a[1] == b[1];+ }+};+using t_ind_0 = btree_set<t_tuple,t_comparator_0>;+t_ind_0 ind_0;+using iterator = t_ind_0::iterator;+struct context {+t_ind_0::operation_hints hints_0;+};+context createContext() { return context(); }+bool insert(const t_tuple& t) {+context h;+return insert(t, h);+}+bool insert(const t_tuple& t, context& h) {+if (ind_0.insert(t, h.hints_0)) {+return true;+} else return false;+}+bool insert(const RamDomain* ramDomain) {+RamDomain data[2];+std::copy(ramDomain, ramDomain + 2, data);+const t_tuple& tuple = reinterpret_cast<const t_tuple&>(data);+context h;+return insert(tuple, h);+}+bool insert(RamDomain a0,RamDomain a1) {+RamDomain data[2] = {a0,a1};+return insert(data);+}+bool contains(const t_tuple& t, context& h) const {+return ind_0.contains(t, h.hints_0);+}+bool contains(const t_tuple& t) const {+context h;+return contains(t, h);+}+std::size_t size() const {+return ind_0.size();+}+iterator find(const t_tuple& t, context& h) const {+return ind_0.find(t, h.hints_0);+}+iterator find(const t_tuple& t) const {+context h;+return find(t, h);+}+range<iterator> lowerUpperRange_00(const t_tuple& lower, const t_tuple& upper, context& h) const {+return range<iterator>(ind_0.begin(),ind_0.end());+}+range<iterator> lowerUpperRange_00(const t_tuple& lower, const t_tuple& upper) const {+return range<iterator>(ind_0.begin(),ind_0.end());+}+range<t_ind_0::iterator> lowerUpperRange_01(const t_tuple& lower, const t_tuple& upper, context& h) const {+t_tuple low(lower); t_tuple high(lower);+low[1] = MIN_RAM_SIGNED;+high[1] = MAX_RAM_SIGNED;+return make_range(ind_0.lower_bound(low, h.hints_0), ind_0.upper_bound(high, h.hints_0));+}+range<t_ind_0::iterator> lowerUpperRange_01(const t_tuple& lower, const t_tuple& upper) const {+context h;+return lowerUpperRange_01(lower,upper,h);+}+range<t_ind_0::iterator> lowerUpperRange_11(const t_tuple& lower, const t_tuple& upper, context& h) const {+auto pos = ind_0.find(lower, h.hints_0);+auto fin = ind_0.end();+if (pos != fin) {fin = pos; ++fin;}+return make_range(pos, fin);+}+range<t_ind_0::iterator> lowerUpperRange_11(const t_tuple& lower, const t_tuple& upper) const {+context h;+return lowerUpperRange_11(lower,upper,h);+}+bool empty() const {+return ind_0.empty();+}+std::vector<range<iterator>> partition() const {+return ind_0.getChunks(400);+}+void purge() {+ind_0.clear();+}+iterator begin() const {+return ind_0.begin();+}+iterator end() const {+return ind_0.end();+}+void printStatistics(std::ostream& o) const {+o << " arity 2 direct b-tree index 0 lex-order [0,1]\n";+ind_0.printStats(o);+}+};+struct t_btree_2__0_1__11 {+using t_tuple = Tuple<RamDomain, 2>;+struct t_comparator_0{+ int operator()(const t_tuple& a, const t_tuple& b) const {+  return (a[0] < b[0]) ? -1 : ((a[0] > b[0]) ? 1 :((a[1] < b[1]) ? -1 : ((a[1] > b[1]) ? 1 :(0))));+ }+bool less(const t_tuple& a, const t_tuple& b) const {+  return  a[0] < b[0]|| (a[0] == b[0] && ( a[1] < b[1]));+ }+bool equal(const t_tuple& a, const t_tuple& b) const {+return a[0] == b[0]&&a[1] == b[1];+ }+};+using t_ind_0 = btree_set<t_tuple,t_comparator_0>;+t_ind_0 ind_0;+using iterator = t_ind_0::iterator;+struct context {+t_ind_0::operation_hints hints_0;+};+context createContext() { return context(); }+bool insert(const t_tuple& t) {+context h;+return insert(t, h);+}+bool insert(const t_tuple& t, context& h) {+if (ind_0.insert(t, h.hints_0)) {+return true;+} else return false;+}+bool insert(const RamDomain* ramDomain) {+RamDomain data[2];+std::copy(ramDomain, ramDomain + 2, data);+const t_tuple& tuple = reinterpret_cast<const t_tuple&>(data);+context h;+return insert(tuple, h);+}+bool insert(RamDomain a0,RamDomain a1) {+RamDomain data[2] = {a0,a1};+return insert(data);+}+bool contains(const t_tuple& t, context& h) const {+return ind_0.contains(t, h.hints_0);+}+bool contains(const t_tuple& t) const {+context h;+return contains(t, h);+}+std::size_t size() const {+return ind_0.size();+}+iterator find(const t_tuple& t, context& h) const {+return ind_0.find(t, h.hints_0);+}+iterator find(const t_tuple& t) const {+context h;+return find(t, h);+}+range<iterator> lowerUpperRange_00(const t_tuple& lower, const t_tuple& upper, context& h) const {+return range<iterator>(ind_0.begin(),ind_0.end());+}+range<iterator> lowerUpperRange_00(const t_tuple& lower, const t_tuple& upper) const {+return range<iterator>(ind_0.begin(),ind_0.end());+}+range<t_ind_0::iterator> lowerUpperRange_11(const t_tuple& lower, const t_tuple& upper, context& h) const {+auto pos = ind_0.find(lower, h.hints_0);+auto fin = ind_0.end();+if (pos != fin) {fin = pos; ++fin;}+return make_range(pos, fin);+}+range<t_ind_0::iterator> lowerUpperRange_11(const t_tuple& lower, const t_tuple& upper) const {+context h;+return lowerUpperRange_11(lower,upper,h);+}+bool empty() const {+return ind_0.empty();+}+std::vector<range<iterator>> partition() const {+return ind_0.getChunks(400);+}+void purge() {+ind_0.clear();+}+iterator begin() const {+return ind_0.begin();+}+iterator end() const {+return ind_0.end();+}+void printStatistics(std::ostream& o) const {+o << " arity 2 direct b-tree index 0 lex-order [0,1]\n";+ind_0.printStats(o);+}+};++class Sf_path : public SouffleProgram {+private:+static inline bool regex_wrapper(const std::string& pattern, const std::string& text) {+   bool result = false;+   try { result = std::regex_match(text, std::regex(pattern)); } catch(...) {+     std::cerr << "warning: wrong pattern provided for match(\"" << pattern << "\",\"" << text << "\").\n";+}+   return result;+}+private:+static inline std::string substr_wrapper(const std::string& str, size_t idx, size_t len) {+   std::string result;+   try { result = str.substr(idx,len); } catch(...) {+     std::cerr << "warning: wrong index position provided by substr(\"";+     std::cerr << str << "\"," << (int32_t)idx << "," << (int32_t)len << ") functor.\n";+   } return result;+}+public:+// -- initialize symbol table --+SymbolTable symTable{+	R"_(a)_",+	R"_(b)_",+	R"_(c)_",+};// -- initialize record table --+RecordTable recordTable;+// -- Table: @delta_reachable+std::unique_ptr<t_btree_2__0_1__01__11> rel_1_delta_reachable = std::make_unique<t_btree_2__0_1__01__11>();+// -- Table: @new_reachable+std::unique_ptr<t_btree_2__0_1__01__11> rel_2_new_reachable = std::make_unique<t_btree_2__0_1__01__11>();+// -- Table: edge+std::unique_ptr<t_btree_2__0_1__11> rel_3_edge = std::make_unique<t_btree_2__0_1__11>();+souffle::RelationWrapper<0,t_btree_2__0_1__11,Tuple<RamDomain,2>,2,0> wrapper_rel_3_edge;+// -- Table: reachable+std::unique_ptr<t_btree_2__0_1__11> rel_4_reachable = std::make_unique<t_btree_2__0_1__11>();+souffle::RelationWrapper<1,t_btree_2__0_1__11,Tuple<RamDomain,2>,2,0> wrapper_rel_4_reachable;+public:+Sf_path() :+wrapper_rel_3_edge(*rel_3_edge,symTable,"edge",std::array<const char *,2>{{"s:symbol","s:symbol"}},std::array<const char *,2>{{"n","m"}}),++wrapper_rel_4_reachable(*rel_4_reachable,symTable,"reachable",std::array<const char *,2>{{"s:symbol","s:symbol"}},std::array<const char *,2>{{"n","m"}}){+addRelation("edge",&wrapper_rel_3_edge,true,true);+addRelation("reachable",&wrapper_rel_4_reachable,false,true);+}+~Sf_path() {+}+private:+std::string inputDirectory;+std::string outputDirectory;+bool performIO;+std::atomic<RamDomain> ctr{};++std::atomic<size_t> iter{};+void runFunction(std::string inputDirectory = ".", std::string outputDirectory = ".", bool performIO = false) {+this->inputDirectory = inputDirectory;+this->outputDirectory = outputDirectory;+this->performIO = performIO;+SignalHandler::instance()->set();+#if defined(_OPENMP)+if (getNumThreads() > 0) {omp_set_num_threads(getNumThreads());}+#endif++// -- query evaluation --+{+ std::vector<RamDomain> args, ret;+subroutine_0(args, ret);+}+{+ std::vector<RamDomain> args, ret;+subroutine_1(args, ret);+}++// -- relation hint statistics --+SignalHandler::instance()->reset();+}+public:+void run() override { runFunction(".", ".", false); }+public:+void runAll(std::string inputDirectory = ".", std::string outputDirectory = ".") override { runFunction(inputDirectory, outputDirectory, true);+}+public:+void printAll(std::string outputDirectory = ".") override {+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"attributeNames","n\tm"},{"filename","./edge.csv"},{"name","edge"},{"operation","output"},{"types","{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!outputDirectory.empty() && directiveMap["IO"] == "file" && directiveMap["filename"].front() != '/') {directiveMap["filename"] = outputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getWriter(directiveMap, symTable, recordTable)->writeAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"attributeNames","n\tm"},{"filename","./reachable.csv"},{"name","reachable"},{"operation","output"},{"types","{\"reachable\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!outputDirectory.empty() && directiveMap["IO"] == "file" && directiveMap["filename"].front() != '/') {directiveMap["filename"] = outputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getWriter(directiveMap, symTable, recordTable)->writeAll(*rel_4_reachable);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+}+public:+void loadAll(std::string inputDirectory = ".") override {+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"filename","./edge.facts"},{"name","edge"},{"operation","input"},{"types","{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!inputDirectory.empty() && directiveMap["IO"] == "file" && directiveMap["filename"].front() != '/') {directiveMap["filename"] = inputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getReader(directiveMap, symTable, recordTable)->readAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << "Error loading data: " << e.what() << '\n';}+}+public:+void dumpInputs(std::ostream& out = std::cout) override {+try {std::map<std::string, std::string> rwOperation;+rwOperation["IO"] = "stdout";+rwOperation["name"] = "edge";+rwOperation["types"] = "{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}}";+IOSystem::getInstance().getWriter(rwOperation, symTable, recordTable)->writeAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+}+public:+void dumpOutputs(std::ostream& out = std::cout) override {+try {std::map<std::string, std::string> rwOperation;+rwOperation["IO"] = "stdout";+rwOperation["name"] = "edge";+rwOperation["types"] = "{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}}";+IOSystem::getInstance().getWriter(rwOperation, symTable, recordTable)->writeAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+try {std::map<std::string, std::string> rwOperation;+rwOperation["IO"] = "stdout";+rwOperation["name"] = "reachable";+rwOperation["types"] = "{\"reachable\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}}";+IOSystem::getInstance().getWriter(rwOperation, symTable, recordTable)->writeAll(*rel_4_reachable);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+}+public:+SymbolTable& getSymbolTable() override {+return symTable;+}+void executeSubroutine(std::string name, const std::vector<RamDomain>& args, std::vector<RamDomain>& ret) override {+if (name == "stratum_0") {+subroutine_0(args, ret);+return;}+if (name == "stratum_1") {+subroutine_1(args, ret);+return;}+fatal("unknown subroutine");+}+void subroutine_0(const std::vector<RamDomain>& args, std::vector<RamDomain>& ret) {+if (performIO) {+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"filename","./edge.facts"},{"name","edge"},{"operation","input"},{"types","{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!inputDirectory.empty() && directiveMap["filename"].front() != '/') {directiveMap["filename"] = inputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getReader(directiveMap, symTable, recordTable)->readAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << "Error loading data: " << e.what() << '\n';}+}+SignalHandler::instance()->setMsg(R"_(edge("a","b").+in file /Users/luc/personal/souffle-hs/tests/fixtures/path.dl [11:1-11:16])_");+[&](){+CREATE_OP_CONTEXT(rel_3_edge_op_ctxt,rel_3_edge->createContext());+Tuple<RamDomain,2> tuple{{ramBitCast(RamSigned(0)),ramBitCast(RamSigned(1))}};+rel_3_edge->insert(tuple,READ_OP_CONTEXT(rel_3_edge_op_ctxt));+}+();SignalHandler::instance()->setMsg(R"_(edge("b","c").+in file /Users/luc/personal/souffle-hs/tests/fixtures/path.dl [12:1-12:16])_");+[&](){+CREATE_OP_CONTEXT(rel_3_edge_op_ctxt,rel_3_edge->createContext());+Tuple<RamDomain,2> tuple{{ramBitCast(RamSigned(1)),ramBitCast(RamSigned(2))}};+rel_3_edge->insert(tuple,READ_OP_CONTEXT(rel_3_edge_op_ctxt));+}+();if (performIO) {+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"attributeNames","n\tm"},{"filename","./edge.csv"},{"name","edge"},{"operation","output"},{"types","{\"edge\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!outputDirectory.empty() && directiveMap["filename"].front() != '/') {directiveMap["filename"] = outputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getWriter(directiveMap, symTable, recordTable)->writeAll(*rel_3_edge);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+}+}+void subroutine_1(const std::vector<RamDomain>& args, std::vector<RamDomain>& ret) {+SignalHandler::instance()->setMsg(R"_(reachable(x,y) :-+   edge(x,y).+in file /Users/luc/personal/souffle-hs/tests/fixtures/path.dl [14:1-14:31])_");+if(!(rel_3_edge->empty())) {+[&](){+CREATE_OP_CONTEXT(rel_3_edge_op_ctxt,rel_3_edge->createContext());+CREATE_OP_CONTEXT(rel_4_reachable_op_ctxt,rel_4_reachable->createContext());+for(const auto& env0 : *rel_3_edge) {+Tuple<RamDomain,2> tuple{{ramBitCast(env0[0]),ramBitCast(env0[1])}};+rel_4_reachable->insert(tuple,READ_OP_CONTEXT(rel_4_reachable_op_ctxt));+}+}+();}+[&](){+CREATE_OP_CONTEXT(rel_4_reachable_op_ctxt,rel_4_reachable->createContext());+CREATE_OP_CONTEXT(rel_1_delta_reachable_op_ctxt,rel_1_delta_reachable->createContext());+for(const auto& env0 : *rel_4_reachable) {+Tuple<RamDomain,2> tuple{{ramBitCast(env0[0]),ramBitCast(env0[1])}};+rel_1_delta_reachable->insert(tuple,READ_OP_CONTEXT(rel_1_delta_reachable_op_ctxt));+}+}+();iter = 0;+for(;;) {+SignalHandler::instance()->setMsg(R"_(reachable(x,z) :-+   edge(x,y),+   reachable(y,z).+in file /Users/luc/personal/souffle-hs/tests/fixtures/path.dl [15:1-15:48])_");+if(!(rel_3_edge->empty()) && !(rel_1_delta_reachable->empty())) {+[&](){+CREATE_OP_CONTEXT(rel_3_edge_op_ctxt,rel_3_edge->createContext());+CREATE_OP_CONTEXT(rel_4_reachable_op_ctxt,rel_4_reachable->createContext());+CREATE_OP_CONTEXT(rel_1_delta_reachable_op_ctxt,rel_1_delta_reachable->createContext());+CREATE_OP_CONTEXT(rel_2_new_reachable_op_ctxt,rel_2_new_reachable->createContext());+for(const auto& env0 : *rel_3_edge) {+const Tuple<RamDomain,2> lower{{ramBitCast(env0[1]),0}};+const Tuple<RamDomain,2> upper{{ramBitCast(env0[1]),0}};+auto range = rel_1_delta_reachable->lowerUpperRange_01(lower, upper,READ_OP_CONTEXT(rel_1_delta_reachable_op_ctxt));+for(const auto& env1 : range) {+if( !(rel_4_reachable->contains(Tuple<RamDomain,2>{{ramBitCast(env0[0]),ramBitCast(env1[1])}},READ_OP_CONTEXT(rel_4_reachable_op_ctxt)))) {+Tuple<RamDomain,2> tuple{{ramBitCast(env0[0]),ramBitCast(env1[1])}};+rel_2_new_reachable->insert(tuple,READ_OP_CONTEXT(rel_2_new_reachable_op_ctxt));+}+}+}+}+();}+if(rel_2_new_reachable->empty()) break;+[&](){+CREATE_OP_CONTEXT(rel_4_reachable_op_ctxt,rel_4_reachable->createContext());+CREATE_OP_CONTEXT(rel_2_new_reachable_op_ctxt,rel_2_new_reachable->createContext());+for(const auto& env0 : *rel_2_new_reachable) {+Tuple<RamDomain,2> tuple{{ramBitCast(env0[0]),ramBitCast(env0[1])}};+rel_4_reachable->insert(tuple,READ_OP_CONTEXT(rel_4_reachable_op_ctxt));+}+}+();std::swap(rel_1_delta_reachable, rel_2_new_reachable);+rel_2_new_reachable->purge();+iter++;+}+iter = 0;+rel_1_delta_reachable->purge();+rel_2_new_reachable->purge();+if (performIO) {+try {std::map<std::string, std::string> directiveMap({{"IO","file"},{"attributeNames","n\tm"},{"filename","./reachable.csv"},{"name","reachable"},{"operation","output"},{"types","{\"reachable\": {\"arity\": 2, \"auxArity\": 0, \"types\": [\"s:symbol\", \"s:symbol\"]}, \"records\": {}}"}});+if (!outputDirectory.empty() && directiveMap["filename"].front() != '/') {directiveMap["filename"] = outputDirectory + "/" + directiveMap["filename"];}+IOSystem::getInstance().getWriter(directiveMap, symTable, recordTable)->writeAll(*rel_4_reachable);+} catch (std::exception& e) {std::cerr << e.what();exit(1);}+}+if (performIO) rel_3_edge->purge();+if (performIO) rel_4_reachable->purge();+}+};+SouffleProgram *newInstance_path(){return new Sf_path;}+SymbolTable *getST_path(SouffleProgram *p){return &reinterpret_cast<Sf_path*>(p)->symTable;}++#ifdef __EMBEDDED_SOUFFLE__+class factory_Sf_path: public souffle::ProgramFactory {+SouffleProgram *newInstance() {+return new Sf_path();+};+public:+factory_Sf_path() : ProgramFactory("path"){}+};+extern "C" {+factory_Sf_path __factory_Sf_path_instance;+}+}+#else+}+int main(int argc, char** argv)+{+try{+souffle::CmdOptions opt(R"(path.dl)",+R"(.)",+R"(.)",+false,+R"()",+1);+if (!opt.parse(argc,argv)) return 1;+souffle::Sf_path obj;+#if defined(_OPENMP)+obj.setNumThreads(opt.getNumJobs());++#endif+obj.runAll(opt.getInputFileDir(), opt.getOutputFileDir());+return 0;+} catch(std::exception &e) { souffle::SignalHandler::instance()->error(e.what());}+}++#endif