packages feed

futhark 0.28.2 → 0.29.1

raw patch · 61 files changed

+3279/−2181 lines, 61 files

Files

CHANGELOG.md view
@@ -5,6 +5,51 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [0.29.1]++### Added++* Better GPU code generation for reductions with very complicated operators.++* Optimised internal book-keeping of GPU operations, resulting in lower static+  memory usage for large programs.++### Changed++* The flattening attributes have changed somewhat. `#[sequential_inner]`+  replaced by `#[flattening(only_outer)]`, and `#[flattening(only_inner)]` has+  been renamed to `#[flattening(only_full)]`, which now also affects any nested+  parallelism.++### Fixed++* Invalid memory expansion in some rare cases.++* The ISPC backend would handle `f16` incorrectly in some cases.++* Fixed a very rare out-of-bounds access in scans with array operands on GPU+  backends.++* Invalid handling of aliasing in flattening. (#2595)++* Invalid short-circuiting could cause a compiler crash. (#2597)++* Execution errors with array-operator scans with nonstandard tuning settings.++* On GPU backends, a failing safety check in code fused with a scan could cause+  barrier divergence or deadlocks.++* On the `cuda` and `hip` backends, a failing safety check in a kernel with+  barriers stopped only the failing thread rather than its whole thread block,+  which could cause deadlocks.++* On GPU backends, reductions and scans with operators that contain assertions+  could cause barrier divergence.++* On GPU and multicore backends, a failing operator in a `reduce_by_index` or+  `hist` that is implemented with locks could leave a lock held, causing a+  deadlock.+ ## [0.28.2]  ### Added
docs/installation.rst view
@@ -79,7 +79,7 @@  Tarballs of binary releases can be `found online <https://futhark-lang.org/releases/>`_, but are available only for-very few platforms (as of this writing, only GNU/Linux on x86_64).+very few platforms (GNU/Linux on x86_64 and aarch64). See the enclosed ``README.md`` for installation instructions.  Furthermore, every day a program automatically clones the Git@@ -96,8 +96,11 @@ **Linux (x86_64)**   `futhark-nightly-linux-x86_64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-x86_64.tar.xz>`_ -**macOS (ARM64)**-  `futhark-nightly-macos-arm64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-arm64.tar.xz>`_+**Linux (Aarch64)**+  `futhark-nightly-linux-aarch64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-aarch64.tar.xz>`_++**macOS (AARCH64)**+  `futhark-nightly-macos-aarch64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-aarch64.tar.xz>`_  **Windows (x86_64)**   `futhark-nightly-windows-x86_64.zip <https://futhark-lang.org/releases/futhark-nightly-windows-x86_64.zip>`_
docs/js-api.rst view
@@ -3,69 +3,36 @@ JavaScript API Reference ======================== -The :ref:`futhark-wasm(1)`, :ref:`futhark-wasm-multicore(1)`, and-:ref:`futhark-webgpu(1)` compilers can generate JavaScript wrapper code-when used with ``--library``.  This chapter describes the API exposed by-those wrappers.--The exact generated files and the top-level JavaScript interface differ-somewhat between the WASM backend and the WebGPU backend, so the-relevant differences are described below.+The :ref:`futhark-wasm(1)` and :ref:`futhark-wasm-multicore(1)` compilers can+generate JavaScript wrapper code when used with ``--library``. This chapter+describes the API exposed by those wrappers. -First a warning: **the JavaScript API is experimental**.  It may-change incompatibly even in minor versions of the compiler.+First a warning: **the JavaScript API is experimental**. It may change+incompatibly even in minor versions of the compiler.  A Futhark program ``futlib.fut`` compiled with ``--library`` produces-generated C code as well as JavaScript-facing runtime files.  Some of-these files are backend-specific.--For the WASM backend, a build typically produces:--* ``futlib.c``, ``futlib.h``: implementation and header C files-  generated by the compiler, similar to ``futhark c``.-* ``futlib.class.js``: an intermediate JavaScript build artifact.-* ``futlib.wasm``: the compiled WebAssembly module, which must be-  present at runtime.-* ``futlib.mjs``: an ES module that exports the JavaScript wrapper API.--For the WebGPU backend, a build typically produces:--* ``futlib.c``, ``futlib.h``: implementation and header C files-  generated by the compiler.-* ``futlib.js``: the generated backend runtime module.-* ``futlib.wrapper.js``: the JavaScript wrapper that exposes the-  Futhark-facing API.-* ``futlib.json``: manifest data used by the generated wrapper.-* ``futlib.wasm``: a generated WebAssembly artifact used by the runtime.--The exact file set is backend-dependent and may change over time.+the following files: -The generated JavaScript wrapper exposes a ``FutharkModule`` class,-which is the preferred top-level interface for using a compiled Futhark-program from JavaScript.  The module is initialised with the generated-backend runtime module.+* ``futlib.mjs``: an ES module that exports the JavaScript API.+* ``futlib.wasm``: the compiled WebAssembly module, which is loaded by+  ``futlib.mjs`` and must be in the same directory.+* ``futlib.c`` and ``futlib.class.js``: intermediate files, which are+  not needed at runtime. -The initialisation pattern is the same for the WASM-style backend and-the WebGPU backend. The only difference is how ``Module`` is made available.-For the WASM-style backend, the generated ``.mjs`` file exports it as the-default export, so it is imported together with ``FutharkModule``:+The module exports a ``FutharkContext`` class, which is the top-level+interface for using a compiled Futhark program from JavaScript:  .. code-block:: javascript -   import Module, { FutharkModule } from './futlib.mjs';--   const module = await Module();--   const fut = new FutharkModule();-   await fut.init(module);--For the WebGPU backend, the generated runtime module is usually loaded-before the application code, for example:+   import { FutharkContext } from './futlib.mjs'; -.. code-block:: html+   const fut = new FutharkContext();+   await fut.init(); -   <script src="futlib.js"></script>-   <script src="main.js"></script>+In a browser, the module must be loaded by a ``<script+type="module">`` element (or imported by another module), and the+files must be served over HTTP.  In Node.js, a CommonJS module can+load it with ``await import('./futlib.mjs')``.  General concerns ----------------@@ -75,102 +42,110 @@ You are responsible for eventually freeing all objects produced by the API, using the appropriate methods. +Methods that may run Futhark code are asynchronous, and return a+promise.  The asynchronous operations on a ``FutharkContext`` run one at+a time, in the order they were started, and while one is running, no+synchronous method may be called on the ``FutharkContext`` or its+values.  Errors, including errors that occur while running Futhark+code, are reported by throwing an ``Error`` (or rejecting the promise+with one).  The arguments of every method are checked, and an invalid+argument is reported with a ``TypeError`` or ``RangeError``.+ Top-level wrapper objects ------------------------- -The preferred top-level JavaScript interface is ``FutharkModule``.  A-``FutharkModule`` object represents an instance of a compiled Futhark-program. It owns the underlying Futhark context and is used to construct-arrays, call entry points, and free resources.+The top-level JavaScript interface is ``FutharkContext``.  A+``FutharkContext`` object represents an instance of a compiled Futhark+program, and corresponds to a context in the C API.  It is used to+construct arrays, call entry points, and free resources. -.. js:class:: FutharkModule()+.. js:class:: FutharkContext()     A bookkeeping class representing an instance of a compiled Futhark    program. -.. js:function:: FutharkModule.init(module)--   Asynchronously initialise the ``FutharkModule`` object.--   For the WASM-style backend, the ``module`` argument is optional. If-   omitted, the generated wrapper loads the WebAssembly module itself.--   For the WebGPU backend, ``module`` must be the generated backend-   runtime module.--.. js:function:: FutharkModule.free()+.. js:function:: FutharkContext.init(options) -   Frees the Futhark context and configuration associated with the-   module. It is an error to use a ``FutharkArray`` or ``FutharkOpaque``-   after the ``FutharkModule`` on which it was created has been freed.+   Asynchronously initialise the ``FutharkContext`` object.  This must+   be done exactly once, before the object is used for anything else.+   Each ``FutharkContext`` loads its own instance of the WebAssembly+   module, so values cannot be passed between different+   ``FutharkContext`` objects. -.. js:attribute:: FutharkModule.entry+   The ``options`` argument is optional.  If given, it must be an object+   with any of the following fields, which correspond to the+   configuration functions of the C API (see :ref:`c-api`): -   Object containing the generated entry point functions. For example,-   an entry point named ``main`` is available as ``fut.entry.main``.+   * ``debugging``, ``logging``, ``profiling``: booleans that enable+     debugging, logging, or profiling.  Logging information is printed+     with ``console.error``. -.. js:attribute:: FutharkModule.types+   * ``tuning_params``: an object mapping the names of tuning parameters+     to their values. -   Object containing generated type information and array constructor-   objects for the array types used by the program.+   * ``num_threads``: the number of threads to use.  Only supported by+     :ref:`futhark-wasm-multicore(1)`. -The backends also provides utility methods for synchronisation,-cache management, and profiling.+   With :ref:`futhark-wasm-multicore(1)`, ``init()`` fails unless+   ``SharedArrayBuffer`` is available, which in a browser requires the+   page to be *cross-origin isolated*. -.. js:function:: FutharkModule.context_sync()+.. js:function:: FutharkContext.free() -   Wait for pending backend work associated with the context to finish.+   Frees the underlying Futhark context and configuration.  It is an+   error to use a ``FutharkArray`` or ``FutharkOpaque`` after the+   ``FutharkContext`` on which it was created has been freed. -.. js:function:: FutharkModule.clear_caches()+.. js:attribute:: FutharkContext.entry -   Clear backend caches associated with the context.+   Object containing the entry point functions. For example, an entry+   point named ``main`` is available as ``fut.entry.main``. -.. js:function:: FutharkModule.report()+.. js:attribute:: FutharkContext.types -   Return a profiling report string.+   Object mapping the name of each array type used by the program, such+   as ``[]i32``, to its constructor object (see `Array construction`_). -.. js:function:: FutharkModule.pause_profiling()+The ``FutharkContext`` also provides utility methods for+synchronisation, cache management, and profiling.  They are all+asynchronous. -   Pause profiling.+.. js:function:: FutharkContext.context_sync() -.. js:function:: FutharkModule.unpause_profiling()+   Wait for pending work associated with the context to finish. -   Resume profiling.+.. js:function:: FutharkContext.clear_caches() -WASM compatibility interface-~~~~~~~~~~~~~~~~~~~~~~~~~~~~+   Clear caches associated with the context. -The WASM-style backend also keep the older ``FutharkContext`` interface-for backwards compatibility.+.. js:function:: FutharkContext.report() -.. js:function:: newFutharkContext()+   Return a profiling report string. -   Asynchronously create a new ``FutharkContext`` object.+.. js:function:: FutharkContext.pause_profiling() -.. js:class:: FutharkContext()+   Pause profiling. -   Backwards-compatible wrapper class for WASM-style backend.-   ``FutharkContext`` supports the same main interface as-   ``FutharkModule``, including ``entry``, ``types``, array constructors,-   and ``free``.+.. js:function:: FutharkContext.unpause_profiling() -New code should prefer ``FutharkModule``.+   Resume profiling.  Values ------  Numeric types ``u8``, ``u16``, ``u32``, ``i8``, ``i16``, ``i32``, ``f16``, ``f32``, and ``f64`` are mapped to JavaScript's standard number type.-64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``.  ``bool``-is mapped to JavaScript's ``boolean`` type.  Arrays are represented by-``FutharkArray`` objects.  Complex types (records, nested tuples, etc.)-are represented by ``FutharkOpaque`` objects.+64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``, although a+number is also accepted as an argument.  An integer argument must be in+the range of its type.  ``bool`` is mapped to JavaScript's ``boolean``+type.  Arrays are represented by ``FutharkArray`` objects.  Complex+types (records, nested tuples, etc.) are represented by+``FutharkOpaque`` objects.  FutharkArray ------------ -Arrays are represented by ``FutharkArray`` objects.  The common array API is-shared by the WASM backend and the WebGPU backend.+Arrays are represented by ``FutharkArray`` objects.  .. js:function:: FutharkArray.shape() @@ -178,61 +153,69 @@  .. js:function:: FutharkArray.values() -   Asynchronously returns a flat typed array containing the array data.--.. js:function:: FutharkArray.free()--   Frees the memory used by the ``FutharkArray``.--The WASM backend also provides a few older convenience methods.--.. js:function:: FutharkArray.toArray()--   Returns a nested JavaScript array.+   Asynchronously returns a flat typed array, in row-major order,+   containing a copy of the array elements.  It remains valid after the+   ``FutharkArray`` is freed.  The typed array corresponds to the+   element type, such as ``Int32Array`` for ``i32``, except that: -.. js:function:: FutharkArray.toTypedArray()+   * For ``f16``, it is a ``Float16Array`` if supported by the+     JavaScript implementation, and otherwise a ``Float32Array``. -   Returns a flat typed array of the underlying data.+   * For ``bool``, it is a ``Uint8Array`` whose elements are 0 or 1. -.. js:function:: FutharkArray.shape()+.. js:function:: FutharkArray.free() -   Returns the shape of the ``FutharkArray`` as an array of ``BigInt`` values.+   Frees the memory used by the ``FutharkArray``.  Array construction ~~~~~~~~~~~~~~~~~~ -Each array type used by an entry point is exposed through a generated-constructor object.  For example, for the type ``[]i32``:+Each array type used by an entry point, including as the field of a+record, is exposed through a generated constructor object.  For+example, for the type ``[]i32``:  .. js:function:: fut.i32_1d.from_data(data, dim0)     Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from-   a JavaScript ``Array`` or the corresponding typed array, with the-   given shape.--For the WASM backend, the generated constructor object also-provides a convenience method for nested JavaScript arrays.--.. js:function:: fut.i32_1d.from_jsarray(jsarray)+   a JavaScript ``Array`` or typed array in row-major order, with the+   given shape.  The dimensions can be numbers or ``BigInt``\ s, and+   their product must be the length of ``data``.  The elements must be+   valid entry point arguments of the element type, except that for+   ``bool``, they may also be 0 or 1 (as in the result of+   :js:func:`FutharkArray.values`). -   Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from-   a JavaScript array.  For higher-dimensional arrays, the JavaScript-   array is expected to be nested according to the array rank.+The constructor object for an array type with ``n`` dimensions has a+``from_data`` method that takes ``n`` dimensions.  FutharkOpaque -------------  Complex types (records, nested tuples, etc) are represented by-``FutharkOpaque``.  It has no use outside of being accepted and-returned by entry point functions. For this reason the method only has-one function for freeing the memory when ``FutharkOpaque`` is no-longer used.+``FutharkOpaque``.  They can be passed to and returned by entry point+functions, and the fields of records (including tuples) can be read+by projection, as described below.  .. js:function:: FutharkOpaque.free() -   Frees  memory used by FutharkOpaque. Should be called when Futhark-   Opaque is no longer used.+   Frees the memory used by the ``FutharkOpaque``. +Records+~~~~~~~++Records, including tuples, are ``FutharkOpaque`` objects whose fields+can be *projected*.  For each field of a record type, the top-level+wrapper object has a method named after the corresponding projection+function in the C API (see :ref:`records`), without the ``futhark_``+prefix.  For example, the field ``x`` of a record with C type ``struct+futhark_opaque_foo`` is projected as follows:++.. js:function:: fut.project_opaque_foo_x(obj)++   Asynchronously returns the value of field ``x`` of the record+   ``obj``.  An array or opaque field is returned as a new+   ``FutharkArray`` or ``FutharkOpaque``, which must be freed+   independently of ``obj``.+ Entry Points ------------ @@ -246,8 +229,13 @@  .. js:function:: fut.entry.<entry_point_name>(in1, ..., inN) -   The entry point function takes the JavaScript representations of the-   Futhark entry point arguments and returns the result.+   Asynchronously call the entry point with the JavaScript+   representations of its arguments, and return the result. -   If the entry point has multiple outputs, the return value is an array-   containing the outputs in order.+   An entry point that returns a tuple returns a single+   ``FutharkOpaque``, whose elements can be obtained by projection (see+   `Records`_).++   An argument that the entry point consumes (see+   :ref:`in-place-updates`) can afterwards only be freed, and must not+   also be passed as another argument.
docs/language-reference.rst view
@@ -717,11 +717,13 @@ Slicing of multiple dimensions can be done by separating with commas, and may be intermixed freely with indexing. -If ``s`` is elided it defaults to ``1``.  If ``i`` or ``j`` is elided, their-value depends on the sign of ``s``.  If ``s`` is positive, ``i`` become ``0``-and ``j`` become the length of the array.  If ``s`` is negative, ``i`` becomes-the length of the array minus one, and ``j`` becomes minus one.  This means that-``a[::-1]`` is the reverse of the array ``a``.+If ``s`` is elided it defaults to ``1``. If ``i`` or ``j`` is elided, their+value depends on the sign of ``s``. If ``s`` is positive, ``i`` becomes ``0`` and+``j`` becomes the length of the array. If ``s`` is negative, ``i`` becomes the+length of the array minus one, and ``j`` becomes minus one. This means that+``a[::-1]`` is the reverse of the array ``a``, and that ``a[:]`` retrieves the+entire dimension - this is only useful when slicing multiple dimensions. Note+that ``a[::]`` is not syntactically valid.  In the general case, the size of the array produced by a slice is unknown (see `Size types`_).  In a few cases, the size is known@@ -1767,12 +1769,13 @@ attribute, written as ``#[attr]``. This may affect how it is treated by the compiler or other tools. In no case will attributes affect or change the *semantics* of a program, but it may affect how well it compiles and runs (or in-some cases, whether it compiles or runs at all). Unknown attributes are silently-ignored. Most have no effect in the interpreter. An attribute can be either an-*atom*, written as an identifier or number, or *compound*, consisting of an-identifier and a comma-separated sequence of attributes. The latter is used for-grouping and encoding of more complex information. An identifier prefixed with a-``$`` must be the name of a term variable in scope.+some cases, whether it compiles or runs at all). Unknown attributes are usually+ignored silently, but may in some cases cause a warning. Most have no effect in+the interpreter. An attribute can be either an *atom*, written as an identifier+or number, or *compound*, consisting of an identifier and a comma-separated+sequence of attributes. The latter is used for grouping and encoding of more+complex information. An identifier prefixed with a ``$`` must be the name of a+term variable in scope.  Expression attributes ~~~~~~~~~~~~~~~~~~~~~@@ -1838,29 +1841,35 @@ ``flattening(no_outer)`` ........................ -When using incremental flattening, do not generate the "only outer-parallelism" version for the attributed SOACs.+During incremental flattening, do not generate the version that exploits only+outermost parallelism.  ``flattening(no_intra)`` ........................ -When using incremental flattening, do not generate the "intra-block-parallelism" version for the attributed SOACs.+During incremental flattening, do not generate the version that maps outermost+parallelism onto GPU thread blocks. +``flattening(only_outer)``+..........................++During incremental flattening, only generate the version that exploits outermost+parallelism and sequentialises inner parallelism.+ ``flattening(only_intra)`` .......................... -When using incremental flattening, *only* generate the "intra-block-parallelism" version of the attributed SOACs.  **Beware**: the-resulting program will fail to run if the inner parallelism does not-fit on the device.+During incremental flattening, *only* generate the version that maps outermost+parallelism onto GPU thread blocks. -``flattening(only_inner)``+**Beware**: the resulting program will fail to run if the inner parallelism does+not fit on the device.++``flattening(only_full)`` .......................... -When using incremental flattening, do not generate multiple versions-for this SOAC, but do exploit inner parallelism (which may give rise-to multiple versions at deeper levels).+During incremental flattening, generate only the fully flattened version. Any+versioning attributes inside the SOAC are ignored.  ``noinline`` ............@@ -1886,11 +1895,6 @@  Turn the outer parallelism in the attributed SOAC sequential, but preserve any inner parallelism.--``sequential_inner``-....................--Exploit only outer parallelism in the attributed SOAC.  ``unroll`` ..........
docs/man/futhark-wasm-multicore.rst view
@@ -16,10 +16,8 @@  ``futhark wasm-multicore`` translates a Futhark program to multi-threaded WebAssembly code by first generating C as ``futhark-c``, and then using Emscripten (``emcc``).  This produces a ``.js``-file that allows the compiled code to be invoked from JavaScript.-Executables implement the Futhark server protocol and can be run with-Node.js.+c``, and then using Emscripten (``emcc``).  The files produced are+the same as for :ref:`futhark-wasm(1)`.  OPTIONS =======@@ -35,7 +33,12 @@ ==================  Generated executables accept the same options as those generated by-:ref:`futhark-wasm(1)`.+:ref:`futhark-wasm(1)`.  The following additional options are+accepted.++--num-threads=INT++  Use this many threads.  SEE ALSO ========
docs/man/futhark-wasm.rst view
@@ -16,9 +16,12 @@  ``futhark wasm`` translates a Futhark program to sequential WebAssembly code by first generating C as ``futhark c``, and then-using Emscripten (``emcc``).  This produces a ``.js`` file that allows-the compiled code to be invoked from JavaScript.  Executables-implement the Futhark server protocol and can be run with Node.js.+using Emscripten (``emcc``).  For a program ``prog.fut``, this+produces an ES module ``prog.mjs``, along with the WebAssembly module+``prog.wasm``.  With ``--library``, the ES module exports the API+described in :ref:`js-api`.  Otherwise, ``prog`` is an executable+that runs the ES module with Node.js, and implements the Futhark+server protocol.  OPTIONS =======@@ -31,7 +34,7 @@ ``CFLAGS``    Space-separated list of options passed to ``emcc``.  Defaults-  to ``-O3 -std=c99`` if unset.+  to ``-O3 -msimd128`` if unset.  ``EMCFLAGS`` @@ -57,6 +60,27 @@   Print various low-overhead logging information to stderr while   running. +-P, --profile++  Enable the collection of profiling information.++--param=ASSIGNMENT++  Set a tuning parameter to the given value.  ``ASSIGNMENT`` must be of+  the form ``NAME=INT``.++--print-params++  Print all tuning parameters that can be set with ``--param`` or+  ``--tuning``.++--tuning=FILE++  Read size=value assignments from the given file.++--cache-file=FILE++  Accepted for compatibility with other backends, but has no effect.  SEE ALSO ========
futhark.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name:           futhark-version:        0.28.2+version:        0.29.1 synopsis:       An optimising compiler for a functional, array-oriented language.  description:    Futhark is a small programming language designed to be compiled to@@ -70,7 +70,7 @@     rts/style.css     rts/javascript/server.js     rts/javascript/values.js-    rts/javascript/wrapperclasses.js+    rts/javascript/wrappers.js     rts/opencl/copy.cl     rts/opencl/prelude.cl     rts/opencl/transpose.cl
rts/c/backends/cuda.h view
@@ -215,6 +215,8 @@   CUdeviceptr global_failure_args;   // True if a potentially failing kernel has been enqueued.   int32_t failure_is_an_option;+  CUdeviceptr bookkeeping_counters;+  CUdeviceptr bookkeeping_locks;   int total_runs;   long int total_runtime;   int64_t peak_mem_usage_device;@@ -724,6 +726,16 @@   return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+  CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_counters, 0,+                                         sizeof(int32_t)*bookkeeping_counters_size,+                                         ctx->stream));+  CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_locks, 0,+                                         sizeof(int32_t)*bookkeeping_locks_size,+                                         ctx->stream));+  return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) {   CUDA_SUCCEED_OR_RETURN(cuCtxPushCurrent(ctx->cu_ctx));   CUDA_SUCCEED_OR_RETURN(cuCtxSynchronize());@@ -745,6 +757,11 @@                                           &no_failure,                                           sizeof(int32_t))); +      // The failing kernel may have left the bookkeeping buffers nonzero.+      if (zero_bookkeeping(ctx) != 0) {+        return 1;+      }+       int64_t args[max_failure_args+1];       CUDA_SUCCEED_OR_RETURN(                              cuMemcpyDtoH(&args,@@ -856,6 +873,12 @@   // The +1 is to avoid zero-byte allocations.   CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1))); +  CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));+  CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));+  if (zero_bookkeeping(ctx) != 0) {+    return 1;+  }+   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {     return 1;   }@@ -868,7 +891,11 @@     free_builtin_kernels(ctx, ctx->kernels);     cuMemFree(ctx->global_failure);     cuMemFree(ctx->global_failure_args);-    CUDA_SUCCEED_FATAL(gpu_free_all(ctx));+    cuMemFree(ctx->bookkeeping_counters);+    cuMemFree(ctx->bookkeeping_locks);+    if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {+      futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);+    }     CUDA_SUCCEED_FATAL(cuStreamDestroy(ctx->stream));     CUDA_SUCCEED_FATAL(cuModuleUnload(ctx->module));     if (ctx->cfg->use_primary_context) {
rts/c/backends/hip.h view
@@ -182,6 +182,8 @@   void* global_failure_args;   // True if a potentially failing kernel has been enqueued.   int32_t failure_is_an_option;+  void* bookkeeping_counters;+  void* bookkeeping_locks;   int total_runs;   long int total_runtime;   int64_t peak_mem_usage_device;@@ -578,6 +580,16 @@   return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+  HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_counters, 0,+                                       sizeof(int32_t)*bookkeeping_counters_size,+                                       ctx->stream));+  HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_locks, 0,+                                       sizeof(int32_t)*bookkeeping_locks_size,+                                       ctx->stream));+  return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) {   HIP_SUCCEED_OR_RETURN(hipStreamSynchronize(ctx->stream));   if (ctx->failure_is_an_option) {@@ -596,6 +608,11 @@                                           &no_failure,                                           sizeof(int32_t))); +      // The failing kernel may have left the bookkeeping buffers nonzero.+      if (zero_bookkeeping(ctx) != 0) {+        return 1;+      }+       int64_t args[max_failure_args+1];       HIP_SUCCEED_OR_RETURN(hipMemcpyDtoH(&args,                                           ctx->global_failure_args,@@ -685,6 +702,12 @@   // The +1 is to avoid zero-byte allocations.   HIP_SUCCEED_FATAL(hipMalloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1))); +  HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));+  HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));+  if (zero_bookkeeping(ctx) != 0) {+    return 1;+  }+   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {     return 1;   }@@ -697,7 +720,11 @@     free_builtin_kernels(ctx, ctx->kernels);     hipFree(ctx->global_failure);     hipFree(ctx->global_failure_args);-    HIP_SUCCEED_FATAL(gpu_free_all(ctx));+    hipFree(ctx->bookkeeping_counters);+    hipFree(ctx->bookkeeping_locks);+    if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {+      futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);+    }     HIP_SUCCEED_FATAL(hipStreamDestroy(ctx->stream));     HIP_SUCCEED_FATAL(hipModuleUnload(ctx->module));   }
rts/c/backends/opencl.h view
@@ -449,6 +449,8 @@   cl_mem global_failure_args;   // True if a potentially failing kernel has been enqueued.   cl_int failure_is_an_option;+  cl_mem bookkeeping_counters;+  cl_mem bookkeeping_locks;   int total_runs;   long int total_runtime;   int64_t peak_mem_usage_device;@@ -672,6 +674,19 @@   return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+  cl_int zero = 0;+  OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_counters,+                                               &zero, sizeof(zero),+                                               0, sizeof(int32_t)*bookkeeping_counters_size,+                                               0, NULL, NULL));+  OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_locks,+                                               &zero, sizeof(zero),+                                               0, sizeof(int32_t)*bookkeeping_locks_size,+                                               0, NULL, NULL));+  return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) {   // Check for any delayed error.   cl_int failure_idx = -1;@@ -696,6 +711,11 @@                                                   0, sizeof(cl_int), &no_failure,                                                   0, NULL, NULL)); +    // The failing kernel may have left the bookkeeping buffers nonzero.+    if (zero_bookkeeping(ctx) != 0) {+      return 1;+    }+     int64_t args[max_failure_args+1];     OPENCL_SUCCEED_OR_RETURN(                              clEnqueueReadBuffer(ctx->queue,@@ -1154,6 +1174,18 @@                    sizeof(int64_t)*(max_failure_args+1), NULL, &error);   OPENCL_SUCCEED_OR_RETURN(error); +  ctx->bookkeeping_counters =+    clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,+                   sizeof(int32_t)*bookkeeping_counters_size, NULL, &error);+  OPENCL_SUCCEED_OR_RETURN(error);+  ctx->bookkeeping_locks =+    clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,+                   sizeof(int32_t)*bookkeeping_locks_size, NULL, &error);+  OPENCL_SUCCEED_OR_RETURN(error);+  if (zero_bookkeeping(ctx) != 0) {+    return 1;+  }+   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {     return 1;   }@@ -1168,6 +1200,8 @@     free_builtin_kernels(ctx, ctx->kernels);     OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure));     OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure_args));+    OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_counters));+    OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_locks));     (void)gpu_free_all(ctx);     (void)clReleaseProgram(ctx->clprogram);     (void)clReleaseCommandQueue(ctx->queue);
rts/c/gpu.h view
@@ -172,6 +172,7 @@  struct builtin_kernels* init_builtin_kernels(struct futhark_context* ctx) {   struct builtin_kernels *kernels = malloc(sizeof(struct builtin_kernels));+   gpu_create_kernel(ctx, &kernels->map_transpose_1b, "map_transpose_1b");   gpu_create_kernel(ctx, &kernels->map_transpose_1b_large, "map_transpose_1b_large");   gpu_create_kernel(ctx, &kernels->map_transpose_1b_low_height, "map_transpose_1b_low_height");@@ -375,23 +376,27 @@    void* args[15];   size_t args_sizes[15] = {-    sizeof(gpu_mem), sizeof(int64_t),-    sizeof(gpu_mem), sizeof(int64_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t),-    sizeof(int32_t)+    sizeof(int64_t), // dst_offset+    sizeof(int64_t), // src_offset+    sizeof(int32_t), // num_arrays+    sizeof(int32_t), // x_elems+    sizeof(int32_t), // y_elems+    sizeof(int32_t), // dst_array_stride+    sizeof(int32_t), // dst_row_stride+    sizeof(int32_t), // src_array_stride+    sizeof(int32_t), // src_row_stride+    sizeof(int32_t), // mulx+    sizeof(int32_t), // muly+    sizeof(int32_t), // repeat_1+    sizeof(int32_t), // repeat_2+    sizeof(gpu_mem), // dst+    sizeof(gpu_mem)  // src   }; -  args[0] = &dst;-  args[1] = &dst_offset;-  args[2] = &src;-  args[3] = &src_offset;+  args[0] = &dst_offset;+  args[1] = &src_offset;+  args[13] = &dst;+  args[14] = &src;    if (dst_extent <= 2147483647L && src_extent <= 2147483647L) {     if (m <= TR_BLOCK_DIM/2 && n <= TR_BLOCK_DIM/2) {@@ -435,12 +440,12 @@       block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD;       block[2] = 1;     }-    args[4] = &k32;-    args[5] = &m32;-    args[6] = &n32;-    for (int i = 0; i < 4; i++) { args[7+i] = &strides32[i]; }-    args[11] = &mulx32;-    args[12] = &muly32;+    args[2] = &k32;+    args[3] = &m32;+    args[4] = &n32;+    for (int i = 0; i < 4; i++) { args[5+i] = &strides32[i]; }+    args[9] = &mulx32;+    args[10] = &muly32;   } else {     if (ctx->logging) { fprintf(ctx->log, "Using large kernel\n"); }     kernel = kernel_large;@@ -450,13 +455,13 @@     block[0] = TR_TILE_DIM;     block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD;     block[2] = 1;-    args[4] = &k;-    args[5] = &m;-    args[6] = &n;-    for (int i = 0; i < 4; i++) { args[7+i] = &strides[i]; }-    args[11] = &mulx;-    args[12] = &muly;-    for (int i = 4; i < 13; i++) { args_sizes[i] = sizeof(int64_t); }+    args[2] = &k;+    args[3] = &m;+    args[4] = &n;+    for (int i = 0; i < 4; i++) { args[5+i] = &strides[i]; }+    args[9] = &mulx;+    args[10] = &muly;+    for (int i = 2; i < 11; i++) { args_sizes[i] = sizeof(int64_t); }   }    // Cap the number of thead blocks we launch and figure out how many@@ -465,10 +470,10 @@   int32_t repeat_2 = grid[2] / MAX_TR_THREAD_BLOCKS;   grid[1] = repeat_1 > 0 ? MAX_TR_THREAD_BLOCKS : grid[1];   grid[2] = repeat_2 > 0 ? MAX_TR_THREAD_BLOCKS : grid[2];-  args[13] = &repeat_1;-  args[14] = &repeat_2;-  args_sizes[13] = sizeof(repeat_1);-  args_sizes[14] = sizeof(repeat_2);+  args[11] = &repeat_1;+  args[12] = &repeat_2;+  args_sizes[11] = sizeof(repeat_1);+  args_sizes[12] = sizeof(repeat_2);    if (ctx->logging) {     fprintf(ctx->log, "\n");@@ -515,19 +520,20 @@   void* args[6+(8*3)];   size_t args_sizes[6+(8*3)]; -  args[0] = &dst;-  args_sizes[0] = sizeof(gpu_mem);-  args[1] = &dst_offset;-  args_sizes[1] = sizeof(dst_offset);-  args[2] = &src;-  args_sizes[2] = sizeof(gpu_mem);-  args[3] = &src_offset;-  args_sizes[3] = sizeof(src_offset);-  args[4] = &n;-  args_sizes[4] = sizeof(n);-  args[5] = &r;-  args_sizes[5] = sizeof(r);+  args[28] = &dst;+  args_sizes[28] = sizeof(gpu_mem);+  args[29] = &src;+  args_sizes[29] = sizeof(gpu_mem); +  args[0] = &dst_offset;+  args_sizes[0] = sizeof(dst_offset);+  args[1] = &src_offset;+  args_sizes[1] = sizeof(src_offset);+  args[2] = &n;+  args_sizes[2] = sizeof(n);+  args[3] = &r;+  args_sizes[3] = sizeof(r);+   int64_t zero = 0;    // To improve the odds of coalesced memory accesses, we pass the dimensions in@@ -536,7 +542,10 @@   int perm[8];   for (int i = 0; i < r; i++) {     int j = i;-    while (j > 0 && llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])) {+    while (j > 0 &&+           (llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])+            || (llabs(src_strides[perm[j-1]]) == llabs(src_strides[i])+                && llabs(dst_strides[perm[j-1]]) > llabs(dst_strides[i])))) {       perm[j] = perm[j-1];       j--;     }@@ -544,17 +553,17 @@   }    for (int i = 0; i < 8; i++) {-    args_sizes[6+i*3] = sizeof(int64_t);-    args_sizes[6+i*3+1] = sizeof(int64_t);-    args_sizes[6+i*3+2] = sizeof(int64_t);+    args_sizes[4+i*3] = sizeof(int64_t);+    args_sizes[4+i*3+1] = sizeof(int64_t);+    args_sizes[4+i*3+2] = sizeof(int64_t);     if (i < r) {-      args[6+i*3] = &shape[perm[i]];-      args[6+i*3+1] = &dst_strides[perm[i]];-      args[6+i*3+2] = &src_strides[perm[i]];+      args[4+i*3] = &shape[perm[i]];+      args[4+i*3+1] = &dst_strides[perm[i]];+      args[4+i*3+2] = &src_strides[perm[i]];     } else {-      args[6+i*3] = &zero;-      args[6+i*3+1] = &zero;-      args[6+i*3+2] = &zero;+      args[4+i*3] = &zero;+      args[4+i*3+1] = &zero;+      args[4+i*3+2] = &zero;     }   }   const size_t w = 256; // XXX: hardcoded thread block size.
rts/cuda/prelude.cu view
@@ -128,7 +128,9 @@ #define INFINITY (1.0/0.0) #endif -extern volatile __shared__ unsigned char shared_mem[];+// shared_mem itself must be 8-byte aligned no matter what, in case we access it+// with atomics.+extern volatile __shared__ __attribute__((aligned(8))) unsigned char shared_mem[];  #define SHARED_MEM_PARAM #define FUTHARK_KERNEL extern "C" __global__ __launch_bounds__(MAX_THREADS_PER_BLOCK)
rts/javascript/server.js view
@@ -1,8 +1,8 @@ // Start of server.js -// The server is implemented directly on top of the C API exported by the-// WebAssembly module, much like the C server.  The manifest describes the-// types and entry points, and names the C functions that operate on them.+// The server is implemented in terms of the C API exported by the WebAssembly+// module. The manifest describes the types and entry points, and names the C+// functions that operate on them. // // A variable is a pair of a type name and a value.  A value of primitive type // is a JavaScript number, or a BigInt for 64-bit integers.  Booleans are 0 or@@ -38,8 +38,13 @@     this.wasm = ctx.wasm;     this.manifest = manifest;     this._vars = {};+    this._output = [];   } +  _print(line) {+    this._output.push(line);+  }+   _get_arg(args, i) {     if (i < args.length) {       return args[i];@@ -99,18 +104,33 @@    // Calling into the C API. -  _c(fname, ...args) {-    return this.wasm['_' + fname](this.ctx.ctx, ...args);+  async _c(fname, ...args) {+    return await this.wasm.ccall(fname, 'number',+      Array(1 + args.length).fill('number'),+      [this.ctx.ctx, ...args], {async: true});   } +  // Read a NUL-terminated string allocated by C, and free it.+  _c_string(p) {+    var heap = this.wasm.HEAPU8;+    // Browsers' TextDecoder rejects shared memory.+    var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));+    this.wasm._free(p);+    return str;+  }++  _get_error() {+    return this._c_string(this.wasm._futhark_context_get_error(this.ctx.ctx));+  }+   _check(err) {     if (err != 0) {-      throw this.ctx.get_error();+      throw this._get_error();     }   } -  _sync() {-    this._check(this.wasm._futhark_context_sync(this.ctx.ctx));+  async _sync() {+    this._check(await this._c('futhark_context_sync'));   }    _malloc(n) {@@ -141,10 +161,10 @@    // Call 'f' with a pointer to space for one value of the given type, and   // return the value that it stores there.-  _with_out(tname, f) {+  async _with_out(tname, f) {     var p = this._malloc(this._sizeof(tname));     try {-      f(p);+      await f(p);       return this._load(tname, p);     } finally {       this.wasm._free(p);@@ -152,9 +172,19 @@   }    _bytes(p, n) {-    return Buffer.from(this.wasm.HEAPU8.slice(p, p + n));+    return this.wasm.HEAPU8.slice(p, p + n);   } +  async _read_file(fname) {+    const fs = await import('node:fs');+    return new Uint8Array(fs.readFileSync(fname));+  }++  async _write_file(fname, bytes) {+    const fs = await import('node:fs');+    fs.writeFileSync(fname, bytes);+  }+   // Types.    _type(tname) {@@ -210,16 +240,16 @@     return this._type(tname).sum;   } -  _free_value(tname, value) {+  async _free_value(tname, value) {     var t = this._type(tname);     if (t !== null) {-      this._check(this._c(t.ops.free, value));+      this._check(await this._c(t.ops.free, value));     }   } -  _shape(a, arr) {+  async _shape(a, arr) {     var fshape = a.ops ? a.ops.shape : a.shape;-    var p = this._c(fshape, arr);+    var p = await this._c(fshape, arr);     var shape = [];     for (var i = 0; i < a.rank; i++) {       shape.push(this._load('i64', p + i * 8));@@ -237,7 +267,7 @@    // Values. -  _restore_value(reader, tname) {+  async _restore_value(reader, tname) {     var t = this._type(tname);     if (t === null) {       return read_value(tname, reader);@@ -247,11 +277,11 @@       var p = this._malloc(bytes.length);       try {         this.wasm.HEAPU8.set(bytes, p);-        var arr = this._c(t.ops.new, p, ...shape);+        var arr = await this._c(t.ops.new, p, ...shape);         if (arr == 0) {-          throw this.ctx.get_error();+          throw this._get_error();         }-        this._sync();+        await this._sync();         return arr;       } finally {         this.wasm._free(p);@@ -263,22 +293,22 @@       var p = this._malloc(buff.length);       try {         this.wasm.HEAPU8.set(buff, p);-        var obj = this._c(t.ops.restore, p);+        var obj = await this._c(t.ops.restore, p);         if (obj == 0) {-          throw this.ctx.get_error();+          throw this._get_error();         }       } finally {         this.wasm._free(p);       }-      var n = this._with_out('size', (np) => {-        this._check(this._c(t.ops.store, obj, 0, np));+      var n = await this._with_out('size', async (np) => {+        this._check(await this._c(t.ops.store, obj, 0, np));       });       reader.buff = buff.slice(n);       return obj;     }   } -  _store_value(tname, value) {+  async _store_value(tname, value) {     var t = this._type(tname);     if (t === null) {       var p = this._malloc(this._sizeof(tname));@@ -289,12 +319,12 @@         this.wasm._free(p);       }     } else if (t.kind == "array") {-      var shape = this._shape(this._array_type(tname), value);+      var shape = await this._shape(this._array_type(tname), value);       var n = shape.reduce((x, y) => x * y, 1n) * BigInt(this._sizeof(t.elemtype));       var p = this._malloc(Number(n));       try {-        this._check(this._c(t.ops.values, value, p));-        this._sync();+        this._check(await this._c(t.ops.values, value, p));+        await this._sync();         return binary_value(t.elemtype, shape, this._bytes(p, Number(n)));       } finally {         this.wasm._free(p);@@ -304,8 +334,8 @@       var np = pp + serverPointer.size;       try {         this._poke('ptr', pp, 0);-        this._check(this._c(t.ops.store, value, pp, np));-        this._sync();+        this._check(await this._c(t.ops.store, value, pp, np));+        await this._sync();         var p = this._load('ptr', pp);         var bytes = this._bytes(p, this._load('size', np));         this.wasm._free(p);@@ -321,23 +351,35 @@   _cmd_inputs(args) {     var entry = this._get_entry_point(this._get_arg(args, 0));     for (var input of entry.inputs) {-      console.log((input.consumed ? "*" : "") + input.type);+      this._print((input.consumed ? "*" : "") + input.type);     }   }    _cmd_output(args) {     var entry = this._get_entry_point(this._get_arg(args, 0));-    console.log((entry.output.fresh ? "*" : "") + entry.output.type);+    this._print((entry.output.fresh ? "*" : "") + entry.output.type);   } -  _cmd_dummy(args) {-    // pass+  async _cmd_clear(args) {+    this._check(await this._c('futhark_context_clear_caches'));   } -  _cmd_free(args) {+  async _cmd_pause_profiling(args) {+    await this._c('futhark_context_pause_profiling');+  }++  async _cmd_unpause_profiling(args) {+    await this._c('futhark_context_unpause_profiling');+  }++  async _cmd_report(args) {+    this._print(this._c_string(await this._c('futhark_context_report')));+  }++  async _cmd_free(args) {     for (var vname of args) {       var v = this._get_var(vname);-      this._free_value(v.type, v.value);+      await this._free_value(v.type, v.value);       delete this._vars[vname];     }   }@@ -353,27 +395,27 @@    _cmd_types(args) {     for (var t in serverPrimtypes) {-      console.log(t);+      this._print(t);     }     for (var t in this.manifest.types) {-      console.log(t);+      this._print(t);     }   }    _cmd_entry_points(args) {     for (var e in this.manifest.entry_points) {-      console.log(e);+      this._print(e);     }   }    _cmd_attributes(args) {     var entry = this._get_entry_point(this._get_arg(args, 0));     for (var attr of entry.attributes) {-      console.log(attr);+      this._print(attr);     }   } -  _cmd_call(args) {+  async _cmd_call(args) {     var entry = this._get_entry_point(this._get_arg(args, 0));     var out_vname = this._get_arg(args, 1);     var in_vnames = args.slice(2);@@ -384,39 +426,39 @@     this._check_new_var(out_vname);      var bef = performance.now() * 1000;-    var out = this._with_out(entry.output.type, (p) => {-      this._check(this._c(entry.cfun, p, ...ins));-      this._sync();+    var out = await this._with_out(entry.output.type, async (p) => {+      this._check(await this._c(entry.cfun, p, ...ins));+      await this._sync();     });     var aft = performance.now() * 1000;-    console.log("runtime: " + Math.round(aft - bef));+    this._print("runtime: " + Math.round(aft - bef));     this._set_var(out_vname, entry.output.type, out);   } -  _cmd_store(args) {+  async _cmd_store(args) {     var fname = this._get_arg(args, 0);     var bufs = [];     for (var vname of args.slice(1)) {       var v = this._get_var(vname);-      bufs.push(this._store_value(v.type, v.value));+      bufs.push(await this._store_value(v.type, v.value));     }-    require("fs").writeFileSync(fname, Buffer.concat(bufs));+    await this._write_file(fname, concat_bytes(bufs));   } -  _cmd_restore(args) {+  async _cmd_restore(args) {     var fname = this._get_arg(args, 0);     if (args.length % 2 == 0) {       throw "Invalid argument count";     } -    var reader = new Reader(fname);+    var reader = new Reader(await this._read_file(fname));     for (var i = 1; i < args.length; i += 2) {       var vname = args[i];       var tname = args[i + 1];       this._check_new_var(vname);       this._type(tname);       try {-        this._set_var(vname, tname, this._restore_value(reader, tname));+        this._set_var(vname, tname, await this._restore_value(reader, tname));       } catch (err) {         throw "Failed to restore variable " + vname + ".\nPossibly malformed data in " + fname + ".\n" + err.toString();       }@@ -428,29 +470,29 @@   }    _cmd_kind(args) {-    console.log(this._kind(this._get_arg(args, 0)));+    this._print(this._kind(this._get_arg(args, 0)));   }    _cmd_type(args) {-    console.log(this._get_var(this._get_arg(args, 0)).type);+    this._print(this._get_var(this._get_arg(args, 0)).type);   }    _cmd_rank(args) {-    console.log(this._array_type(this._get_arg(args, 0)).rank);+    this._print(this._array_type(this._get_arg(args, 0)).rank);   }    _cmd_elemtype(args) {-    console.log(this._array_type(this._get_arg(args, 0)).elemtype);+    this._print(this._array_type(this._get_arg(args, 0)).elemtype);   } -  _cmd_shape(args) {+  async _cmd_shape(args) {     var v = this._get_var(this._get_arg(args, 0));-    for (var d of this._shape(this._array_type(v.type), v.value)) {-      console.log(d.toString());+    for (var d of await this._shape(this._array_type(v.type), v.value)) {+      this._print(d.toString());     }   } -  _cmd_new_array(args) {+  async _cmd_new_array(args) {     var dst = this._get_arg(args, 0);     var tname = this._get_arg(args, 1);     this._check_new_var(dst);@@ -469,49 +511,49 @@       vs.forEach((v, i) => this._poke(a.elemtype, p + i * size, v));       var arr;       if (a.ops) {-        arr = this._c(a.ops.new, p, ...dims);+        arr = await this._c(a.ops.new, p, ...dims);         if (arr == 0) {-          throw this.ctx.get_error();+          throw this._get_error();         }       } else {-        arr = this._with_out(tname, (out) => {-          this._check(this._c(a.new, out, p, ...dims));+        arr = await this._with_out(tname, async (out) => {+          this._check(await this._c(a.new, out, p, ...dims));         });       }-      this._sync();+      await this._sync();       this._set_var(dst, tname, arr);     } finally {       this.wasm._free(p);     }   } -  _cmd_set(args) {+  async _cmd_set(args) {     var arr = this._get_var(this._get_arg(args, 0));     var a = this._array_type(arr.type);     var val = this._get_typed_var(this._get_arg(args, 1), a.elemtype);     var is = this._parse_ints(args.slice(2), a.rank);-    this._check_bounds(this._shape(a, arr.value), is);+    this._check_bounds(await this._shape(a, arr.value), is);     var fset = a.ops ? a.ops.set : a.set;-    this._check(this._c(fset, arr.value, val, ...is));-    this._sync();+    this._check(await this._c(fset, arr.value, val, ...is));+    await this._sync();   } -  _cmd_index(args) {+  async _cmd_index(args) {     var dst = this._get_arg(args, 0);     var arr = this._get_var(this._get_arg(args, 1));     this._check_new_var(dst);     var a = this._array_type(arr.type);     var is = this._parse_ints(args.slice(2), a.rank);-    this._check_bounds(this._shape(a, arr.value), is);+    this._check_bounds(await this._shape(a, arr.value), is);     var findex = a.ops ? a.ops.index : a.index;-    var v = this._with_out(a.elemtype, (p) => {-      this._check(this._c(findex, p, arr.value, ...is));-      this._sync();+    var v = await this._with_out(a.elemtype, async (p) => {+      this._check(await this._c(findex, p, arr.value, ...is));+      await this._sync();     });     this._set_var(dst, a.elemtype, v);   } -  _cmd_zip(args) {+  async _cmd_zip(args) {     var dst = this._get_arg(args, 0);     var tname = this._get_arg(args, 1);     this._check_new_var(dst);@@ -524,14 +566,14 @@       throw a.fields.length + " arrays expected but " + vnames.length + " values provided.";     }     var vs = vnames.map((v, i) => this._get_typed_var(v, a.fields[i].type));-    var arr = this._with_out(tname, (p) => {-      this._check(this._c(a.zip, p, ...vs));-      this._sync();+    var arr = await this._with_out(tname, async (p) => {+      this._check(await this._c(a.zip, p, ...vs));+      await this._sync();     });     this._set_var(dst, tname, arr);   } -  _cmd_unzip(args) {+  async _cmd_unzip(args) {     var arr = this._get_var(this._get_arg(args, 0));     var a = this._array_type(arr.type);     if (!a.fields) {@@ -542,22 +584,23 @@       throw a.fields.length + " arrays expected but " + dsts.length + " values provided.";     }     dsts.forEach((dst) => this._check_new_var(dst));-    a.fields.forEach((f, i) => {-      var v = this._with_out(f.type, (p) => {-        this._check(this._c(f.project, p, arr.value));+    for (var i = 0; i < a.fields.length; i++) {+      var f = a.fields[i];+      var v = await this._with_out(f.type, async (p) => {+        this._check(await this._c(f.project, p, arr.value));       });       this._set_var(dsts[i], f.type, v);-    });-    this._sync();+    }+    await this._sync();   }    _cmd_fields(args) {     for (var f of this._record_type(this._get_arg(args, 0)).fields) {-      console.log(f.name + " " + f.type);+      this._print(f.name + " " + f.type);     }   } -  _cmd_new(args) {+  async _cmd_new(args) {     var dst = this._get_arg(args, 0);     var tname = this._get_arg(args, 1);     this._check_new_var(dst);@@ -567,13 +610,13 @@       throw r.fields.length + " fields expected but " + vnames.length + " values provided.";     }     var vs = vnames.map((v, i) => this._get_typed_var(v, r.fields[i].type));-    var obj = this._with_out(tname, (p) => {-      this._check(this._c(r.new, p, ...vs));+    var obj = await this._with_out(tname, async (p) => {+      this._check(await this._c(r.new, p, ...vs));     });     this._set_var(dst, tname, obj);   } -  _cmd_project(args) {+  async _cmd_project(args) {     var dst = this._get_arg(args, 0);     var from = this._get_var(this._get_arg(args, 1));     var field = this._get_arg(args, 2);@@ -582,31 +625,31 @@     if (f === undefined) {       throw "No such field: " + field;     }-    var v = this._with_out(f.type, (p) => {-      this._check(this._c(f.project, p, from.value));+    var v = await this._with_out(f.type, async (p) => {+      this._check(await this._c(f.project, p, from.value));     });     this._set_var(dst, f.type, v);   }    _cmd_variants(args) {     for (var v of this._sum_type(this._get_arg(args, 0)).variants) {-      console.log(v.name);+      this._print(v.name);       for (var t of v.payload) {-        console.log("- " + t);+        this._print("- " + t);       }     }   } -  _variant_of(v) {+  async _variant_of(v) {     var s = this._sum_type(v.type);-    return s.variants[this._c(s.variant, v.value)];+    return s.variants[await this._c(s.variant, v.value)];   } -  _cmd_variant(args) {-    console.log(this._variant_of(this._get_var(this._get_arg(args, 0))).name);+  async _cmd_variant(args) {+    this._print((await this._variant_of(this._get_var(this._get_arg(args, 0)))).name);   } -  _cmd_construct(args) {+  async _cmd_construct(args) {     var dst = this._get_arg(args, 0);     var tname = this._get_arg(args, 1);     var vname = this._get_arg(args, 2);@@ -620,15 +663,15 @@       throw variant.payload.length + " values expected but " + vnames.length + " provided.";     }     var vs = vnames.map((v, i) => this._get_typed_var(v, variant.payload[i]));-    var obj = this._with_out(tname, (p) => {-      this._check(this._c(variant.construct, p, ...vs));+    var obj = await this._with_out(tname, async (p) => {+      this._check(await this._c(variant.construct, p, ...vs));     });     this._set_var(dst, tname, obj);   } -  _cmd_destruct(args) {+  async _cmd_destruct(args) {     var v = this._get_var(this._get_arg(args, 0));-    var variant = this._variant_of(v);+    var variant = await this._variant_of(v);     var dsts = args.slice(1);     if (dsts.length != variant.payload.length) {       throw variant.payload.length + " variables expected but " + dsts.length + " provided.";@@ -636,75 +679,82 @@     dsts.forEach((dst) => this._check_new_var(dst));     var ps = variant.payload.map((t) => this._malloc(this._sizeof(t)));     try {-      this._check(this._c(variant.destruct, ...ps, v.value));+      this._check(await this._c(variant.destruct, ...ps, v.value));       variant.payload.forEach((t, i) => this._set_var(dsts[i], t, this._load(t, ps[i])));     } finally {       ps.forEach((p) => this.wasm._free(p));     }   } -  _process_line(line) {+  async _process_command(cmd, args) {+    switch (cmd) {+    case 'inputs': return this._cmd_inputs(args);+    case 'output': return this._cmd_output(args);+    case 'call': return this._cmd_call(args);+    case 'restore': return this._cmd_restore(args);+    case 'store': return this._cmd_store(args);+    case 'free': return this._cmd_free(args);+    case 'clear': return this._cmd_clear(args);+    case 'pause_profiling': return this._cmd_pause_profiling(args);+    case 'unpause_profiling': return this._cmd_unpause_profiling(args);+    case 'report': return this._cmd_report(args);+    case 'rename': return this._cmd_rename(args);+    case 'types': return this._cmd_types(args);+    case 'entry_points': return this._cmd_entry_points(args);+    case 'attributes': return this._cmd_attributes(args);+    case 'kind': return this._cmd_kind(args);+    case 'type': return this._cmd_type(args);+    case 'rank': return this._cmd_rank(args);+    case 'elemtype': return this._cmd_elemtype(args);+    case 'shape': return this._cmd_shape(args);+    case 'new_array': return this._cmd_new_array(args);+    case 'set': return this._cmd_set(args);+    case 'index': return this._cmd_index(args);+    case 'zip': return this._cmd_zip(args);+    case 'unzip': return this._cmd_unzip(args);+    case 'fields': return this._cmd_fields(args);+    case 'new': return this._cmd_new(args);+    case 'project': return this._cmd_project(args);+    case 'variants': return this._cmd_variants(args);+    case 'construct': return this._cmd_construct(args);+    case 'destruct': return this._cmd_destruct(args);+    case 'variant': return this._cmd_variant(args);+    default: throw "Unknown command: " + cmd;+    }+  }++  async _process_line(line) {     var words = split_words(line);     if (words.length == 0) {       throw "Empty line";-    } else {-      var cmd = words[0];-      var args = words.slice(1);-      switch (cmd) {-      case 'inputs': this._cmd_inputs(args); break;-      case 'output': this._cmd_output(args); break;-      case 'call': this._cmd_call(args); break;-      case 'restore': this._cmd_restore(args); break;-      case 'store': this._cmd_store(args); break;-      case 'free': this._cmd_free(args); break;-      case 'clear': this._cmd_dummy(args); break;-      case 'pause_profiling': this._cmd_dummy(args); break;-      case 'unpause_profiling': this._cmd_dummy(args); break;-      case 'report': this._cmd_dummy(args); break;-      case 'rename': this._cmd_rename(args); break;-      case 'types': this._cmd_types(args); break;-      case 'entry_points': this._cmd_entry_points(args); break;-      case 'attributes': this._cmd_attributes(args); break;-      case 'kind': this._cmd_kind(args); break;-      case 'type': this._cmd_type(args); break;-      case 'rank': this._cmd_rank(args); break;-      case 'elemtype': this._cmd_elemtype(args); break;-      case 'shape': this._cmd_shape(args); break;-      case 'new_array': this._cmd_new_array(args); break;-      case 'set': this._cmd_set(args); break;-      case 'index': this._cmd_index(args); break;-      case 'zip': this._cmd_zip(args); break;-      case 'unzip': this._cmd_unzip(args); break;-      case 'fields': this._cmd_fields(args); break;-      case 'new': this._cmd_new(args); break;-      case 'project': this._cmd_project(args); break;-      case 'variants': this._cmd_variants(args); break;-      case 'construct': this._cmd_construct(args); break;-      case 'destruct': this._cmd_destruct(args); break;-      case 'variant': this._cmd_variant(args); break;-      default: throw "Unknown command: " + cmd;-      }     }+    await this._process_command(words[0], words.slice(1));   } -  run() {+  async run() {     console.log('%%% OK'); // TODO figure out if flushing is neccesary for JS-    const readline = require('readline');+    const readline = await import('node:readline');     const rl = readline.createInterface(process.stdin);-    rl.on('line', (line) => {+    // Commands are asynchronous, so we must finish one before reading the+    // next.+    for await (const line of rl) {       if (line == "") {-        rl.close();-        return;+        break;       }+      this._output = [];       try {-        this._process_line(line);+        await this._process_line(line);+        this._output.forEach((l) => console.log(l));         console.log('%%% OK');       } catch (err) {         console.log('%%% FAILURE');+        this._output.forEach((l) => console.log(l));         console.log(err);         console.log('%%% OK');       }-    }).on('close', () => { process.exit(0); });+    }+    this.ctx.free();+    process.exit(0);   } } @@ -726,6 +776,142 @@     throw "Unterminated quote";   }   return words;+}++// Run a server as the main program of a Node.js process.+async function futhark_server_main() {+  const fs = await import('node:fs');+  const progname = process.argv[1];+  const fail = (msg) => {+    console.error(progname + ": " + msg);+    process.exit(1);+  };+  const parse_int = (what, x) => {+    if (!/^[0-9]+$/.test(x)) {+      fail("Invalid " + what + ": " + x);+    }+    return Number(x);+  };++  const options = { tuning_params: {} };+  var print_params = false;+  // Long name, short name, argument, description, and action.+  const table = [+    ["debugging", "D", null,+     "Perform possibly expensive internal correctness checks and verbose logging.",+     () => { options.debugging = true; }],+    ["log", "L", null,+     "Print various low-overhead logging information while running.",+     () => { options.logging = true; }],+    ["profile", "P", null,+     "Enable the collection of profiling information.",+     () => { options.profiling = true; }],+    ["help", "h", null,+     "Print help information and exit.",+     () => {+       console.log("Usage: " + progname + " [OPTIONS]...\nOptions:\n");+       for (const [long, short, arg, desc] of table) {+         console.log("  " + (short ? "-" + short + "/" : "") + "--" + long ++                     (arg ? " " + arg : "") + "\n    " + desc);+       }+       console.log("\nFor more information, consult the Futhark User's Guide or the man pages.");+       process.exit(0);+     }],+    ["print-params", null, null,+     "Print all tuning parameters that can be set with --param or --tuning.",+     () => { print_params = true; }],+    ["param", null, "ASSIGNMENT",+     "Set a tuning parameter to the given value.",+     (x) => {+       const eq = x.indexOf("=");+       if (eq < 0) {+         fail("Invalid argument for --param option: " + x);+       }+       options.tuning_params[x.slice(0, eq)] = parse_int("tuning parameter value", x.slice(eq + 1));+     }],+    ["tuning", null, "FILE",+     "Read size=value assignments from the given file.",+     (fname) => {+       var lines;+       try {+         lines = fs.readFileSync(fname, "utf8").split("\n");+       } catch (e) {+         fail("When loading tuning file '" + fname + "': Cannot open file: " + e.message);+       }+       lines.forEach((line, i) => {+         if (/^[ \t]*($|--)/.test(line)) {+           return;+         }+         const m = /^([^=]*)=(-?[0-9]+)$/.exec(line);+         if (m === null) {+           fail("When loading tuning file '" + fname + "': Invalid line " + (i + 1) ++                " (must be of form 'name=int').");+         }+         options.tuning_params[m[1]] = Number(m[2]);+       });+     }],+    // There is nothing to cache.+    ["cache-file", null, "FILE",+     "Store program cache here (has no effect for this backend).",+     () => {}],+  ];+  if (futhark_manifest.backend == "wasm_multicore") {+    table.push(["num-threads", null, "INT",+                "Set number of threads used for execution.",+                (x) => { options.num_threads = parse_int("number of threads", x); }]);+  }++  const args = process.argv.slice(2);+  for (var i = 0; i < args.length; i++) {+    const a = args[i];+    if (a.startsWith("--")) {+      const eq = a.indexOf("=");+      const name = eq < 0 ? a.slice(2) : a.slice(2, eq);+      const opt = table.find((o) => o[0] == name);+      if (opt === undefined) {+        fail("unrecognized option '" + a + "'");+      }+      var arg;+      if (opt[2] !== null) {+        arg = eq < 0 ? args[++i] : a.slice(eq + 1);+        if (arg === undefined) {+          fail("option '--" + name + "' requires an argument");+        }+      } else if (eq >= 0) {+        fail("option '--" + name + "' doesn't allow an argument");+      }+      opt[4](arg);+    } else if (a.startsWith("-") && a.length > 1) {+      for (const c of a.slice(1)) {+        const opt = table.find((o) => o[1] == c);+        if (opt === undefined) {+          fail("invalid option -- '" + c + "'");+        }+        opt[4]();+      }+    } else {+      fail("unexpected argument: " + a);+    }+  }++  if (print_params) {+    const m = await loadWASM();+    const heap = () => m.HEAPU8;+    const str = (p) => new TextDecoder().decode(heap().slice(p, heap().indexOf(0, p)));+    for (var j = 0; j < m._futhark_get_tuning_param_count(); j++) {+      console.log(str(m._futhark_get_tuning_param_name(j)) + " (" ++                  str(m._futhark_get_tuning_param_class(j)) + ")");+    }+    process.exit(0);+  }++  const context = new FutharkContext();+  try {+    await context.init(options);+  } catch (e) {+    fail(e.message);+  }+  await new Server(context, futhark_manifest).run(); }  // End of server.js
rts/javascript/values.js view
@@ -1,4 +1,8 @@ // Start of values.js+//+// Reading and writing values in the binary data format.  Values are read from+// and written to byte buffers (Uint8Arrays).+ var futharkPrimtypes =   new Set([     'i8',@@ -29,19 +33,6 @@                   'bool' : Uint8Array                 }; -function binToStringArray(buff, array) {-  for (var i = 0; i < array.length; i++) {-    array[i] = buff[i];-  }-}--function fileToBuff(fname) {-  var readline = require('readline');-  var fs = require('fs');-  var buff =  fs.readFileSync(fname);-  return buff;-}- var typToSize = {   "bool" : 1,   "  u8" : 1,@@ -61,6 +52,16 @@   return new Uint8Array(ta.buffer, ta.byteOffset, ta.byteLength); } +function concat_bytes(arrs) {+  var out = new Uint8Array(arrs.reduce((n, a) => n + a.length, 0));+  var off = 0;+  for (var a of arrs) {+    out.set(a, off);+    off += a.length;+  }+  return out;+}+ // The binary encoding of a value with the given primitive type and shape (a // list of BigInts), where 'bytes' is the payload. function binary_value(typ, shape, bytes) {@@ -73,65 +74,59 @@     header[3+i] = ftype.charCodeAt(i);   }   var dims = new BigInt64Array(shape);-  return Buffer.concat([header, toU8(dims), bytes]);+  return concat_bytes([header, toU8(dims), bytes]); } +function is_space(b) {+  return b == 32 || (b >= 9 && b <= 13);+}+ class Reader {-  constructor(f) {-    this.f = f;-    this.buff = fileToBuff(f);+  constructor(buff) {+    this.buff = buff;   } -  read_bin_array(num_dim, typ) {-    var u8_array = new Uint8Array(num_dim * 8);-    binToStringArray(this.buff.slice(0, num_dim * 8), u8_array);-    var shape = new BigInt64Array(u8_array.buffer);-    var length = shape[0];-    for (var i = 1; i < shape.length; i++) {-      length = length * shape[i];+  // Remove the first n bytes from the buffer, returning a copy of them.+  take(n) {+    if (this.buff.length < n) {+      throw "Unexpected end of input";     }-    length = Number(length);-    var dbytes = typToSize[typ];-    var u8_data = new Uint8Array(length * dbytes);-    binToStringArray(this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length), u8_data);-    var data  = new (typToType[typ])(u8_data.buffer);-    var tmp_buff = this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length);-    this.buff = this.buff.slice(num_dim * 8 + dbytes * length);-   return [shape, data];+    var bytes = this.buff.slice(0, n);+    this.buff = this.buff.subarray(n);+    return bytes;   } +  read_bin_array(num_dim, typ) {+    var shape = new BigInt64Array(this.take(num_dim * 8).buffer);+    var length = Number(shape.reduce((x, y) => x * y, 1n));+    var data = new (typToType[typ])(this.take(length * typToSize[typ]).buffer);+    return [shape, data];+  }+   read_bin_scalar(typ) {-    var size = typToSize[typ];-    var u8_array = new Uint8Array(size);-    binToStringArray(this.buff, u8_array);-    var array = new (typToType[typ])(u8_array.buffer);-    this.buff = this.buff.slice(u8_array.length); // Update buff to be unread part of the string-    return array[0];+    return new (typToType[typ])(this.take(typToSize[typ]).buffer)[0];   }    skip_spaces() {-    while (this.buff.length > 0 && this.buff.slice(0, 1).toString().trim() == "") {-      this.buff = this.buff.slice(1);+    while (this.buff.length > 0 && is_space(this.buff[0])) {+      this.buff = this.buff.subarray(1);     }   }    read_binary(typename, dim) {-    // Skip leading white space-    while (this.buff.slice(0, 1).toString().trim() == "") {-      this.buff = this.buff.slice(1);-    }-    if (this.buff[0] != 'b'.charCodeAt(0)) {+    this.skip_spaces();+    var header = this.take(7);+    if (header[0] != 'b'.charCodeAt(0)) {       throw "Not in binary format"     }-    var version = this.buff[1];+    var version = header[1];     if (version != 2) {       throw "Not version 2";     }-    var num_dim = this.buff[2];-    var typ = this.buff.slice(3, 7);-    this.buff = this.buff.slice(7);+    var num_dim = header[2];+    var typ = String.fromCharCode(...header.subarray(3, 7));     var exp_typ = "[]".repeat(dim) + typename;-    var given_typ = "[]".repeat(num_dim) + typ.toString().trim();+    var given_typ = "[]".repeat(num_dim) + typ.trim();     if (exp_typ !== given_typ) {       throw ("Expected type : " + exp_typ + ", Actual type : " + given_typ);     }
− rts/javascript/wrapperclasses.js
@@ -1,91 +0,0 @@-// Start of wrapperclasses.js--class FutharkArray {-  constructor(ctx, ptr, type_name, dim, heap, fshape, fvalues, ffree) {-    this.ctx = ctx;-    this.ptr = ptr;-    this.type_name = type_name;-    this.dim = dim;-    this.heap = heap;-    this.fshape = fshape;-    this.fvalues = fvalues;-    this.ffree = ffree;-    this.valid = true;-  }--  validCheck() {-    if (!this.valid) {-      throw "Using freed memory"-    }-  }--  futharkType() {-    return this.type_name;-  }--  free() {-    this.validCheck();-    this.ffree(this.ctx.ctx, this.ptr);-    this.valid = false;-  }--  shape() {-    this.validCheck();-    var s = this.fshape(this.ctx.ctx, this.ptr) >> 3;-    return this.ctx.wasm.HEAP64.subarray(s, s + this.dim);-  }--  // get_shape() {-  //   return BigInt64Array.from(this.shape());-  // }--  toTypedArray(dims = this.shape()) {-    this.validCheck();-    console.assert(dims.length === this.dim, "dim=%s,dims=%s", this.dim, dims.toString());-    var length = Number(dims.reduce((a, b) => a * b));-    var v = this.fvalues(this.ctx.ctx, this.ptr) / this.heap.BYTES_PER_ELEMENT;-    return this.heap.subarray(v, v + length);-  }--  toArray() {-    this.validCheck();-    var dims = this.shape();-    var ta = this.toTypedArray(dims);-    return (function nest(offs, ds) {-      var d0 = Number(ds[0]);-      if (ds.length === 1) {-        return Array.from(ta.subarray(offs, offs + d0));-      } else {-        var d1 = Number(ds[1]);-        return Array.from(Array(d0), (x,i) => nest(offs + i * d1, ds.slice(1)));-      }-    })(0, dims);-  }--  async values() {-    return this.toTypedArray();-  }-}--class FutharkOpaque {-  constructor(ctx, ptr, ffree) {-    this.ctx = ctx;-    this.ptr = ptr;-    this.ffree = ffree;-    this.valid = true;-  }--  validCheck() {-    if (!this.valid) {-      throw "Using freed memory"-    }-  }--  free() {-    this.validCheck();-    this.ffree(this.ctx.ctx, this.ptr);-    this.valid = false;-  }-}--// End of wrapperclasses.js
+ rts/javascript/wrappers.js view
@@ -0,0 +1,567 @@+// Start of wrappers.js+//+// The JavaScript API (docs/js-api.rst), driven by 'futhark_manifest'.+//+// Asynchronous operations run one at a time, as Asyncify (used by WebGPU)+// allows no other calls into the module while one is suspended.++// f16 values and their bit patterns, rounding to nearest even.+function futhark_f16_bits(x) {+  if (Number.isNaN(x)) {+    return 0x7e00;+  }+  var sign = (x < 0 || Object.is(x, -0)) ? 0x8000 : 0;+  x = Math.abs(x);+  if (x < 2 ** -14) {+    // Subnormal or zero.+    return sign | futhark_round_even(x * 2 ** 24);+  }+  var e = Math.floor(Math.log2(x));+  // Math.log2 may be inexact.+  if (2 ** e > x) {+    e--;+  } else if (2 ** (e + 1) <= x) {+    e++;+  }+  var m = futhark_round_even((x / 2 ** e - 1) * 1024);+  if (m == 1024) {+    m = 0;+    e++;+  }+  if (e > 15) {+    return sign | 0x7c00;+  }+  return sign | ((e + 15) << 10) | m;+}++function futhark_round_even(x) {+  var r = Math.floor(x);+  var d = x - r;+  return (d > 0.5 || (d == 0.5 && r % 2 == 1)) ? r + 1 : r;+}++function futhark_f16_value(b) {+  var sign = (b & 0x8000) ? -1 : 1;+  var e = (b >> 10) & 0x1f;+  var m = b & 0x3ff;+  if (e == 0) {+    return sign * m * 2 ** -24;+  } else if (e == 31) {+    return m == 0 ? sign * Infinity : NaN;+  } else {+    return sign * (1 + m / 1024) * 2 ** (e - 15);+  }+}++function futhark_check_int(x, bits, signed, what) {+  var ok;+  if (typeof x == "bigint") {+    ok = (signed ? BigInt.asIntN(bits, x) : BigInt.asUintN(bits, x)) == x;+  } else if (typeof x == "number") {+    var lo = signed ? -(2 ** (bits - 1)) : 0;+    var hi = signed ? 2 ** (bits - 1) : 2 ** bits;+    ok = Number.isInteger(x) && x >= lo && x < hi;+  } else {+    throw new TypeError(what + " must be a number or a BigInt");+  }+  if (!ok) {+    throw new RangeError(what + " is not a " + (signed ? "i" : "u") + bits + " value: " + x);+  }+}++// 'array' is used for values(), and 'c_array' for the C representation.+function futhark_int_prim(bits, signed, array) {+  var big = bits == 64;+  return {+    size: bits / 8,+    array: array,+    c_array: array,+    to_c: (x, what) => {+      futhark_check_int(x, bits, signed, what);+      return big ? BigInt(x) : Number(x);+    },+    get: (dv, p) => big+      ? (signed ? dv.getBigInt64(p, true) : dv.getBigUint64(p, true))+      : dv["get" + (signed ? "Int" : "Uint") + bits](p, true),+    to_values: (bytes) => new array(bytes.buffer),+  };+}++function futhark_float_prim(bits, array) {+  return {+    size: bits / 8,+    array: array,+    c_array: array,+    to_c: (x, what) => {+      if (typeof x != "number") {+        throw new TypeError(what + " must be a number");+      }+      return x;+    },+    get: (dv, p) => dv["getFloat" + bits](p, true),+    to_values: (bytes) => new array(bytes.buffer),+  };+}++// Float32Array represents all f16 values exactly.+var futhark_f16_array = globalThis.Float16Array ?? Float32Array;++var futhark_prims = {+  i8: futhark_int_prim(8, true, Int8Array),+  i16: futhark_int_prim(16, true, Int16Array),+  i32: futhark_int_prim(32, true, Int32Array),+  i64: futhark_int_prim(64, true, BigInt64Array),+  u8: futhark_int_prim(8, false, Uint8Array),+  u16: futhark_int_prim(16, false, Uint16Array),+  u32: futhark_int_prim(32, false, Uint32Array),+  u64: futhark_int_prim(64, false, BigUint64Array),+  f16: {+    size: 2,+    array: futhark_f16_array,+    c_array: Uint16Array,+    to_c: (x, what) => futhark_f16_bits(futhark_prims.f32.to_c(x, what)),+    get: (dv, p) => futhark_f16_value(dv.getUint16(p, true)),+    to_values: (bytes) =>+      futhark_f16_array.from(new Uint16Array(bytes.buffer), futhark_f16_value),+  },+  f32: futhark_float_prim(32, Float32Array),+  f64: futhark_float_prim(64, Float64Array),+  bool: {+    size: 1,+    array: Uint8Array,+    c_array: Uint8Array,+    to_c: (x, what) => {+      if (typeof x != "boolean") {+        throw new TypeError(what + " must be a boolean");+      }+      return x ? 1 : 0;+    },+    get: (dv, p) => dv.getUint8(p) != 0,+    to_values: (bytes) => bytes,+  },+};++// Booleans may also be 0 or 1, as returned by values().+function futhark_from_data(type, prim, data) {+  if (data instanceof prim.c_array && prim.c_array === prim.array && type != "bool") {+    return data;+  }+  var out = new prim.c_array(data.length);+  for (var i = 0; i < data.length; i++) {+    var x = data[i];+    if (type == "bool" && (x === 0 || x === 1)) {+      x = x === 1;+    }+    out[i] = prim.to_c(x, "Element " + i + " of the array data");+  }+  return out;+}++function futhark_check_value(obj, cls, ctx, type, what) {+  if (!(obj instanceof cls)) {+    throw new TypeError(what + " must be a " + cls.name);+  }+  if (obj.ctx !== ctx) {+    throw new Error(what + " belongs to a different FutharkContext");+  }+  if (obj.type !== type) {+    throw new TypeError(what + " has type " + obj.type + ", but must have type " + type);+  }+  obj._check_usable();+}++class FutharkValue {+  constructor(ctx, type, ptr) {+    this.ctx = ctx;+    this.type = type;+    this.ptr = ptr;+    this.consumed = false;+  }++  _check_usable() {+    this._check_not_freed();+    if (this.consumed) {+      throw new Error("Using a value that has been consumed by an entry point");+    }+  }++  _check_not_freed() {+    if (this.ptr === undefined) {+      throw new Error("Using a value that has been freed");+    }+    this.ctx._check_usable();+  }++  free() {+    this._check_not_freed();+    this.ctx._check_idle();+    this.ctx.wasm["_" + this.ctx.manifest.types[this.type].ops.free](this.ctx.ctx, this.ptr);+    this.ptr = undefined;+  }+}++class FutharkArray extends FutharkValue {+  shape() {+    this._check_usable();+    this.ctx._check_idle();+    return this._shape();+  }++  _shape() {+    var info = this.ctx.manifest.types[this.type];+    var p = this.ctx.wasm["_" + info.ops.shape](this.ctx.ctx, this.ptr);+    return new BigInt64Array(this.ctx.wasm.HEAPU8.slice(p, p + 8 * info.rank).buffer);+  }++  values() {+    var ctx = this.ctx;+    return ctx._exclusive(async () => {+      this._check_usable();+      var info = ctx.manifest.types[this.type];+      var prim = futhark_prims[info.elemtype];+      var bytes = Number(this._shape().reduce((a, b) => a * b, 1n)) * prim.size;+      var p = ctx._malloc(bytes);+      try {+        ctx._check(await ctx._call(info.ops.values, ctx.ctx, this.ptr, p));+        await ctx._sync();+        return prim.to_values(ctx.wasm.HEAPU8.slice(p, p + bytes));+      } finally {+        ctx.wasm._free(p);+      }+    });+  }+}++class FutharkOpaque extends FutharkValue {+}++class FutharkContext {+  constructor() {+    this.manifest = futhark_manifest;+    this.wasm = undefined;+    this.cfg = undefined;+    this.ctx = undefined;+    this.initialised = false;+    this.freed = false;+    this.queue = Promise.resolve();+    this.running = false;+    this.entry = Object.create(null);+    this.types = Object.create(null);+  }++  async init(options = {}) {+    if (this.initialised) {+      throw new Error("FutharkContext.init() may only be called once");+    }+    this.initialised = true;+    const flags = ["debugging", "logging", "profiling"];+    for (const option of Object.keys(options)) {+      if (option != "num_threads" && option != "tuning_params" && !flags.includes(option)) {+        throw new Error("Unknown FutharkContext.init() option: " + option);+      }+    }+    for (const flag of flags) {+      if (options[flag] !== undefined && typeof options[flag] != "boolean") {+        throw new TypeError("The " + flag + " option must be a boolean");+      }+    }+    if (options.num_threads !== undefined) {+      futhark_check_int(options.num_threads, 32, true, "The num_threads option");+    }+    const tuning_params = Object.entries(options.tuning_params ?? {});+    for (const [name, value] of tuning_params) {+      futhark_check_int(value, 32, false, "Tuning parameter " + name);+    }++    if (this.manifest.backend == "wasm_multicore" && globalThis.SharedArrayBuffer === undefined) {+      throw new Error("The wasm-multicore backend requires SharedArrayBuffer, which browsers " ++                      "only provide to pages that are cross-origin isolated");+    }+    try {+      this.wasm = await loadWASM();+    } catch (e) {+      throw e instanceof Error ? e : new Error("Failed to load the WebAssembly module: " + e);+    }+    if (options.num_threads !== undefined &&+        this.wasm._futhark_context_config_set_num_threads === undefined) {+      throw new Error("The num_threads option is not supported by this backend");+    }++    this.cfg = this.wasm._futhark_context_config_new();+    for (const flag of flags) {+      if (options[flag]) {+        this.wasm["_futhark_context_config_set_" + flag](this.cfg, 1);+      }+    }+    if (options.num_threads !== undefined) {+      this.wasm._futhark_context_config_set_num_threads(this.cfg, options.num_threads);+    }+    for (const [name, value] of tuning_params) {+      if (this.wasm.ccall("futhark_context_config_set_tuning_param", "number",+                          ["number", "string", "number"], [this.cfg, name, value]) != 0) {+        this.free();+        throw new Error("Unknown tuning parameter: " + name);+      }+    }+    var ctx = await this._call("futhark_context_new", this.cfg);+    var err = this._get_error(ctx);+    if (err !== null) {+      this.wasm._futhark_context_free(ctx);+      this.free();+      throw new Error(err);+    }+    this.ctx = ctx;++    for (const [name, info] of Object.entries(this.manifest.types)) {+      if (info.kind == "array") {+        const constructor = {+          from_data: (data, ...shape) => this._new_array(name, data, shape),+        };+        this.types[name] = constructor;+        this[info.elemtype + "_" + info.rank + "d"] = constructor;+      } else if (info.record !== undefined) {+        for (const field of info.record.fields) {+          this[field.project.replace(/^futhark_/, "")] =+            (obj) => this._exclusive(() => this._project(name, field, obj));+        }+      }+    }+    for (const [name, info] of Object.entries(this.manifest.entry_points)) {+      this.entry[name] = (...args) => this._exclusive(() => this._call_entry(name, info, args));+    }+  }++  free() {+    this._check_idle();+    if (this.ctx !== undefined) {+      this.wasm._futhark_context_free(this.ctx);+      this.ctx = undefined;+    }+    if (this.cfg !== undefined) {+      this.wasm._futhark_context_config_free(this.cfg);+      this.cfg = undefined;+    }+    this.freed = true;+  }++  context_sync() {+    return this._exclusive(async () => {+      this._check_usable();+      await this._sync();+    });+  }++  clear_caches() {+    return this._exclusive(async () => {+      this._check_usable();+      this._check(await this._call("futhark_context_clear_caches", this.ctx));+    });+  }++  report() {+    return this._exclusive(async () => {+      this._check_usable();+      return this._c_string(await this._call("futhark_context_report", this.ctx));+    });+  }++  pause_profiling() {+    return this._exclusive(async () => {+      this._check_usable();+      await this._call("futhark_context_pause_profiling", this.ctx);+    });+  }++  unpause_profiling() {+    return this._exclusive(async () => {+      this._check_usable();+      await this._call("futhark_context_unpause_profiling", this.ctx);+    });+  }++  // Everything below is internal.++  _check_usable() {+    if (this.freed) {+      throw new Error("The FutharkContext has been freed");+    }+    if (this.ctx === undefined) {+      throw new Error("The FutharkContext has not been initialised");+    }+  }++  // Run 'f' after all previously started asynchronous operations.+  _exclusive(f) {+    var run = this.queue.then(async () => {+      this.running = true;+      try {+        return await f();+      } finally {+        this.running = false;+      }+    });+    this.queue = run.catch(() => {});+    return run;+  }++  _check_idle() {+    if (this.running) {+      throw new Error("Cannot do this while an asynchronous operation on the FutharkContext is running");+    }+  }++  async _call(cfun, ...args) {+    return await this.wasm.ccall(cfun, "number", args.map(() => "number"), args, {async: true});+  }++  _malloc(n) {+    // malloc(0) may return NULL.+    var p = this.wasm._malloc(Math.max(n, 1));+    if (p == 0) {+      throw new RangeError("Failed to allocate " + n + " bytes");+    }+    return p;+  }++  _c_string(p) {+    var heap = this.wasm.HEAPU8;+    // Browsers' TextDecoder rejects shared memory.+    var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));+    this.wasm._free(p);+    return str;+  }++  _get_error(ctx = this.ctx) {+    var p = this.wasm._futhark_context_get_error(ctx);+    return p == 0 ? null : this._c_string(p);+  }++  _error() {+    return new Error(this._get_error() ?? "Unknown error");+  }++  _check(ret) {+    if (ret != 0) {+      throw this._error();+    }+  }++  async _sync() {+    this._check(await this._call("futhark_context_sync", this.ctx));+  }++  _to_c(type, x, what) {+    if (type in futhark_prims) {+      return futhark_prims[type].to_c(x, what);+    }+    var cls = this.manifest.types[type].kind == "array" ? FutharkArray : FutharkOpaque;+    futhark_check_value(x, cls, this, type, what);+    return x.ptr;+  }++  _from_c(type, p) {+    var dv = new DataView(this.wasm.HEAPU8.buffer);+    if (type in futhark_prims) {+      return futhark_prims[type].get(dv, p);+    }+    var ptr = dv.getUint32(p, true);+    if (this.manifest.types[type].kind == "array") {+      return new FutharkArray(this, type, ptr);+    } else {+      return new FutharkOpaque(this, type, ptr);+    }+  }++  _free_c(type, p) {+    if (!(type in futhark_prims)) {+      var ptr = new DataView(this.wasm.HEAPU8.buffer).getUint32(p, true);+      this.wasm["_" + this.manifest.types[type].ops.free](this.ctx, ptr);+    }+  }++  // For C functions with an output parameter after the context.+  async _call_with_output(cfun, type, args, after_call) {+    var out = this._malloc(8);+    try {+      var ret = await this._call(cfun, this.ctx, out, ...args);+      after_call();+      this._check(ret);+      try {+        await this._sync();+      } catch (e) {+        this._free_c(type, out);+        throw e;+      }+      return this._from_c(type, out);+    } finally {+      this.wasm._free(out);+    }+  }++  _new_array(type, data, shape) {+    this._check_usable();+    this._check_idle();+    var info = this.manifest.types[type];+    var prim = futhark_prims[info.elemtype];+    if (shape.length != info.rank) {+      throw new TypeError("Expected " + info.rank + " dimensions, but got " + shape.length);+    }+    shape = shape.map((d, i) => {+      futhark_check_int(d, 64, true, "dimension " + i);+      if (d < 0) {+        throw new RangeError("dimension " + i + " is negative: " + d);+      }+      return BigInt(d);+    });+    if (!(data instanceof Array || (ArrayBuffer.isView(data) && !(data instanceof DataView)))) {+      throw new TypeError("Array data must be an Array or a typed array");+    }+    var n = shape.reduce((a, b) => a * b, 1n);+    if (BigInt(data.length) != n) {+      throw new RangeError("Array data has " + data.length + " elements, but the shape requires " + n);+    }+    var elems = futhark_from_data(info.elemtype, prim, data);+    var bytes = new Uint8Array(elems.buffer, elems.byteOffset, elems.byteLength);+    var p = this._malloc(bytes.length);+    try {+      this.wasm.HEAPU8.set(bytes, p);+      var ptr = this.wasm["_" + info.ops.new](this.ctx, p, ...shape);+      if (ptr == 0) {+        throw this._error();+      }+      return new FutharkArray(this, type, ptr);+    } finally {+      this.wasm._free(p);+    }+  }++  async _project(type, field, obj) {+    this._check_usable();+    futhark_check_value(obj, FutharkOpaque, this, type, "The record");+    return await this._call_with_output(field.project, field.type, [obj.ptr], () => {});+  }++  async _call_entry(name, info, args) {+    this._check_usable();+    if (args.length != info.inputs.length) {+      throw new TypeError("Entry point " + name + " expects " + info.inputs.length ++                          " arguments, but got " + args.length);+    }+    var c_args = info.inputs.map((input, i) =>+      this._to_c(input.type, args[i], "Argument " + i + " of entry point " + name));+    info.inputs.forEach((input, i) => {+      if (input.consumed && args.some((x, j) => j != i && x === args[i])) {+        throw new Error("Argument " + i + " of entry point " + name ++                        " is consumed, so it must not also be passed as another argument");+      }+    });+    var mark_consumed = () => info.inputs.forEach((input, i) => {+      if (input.consumed) {+        args[i].consumed = true;+      }+    });+    return await this._call_with_output(info.cfun, info.output.type, c_args, mark_consumed);+  }+}++// End of wrappers.js
rts/opencl/copy.cl view
@@ -2,9 +2,7 @@  #define GEN_COPY_KERNEL(NAME, ELEM_TYPE) \ FUTHARK_KERNEL void lmad_copy_##NAME(SHARED_MEM_PARAM                   \-                               __global ELEM_TYPE *dst_mem,             \                                int64_t dst_offset,                      \-                               __global ELEM_TYPE *src_mem,             \                                int64_t src_offset,                      \                                int64_t n,                               \                                int r,                                   \@@ -15,7 +13,9 @@                                int64_t shape4, int64_t dst_stride4, int64_t src_stride4, \                                int64_t shape5, int64_t dst_stride5, int64_t src_stride5, \                                int64_t shape6, int64_t dst_stride6, int64_t src_stride6, \-                               int64_t shape7, int64_t dst_stride7, int64_t src_stride7) { \+                               int64_t shape7, int64_t dst_stride7, int64_t src_stride7, \+                               __global ELEM_TYPE *dst_mem,             \+                               __global ELEM_TYPE *src_mem) { \   int64_t gtid = get_global_id(0);                                      \   int64_t remainder = gtid;                                             \                                                                         \
rts/opencl/transpose.cl view
@@ -3,9 +3,7 @@ #define GEN_TRANSPOSE_KERNELS(NAME, ELEM_TYPE)                          \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\ void map_transpose_##NAME(SHARED_MEM_PARAM                              \-                          __global ELEM_TYPE *dst_mem,                  \                           int64_t dst_offset,                           \-                          __global ELEM_TYPE *src_mem,                  \                           int64_t src_offset,                           \                           int32_t num_arrays,                           \                           int32_t x_elems,                              \@@ -17,7 +15,9 @@                           int32_t mulx,                                 \                           int32_t muly,                                 \                           int32_t repeat_1,                             \-                          int32_t repeat_2) {                           \+                          int32_t repeat_2,                             \+                          __global ELEM_TYPE *dst_mem,                  \+                          __global ELEM_TYPE *src_mem) {                \   (void)mulx; (void)muly;                                               \   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \   int tblock_id_0 = get_tblock_id(0);                                   \@@ -29,6 +29,8 @@     int global_id_2 = get_global_id(2);                                 \     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \       if (tblock_id_2 >= num_arrays) { break; }                         \+      /* The previous iteration may still be reading shared memory. */  \+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \       int32_t x_index = global_id_0;                                    \@@ -66,21 +68,21 @@                                                                         \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1)                     \ void map_transpose_##NAME##_low_height(SHARED_MEM_PARAM                 \-                                                __global ELEM_TYPE *dst_mem, \-                                                int64_t dst_offset,     \-                                                __global ELEM_TYPE *src_mem, \-                                                int64_t src_offset,     \-                                                int32_t num_arrays,     \-                                                int32_t x_elems,        \-                                                int32_t y_elems,        \-                                                int32_t dst_array_stride, \-                                                int32_t dst_row_stride, \-                                                int32_t src_array_stride, \-                                                int32_t src_row_stride, \-                                                int32_t mulx,           \-                                                int32_t muly,           \-                                                int32_t repeat_1,       \-                                                int32_t repeat_2) {     \+                                       int64_t dst_offset,              \+                                       int64_t src_offset,              \+                                       int32_t num_arrays,              \+                                       int32_t x_elems,                 \+                                       int32_t y_elems,                 \+                                       int32_t dst_array_stride,        \+                                       int32_t dst_row_stride,          \+                                       int32_t src_array_stride,        \+                                       int32_t src_row_stride,          \+                                       int32_t mulx,                    \+                                       int32_t muly,                    \+                                       int32_t repeat_1,                \+                                       int32_t repeat_2,                \+                                       __global ELEM_TYPE *dst_mem,     \+                                       __global ELEM_TYPE *src_mem) {   \   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \   int tblock_id_0 = get_tblock_id(0);                                   \   int global_id_0 = get_global_id(0);                                   \@@ -91,6 +93,8 @@     int global_id_2 = get_global_id(2);                                 \     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \       if (tblock_id_2 >= num_arrays) { break; }                         \+      /* The previous iteration may still be reading shared memory. */  \+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \       int32_t x_index =                                                 \@@ -124,9 +128,7 @@                                                                         \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1)                     \ void map_transpose_##NAME##_low_width(SHARED_MEM_PARAM                  \-                                      __global ELEM_TYPE *dst_mem,      \                                       int64_t dst_offset,               \-                                      __global ELEM_TYPE *src_mem,      \                                       int64_t src_offset,               \                                       int32_t num_arrays,               \                                       int32_t x_elems,                  \@@ -138,7 +140,9 @@                                       int32_t mulx,                     \                                       int32_t muly,                     \                                       int32_t repeat_1,                 \-                                      int32_t repeat_2) {               \+                                      int32_t repeat_2,                 \+                                      __global ELEM_TYPE *dst_mem,      \+                                      __global ELEM_TYPE *src_mem) {    \   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \   int tblock_id_0 = get_tblock_id(0);                                   \   int global_id_0 = get_global_id(0);                                   \@@ -149,6 +153,8 @@     int global_id_2 = get_global_id(2);                                 \     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \       if (tblock_id_2 >= num_arrays) { break; }                         \+      /* The previous iteration may still be reading shared memory. */  \+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \       int32_t x_index = tblock_id_0 * TR_BLOCK_DIM + get_local_id(0)/muly; \@@ -178,10 +184,8 @@ }                                                                       \                                                                         \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*TR_BLOCK_DIM, 1, 1)                   \-void map_transpose_##NAME##_small(SHARED_MEM_PARAM                       \-                                  __global ELEM_TYPE *dst_mem,          \+void map_transpose_##NAME##_small(SHARED_MEM_PARAM                      \                                   int64_t dst_offset,                   \-                                  __global ELEM_TYPE *src_mem,          \                                   int64_t src_offset,                   \                                   int32_t num_arrays,                   \                                   int32_t x_elems,                      \@@ -193,7 +197,9 @@                                   int32_t mulx,                         \                                   int32_t muly,                         \                                   int32_t repeat_1,                     \-                                  int32_t repeat_2) {                   \+                                  int32_t repeat_2,                     \+                                  __global ELEM_TYPE *dst_mem,          \+                                  __global ELEM_TYPE *src_mem) {        \   (void)mulx; (void)muly;                                               \   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \   int tblock_id_0 = get_tblock_id(0);                                   \@@ -224,9 +230,7 @@                                                                         \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\ void map_transpose_##NAME##_large(SHARED_MEM_PARAM                      \-                                  __global ELEM_TYPE *dst_mem,          \                                   int64_t dst_offset,                   \-                                  __global ELEM_TYPE *src_mem,          \                                   int64_t src_offset,                   \                                   int64_t num_arrays,                   \                                   int64_t x_elems,                      \@@ -238,9 +242,11 @@                                   int64_t mulx,                         \                                   int64_t muly,                         \                                   int32_t repeat_1,                     \-                                  int32_t repeat_2) {                   \+                                  int32_t repeat_2,                     \+                                  __global ELEM_TYPE *dst_mem,          \+                                  __global ELEM_TYPE *src_mem) {        \   (void)mulx; (void)muly;                                               \-  __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;             \+  __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \   int tblock_id_0 = get_tblock_id(0);                                   \   int global_id_0 = get_global_id(0);                                   \   int tblock_id_1 = get_tblock_id(1);                                   \@@ -250,6 +256,8 @@     int global_id_2 = get_global_id(2);                                 \     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \       if (tblock_id_2 >= num_arrays) { break; }                         \+      /* The previous iteration may still be reading shared memory. */  \+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \       int64_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \       int64_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \       int64_t x_index = global_id_0;                                    \
rts/python/opencl.py view
@@ -183,6 +183,14 @@     )     self.failure_is_an_option = np.int32(0) +    self.bookkeeping_counters = self.pool.allocate(+        np.int32().itemsize * self.bookkeeping_counters_size+    )+    self.bookkeeping_locks = self.pool.allocate(+        np.int32().itemsize * self.bookkeeping_locks_size+    )+    zero_bookkeeping(self)+     if "default_group_size" in user_sizes:         default_group_size = user_sizes["default_group_size"]         del user_sizes["default_group_size"]@@ -389,6 +397,23 @@     self.pool.free_held()  +def zero_bookkeeping(self):+    cl.enqueue_fill_buffer(+        self.queue,+        self.bookkeeping_counters,+        np.int32(0),+        0,+        np.int32().itemsize * self.bookkeeping_counters_size,+    )+    cl.enqueue_fill_buffer(+        self.queue,+        self.bookkeeping_locks,+        np.int32(0),+        0,+        np.int32().itemsize * self.bookkeeping_locks_size,+    )++ def sync(self):     failure = np.empty(1, dtype=np.int32)     cl.enqueue_copy(self.queue, failure, self.global_failure, is_blocking=True)@@ -403,6 +428,9 @@             np.int32().itemsize,         ) +        # The failing kernel may have left the bookkeeping buffers nonzero.+        zero_bookkeeping(self)+         # Read failure args.         failure_args = np.empty(             self.global_failure_args_max + 1, dtype=np.int64@@ -441,9 +469,7 @@      kernel.set_args(         cl.LocalMemory(TR_TILE_DIM * (TR_TILE_DIM + 1) * elem_size),-        dst,         dst_offset,-        src,         src_offset,         k32,         m32,@@ -453,6 +479,8 @@         muly32,         np.int32(0),         np.int32(0),+        dst,+        src,     )     cl.enqueue_nd_range_kernel(self.queue, kernel, dims, group_dims) @@ -490,13 +518,13 @@     kernel = self.copy_kernels[elem_size]     kernel.set_args(         cl.LocalMemory(1),-        dst,         dst_offset,-        src,         src_offset,         n,         np.int32(r),         *layout_args,+        dst,+        src,     )     w = 256     dims = ((n + w - 1) // w * w,)
rts/python/scalar.py view
@@ -409,6 +409,13 @@         return ssignum(x)  +def sitofp_T_f16(x):+    return np.float16(x)+++sitofp_i8_f16 = sitofp_i16_f16 = sitofp_i32_f16 = sitofp_i64_f16 = sitofp_T_f16++ def sitofp_T_f32(x):     return np.float32(x) @@ -423,6 +430,13 @@ sitofp_i8_f64 = sitofp_i16_f64 = sitofp_i32_f64 = sitofp_i64_f64 = sitofp_T_f64  +def uitofp_T_f16(x):+    return np.float36(unsigned(x))+++uitofp_i8_f16 = uitofp_i16_f16 = uitofp_i32_f16 = uitofp_i64_f16 = uitofp_T_f16++ def uitofp_T_f32(x):     return np.float32(unsigned(x)) @@ -481,7 +495,7 @@     if np.isnan(x) or x < 0 or x >= (1 << 8):         return np.int8(0)     else:-        return np.int8(np.trunc(x))+        return np.uint8(np.trunc(x)).astype(np.int8)   fptoui_f16_i8 = fptoui_f32_i8 = fptoui_f64_i8 = fptoui_T_i8@@ -491,7 +505,7 @@     if np.isnan(x) or x < 0 or x >= (1 << 16):         return np.int16(0)     else:-        return np.int16(np.trunc(x))+        return np.uint16(np.trunc(x)).astype(np.int16)   fptoui_f16_i16 = fptoui_f32_i16 = fptoui_f64_i16 = fptoui_T_i16@@ -501,7 +515,7 @@     if np.isnan(x) or x < 0 or x >= (1 << 32):         return np.int32(0)     else:-        return np.int32(np.trunc(x))+        return np.uint32(np.trunc(x)).astype(np.int32)   fptoui_f16_i32 = fptoui_f32_i32 = fptoui_f64_i32 = fptoui_T_i32@@ -511,7 +525,7 @@     if np.isnan(x) or x < 0 or x >= (1 << 64):         return np.int64(0)     else:-        return np.int64(np.trunc(x))+        return np.uint64(np.trunc(x)).astype(np.int64)   fptoui_f16_i64 = fptoui_f32_i64 = fptoui_f64_i64 = fptoui_T_i64
src/Futhark/Actions.hs view
@@ -33,12 +33,14 @@ import Control.Monad import Control.Monad.IO.Class import Control.Monad.State (get)+import Data.Aeson qualified as JSON import Data.Bifunctor import Data.ByteString.Lazy.Char8 qualified as BS import Data.List (intercalate) import Data.Map qualified as M import Data.Maybe (fromMaybe) import Data.Text qualified as T+import Data.Text.Encoding qualified as T import Data.Text.IO qualified as T import Futhark.Analysis.AccessPattern import Futhark.Analysis.Alias@@ -49,6 +51,8 @@ import Futhark.Analysis.Metrics import Futhark.CodeGen.Backends.CCUDA qualified as CCUDA import Futhark.CodeGen.Backends.COpenCL qualified as COpenCL+import Futhark.CodeGen.Backends.GenericC qualified as GC+import Futhark.CodeGen.Backends.GenericWASM qualified as GenericWASM import Futhark.CodeGen.Backends.HIP qualified as HIP import Futhark.CodeGen.Backends.MulticoreC qualified as MulticoreC import Futhark.CodeGen.Backends.MulticoreISPC qualified as MulticoreISPC@@ -532,30 +536,46 @@ cmdEMCFLAGS :: [String] -> [String] cmdEMCFLAGS def = maybe def words $ lookup "EMCFLAGS" unixEnvironment -runEMCC :: String -> String -> FilePath -> [String] -> [String] -> [String] -> Bool -> FutharkM ()-runEMCC cpath outpath classpath cflags_def ldflags expfuns lib = do+-- | The typed views of the WASM heap, which the generated JavaScript+-- uses to move array data in and out of the module.+heapExports :: [String]+heapExports =+  [ "'HEAP8'",+    "'HEAP16'",+    "'HEAP32'",+    "'HEAP64'",+    "'HEAPU8'",+    "'HEAPU16'",+    "'HEAPU32'",+    "'HEAPU64'",+    "'HEAPF32'",+    "'HEAPF64'"+  ]++-- | Options passed to @emcc@ by both WASM backends.+wasmFlags :: FilePath -> [String] -> [String]+wasmFlags classpath expfuns =+  [ "--extern-post-js",+    classpath,+    -- Called by the generated JavaScript.+    "-sEXPORT_NAME=loadWASM",+    "-sWASM_BIGINT",+    -- Up to 2GiB; more would need unsigned pointers in JavaScript.+    "-sALLOW_MEMORY_GROWTH",+    "-sEXPORTED_RUNTIME_METHODS=[" ++ intercalate "," ("'ccall'" : heapExports) ++ "]",+    "-sEXPORTED_FUNCTIONS=[" ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns) ++ "]"+  ]++runEMCC :: String -> String -> [String] -> [String] -> [String] -> FutharkM ()+runEMCC cpath outpath cflags_def ldflags extra_flags = do   ret <-     liftIO $       runProgramWithExitCode         "emcc"         ( [cpath, "-o", outpath]-            ++ ["-lnodefs.js"]-            ++ ["-s", "--extern-post-js", classpath]-            ++ ( if lib-                   then ["-s", "EXPORT_NAME=loadWASM"]-                   else []-               )-            ++ ["-s", "WASM_BIGINT"]-            ++ [ "-s",-                 "EXPORTED_RUNTIME_METHODS=['HEAP8','HEAP16','HEAP32','HEAP64','HEAPU8','HEAPU16','HEAPU32','HEAPF32','HEAPF64']"-               ]+            ++ extra_flags             ++ cmdCFLAGS cflags_def             ++ cmdEMCFLAGS [""]-            ++ [ "-s",-                 "EXPORTED_FUNCTIONS=["-                   ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns)-                   ++ "]"-               ]             -- The default LDFLAGS are always added.             ++ ldflags         )@@ -572,73 +592,63 @@     Right (ExitSuccess, _, _) ->       pure () +-- | Compile to the ES module @outpath.mjs@ with emcc.  Unless compiling a+-- library, @outpath@ is an executable that runs it as a server.+compileWASM ::+  CompilerMode ->+  FilePath ->+  [String] ->+  (GC.CParts, T.Text, [String]) ->+  FutharkM ()+compileWASM mode outpath ldflags (cprog, jsprog, exps) = do+  let (_, impl, _) = GC.asLibrary cprog+  liftIO $ T.writeFile cpath impl+  liftIO $ T.writeFile classpath $ jsprog <> extra+  runEMCC cpath mjspath ["-O3", "-msimd128"] ldflags (wasmFlags classpath exps)+  when (mode /= ToLibrary) . liftIO $ do+    T.writeFile outpath $ wasmLauncher $ takeFileName mjspath+    perms <- getPermissions outpath+    setPermissions outpath $ setOwnerExecutable True perms+  where+    extra = case mode of+      ToLibrary -> GenericWASM.libraryExports+      _ -> GenericWASM.runServer+    cpath = outpath `addExtension` "c"+    mjspath = outpath `addExtension` "mjs"+    classpath = outpath `addExtension` "class.js"++-- | A Node.js script that runs the given ES module.  Unlike the module+-- itself, it works whatever module type @package.json@ specifies.+wasmLauncher :: FilePath -> T.Text+wasmLauncher mjsname =+  T.unlines+    [ "#!/usr/bin/env node",+      "import(\"./\" + encodeURIComponent(" <> jsString <> "));"+    ]+  where+    jsString = T.decodeUtf8 $ BS.toStrict $ JSON.encode mjsname+ -- | The @futhark wasm@ action. compileCtoWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action SeqMem compileCtoWASMAction fcfg mode outpath =   Action-    { actionName = "Compile to sequential C",-      actionDescription = "Compile to sequential C",-      actionProcedure = helper+    { actionName = "Compile to WebAssembly",+      actionDescription = "Compile to sequential WebAssembly",+      actionProcedure =+        compileWASM mode outpath ["-lm"]+          <=< handleWarnings fcfg . SequentialWASM.compileProg versionString     }-  where-    helper prog = do-      (cprog, jsprog, exps) <--        handleWarnings fcfg $ SequentialWASM.compileProg versionString prog-      case mode of-        ToLibrary -> do-          writeLibs cprog jsprog-          liftIO $ T.appendFile classpath SequentialWASM.libraryExports-          runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm"] exps True-        _ -> do-          -- Non-server executables are not supported.-          writeLibs cprog jsprog-          liftIO $ T.appendFile classpath $ SequentialWASM.runServer $ SequentialWASM.cManifest cprog-          runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm"] exps False-    writeLibs cprog jsprog = do-      let (h, imp, _) = SequentialC.asLibrary cprog-      liftIO $ T.writeFile hpath h-      liftIO $ T.writeFile cpath imp-      liftIO $ T.writeFile classpath jsprog -    cpath = outpath `addExtension` "c"-    hpath = outpath `addExtension` "h"-    mjspath = outpath `addExtension` "mjs"-    classpath = outpath `addExtension` ".class.js"- -- | The @futhark wasm-multicore@ action. compileMulticoreToWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action MCMem compileMulticoreToWASMAction fcfg mode outpath =   Action-    { actionName = "Compile to sequential C",-      actionDescription = "Compile to sequential C",-      actionProcedure = helper+    { actionName = "Compile to WebAssembly",+      actionDescription = "Compile to multicore WebAssembly",+      actionProcedure =+        compileWASM mode outpath ["-lm", "-pthread"]+          <=< handleWarnings fcfg . MulticoreWASM.compileProg versionString     }-  where-    helper prog = do-      (cprog, jsprog, exps) <--        handleWarnings fcfg $ MulticoreWASM.compileProg versionString prog--      case mode of-        ToLibrary -> do-          writeLibs cprog jsprog-          liftIO $ T.appendFile classpath MulticoreWASM.libraryExports-          runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps True-        _ -> do-          -- Non-server executables are not supported.-          writeLibs cprog jsprog-          liftIO $ T.appendFile classpath $ MulticoreWASM.runServer $ MulticoreWASM.cManifest cprog-          runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps False--    writeLibs cprog jsprog = do-      let (h, imp, _) = MulticoreC.asLibrary cprog-      liftIO $ T.writeFile hpath h-      liftIO $ T.writeFile cpath imp-      liftIO $ T.writeFile classpath jsprog--    cpath = outpath `addExtension` "c"-    hpath = outpath `addExtension` "h"-    mjspath = outpath `addExtension` "mjs"-    classpath = outpath `addExtension` ".class.js"  runWith ::   Name ->
src/Futhark/Analysis/Interference.hs view
@@ -117,6 +117,10 @@           defbody : map caseBody cases     Loop merge _ body ->       analyseLoopParams merge <$> analyseBody lumap inuse_outside body+    -- The memory of the accumulated arrays is that of the pattern, and is thus+    -- in use throughout the lambda.+    WithAcc _ lam ->+      analyseLambda lumap inuse_outside lam     Op (Inner (SegOp segop)) -> do       analyseSegOp lumap inuse_outside segop     _ ->
src/Futhark/CodeGen/Backends/GPU.hs view
@@ -159,6 +159,16 @@   GC.stm [C.cstm|$id:v = $exp:e;|] callKernel (LaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size) =   genLaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size+callKernel (GetBookkeeping v b) =+  GC.stm+    [C.cstm|$id:v = (struct memblock_device) {.references = NULL,+                                              .mem = $exp:buf,+                                              .size = $int:(4 * bookkeepingSize b),+                                              .desc = $string:desc};|]+  where+    (buf, desc) = case b of+      BookkeepingCounters -> ([C.cexp|ctx->bookkeeping_counters|], "bookkeeping_counters")+      BookkeepingLocks -> ([C.cexp|ctx->bookkeeping_locks|], "bookkeeping_locks")  copygpu2gpu :: GC.DoCopy op s copygpu2gpu _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do@@ -472,6 +482,8 @@   mapM_     GC.earlyDecl     [C.cunit|static const int max_failure_args = $int:max_failure_args;+             static const typename int64_t bookkeeping_counters_size = $int:(bookkeepingSize BookkeepingCounters);+             static const typename int64_t bookkeeping_locks_size = $int:(bookkeepingSize BookkeepingLocks);              static const int f64_required = $exp:f64_required;              static const char *gpu_program[] = {$inits:program_fragments};              $esc:(T.unpack gpuPrototypesH)
src/Futhark/CodeGen/Backends/GenericC.hs view
@@ -50,16 +50,18 @@ defCall dests fname args = do   let out_args = [[C.cexp|&$id:d|] | d <- dests]       args' = [C.cexp|ctx|] : out_args ++ args-  item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; goto cleanup; }|]+  failure <- failureJump+  item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; $stm:failure }|]  defError :: ErrorCompiler op s defError msg stacktrace = do   (formatstr, formatargs) <- errorMsgString msg   let formatstr' = "Error: " <> formatstr <> "\n\nBacktrace:\n%s"+  failure <- failureJump   items     [C.citems|set_error(ctx, msgprintf($string:formatstr', $args:formatargs, $string:stacktrace));               err = FUTHARK_PROGRAM_ERROR;-              goto cleanup;|]+              $stm:failure|]  lmadcopyCPU :: DoCopy op s lmadcopyCPU _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do@@ -657,6 +659,7 @@       compileCode init_consts     decl_mem <- declAllocatedMem     free_mem <- freeAllocatedMem+    error_label <- errorLabel     libDecl       [C.cedecl|static int init_constants($ty:ctx_ty *ctx) {         (void)ctx;@@ -666,6 +669,7 @@         $items:init_consts'         $items:free_mem         $items:undefs+        $items:error_label         cleanup:         return err;       }|]@@ -700,7 +704,7 @@      freeConst ScalarParam {} = pure ()     freeConst (MemParam name space) =-      unRefMem [C.cexp|ctx->constants->$id:name|] space [C.cstm|err = 1;|]+      unRefMem [C.cexp|ctx->constants->$id:name|] space $ pure [C.cstm|err = 1;|]      getConst (ScalarParam name bt) = do       let ctp = primTypeToCType bt
src/Futhark/CodeGen/Backends/GenericC/Code.hs view
@@ -286,6 +286,25 @@   stm [C.cstm|if (!$exp:e') { $items:err }|]   where     stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs+compileCode (OnFailure body handler) = do+  -- Failures in the body jump to its own error label, after which the handler+  -- runs and the failure continues to the enclosing error label.+  (body', error_label) <- withoutPendingFailures $ do+    body' <- collect $ compileCode body+    (body',) <$> errorLabel+  if null error_label+    then items [C.citems|{ $items:body' }|]+    else do+      handler' <- collect $ compileCode handler+      failure <- failureJump+      no_failure <- newVName "no_failure"+      items+        [C.citems|{ $items:body' }+                  goto $id:no_failure;+                  $items:error_label+                  { $items:handler' }+                  $stm:failure+                  $id:no_failure: ;|] compileCode (Allocate _ _ ScalarSpace {}) =   -- Handled by the declaration of the memory block, which is   -- translated to an actual array.@@ -294,19 +313,20 @@   size <- compileExp e   cached <- cacheMem name   case cached of-    Just cur_size ->+    Just cur_size -> do+      failure <- failureJump       stm         [C.cstm|if ($exp:cur_size < $exp:size) {                  err = lexical_realloc(ctx, &$exp:name, &$exp:cur_size, $exp:size);                  if (err != FUTHARK_SUCCESS) {-                   goto cleanup;+                   $stm:failure                  }                 }|]     _ ->-      allocMem name size space [C.cstm|{err = 1; goto cleanup;}|]+      allocMem name size space memoryFailure compileCode (Free name space) = do   cached <- isJust <$> cacheMem name-  unless cached $ unRefMem name space [C.cstm|{err = 1; goto cleanup;}|]+  unless cached $ unRefMem name space memoryFailure compileCode (For i bound body) = do   let i' = C.toIdent i       t = primTypeToCType $ primExpType bound@@ -403,7 +423,7 @@   src' <- compileExp src   stm [C.cstm|$id:dest = $exp:src';|] compileCode (SetMem dest src space) =-  setMem dest src space [C.cstm|{err = 1; goto cleanup;}|]+  setMem dest src space compileCode (Call [dest] fname args)   | isBuiltInFunction fname = do       args' <- mapM compileArg args
src/Futhark/CodeGen/Backends/GenericC/Fun.hs view
@@ -32,7 +32,7 @@     setRetVal' p (MemParam name space) =       -- It is required that the memory block is already initialised       -- (although it may be NULL).-      setMem [C.cexp|*$exp:p|] name space [C.cstm|{err = 1; goto cleanup;}|]+      setMem [C.cexp|*$exp:p|] name space     setRetVal' p (ScalarParam name _) =       stm [C.cstm|*$exp:p = $id:name;|] @@ -64,6 +64,7 @@     body' <- collect $ compileFunBody out_ptrs outputs body     decl_mem <- declAllocatedMem     free_mem <- freeAllocatedMem+    error_label <- errorLabel     let futhark_function =           C.DeclSpec [] [C.EscTypeQual "FUTHARK_FUN_ATTR" mempty] (C.Tint Nothing mempty) mempty @@ -76,6 +77,7 @@                $items:decl_mem                $items:get_constants                $items:body'+               $items:error_label               cleanup:                {                $stms:free_cached
src/Futhark/CodeGen/Backends/GenericC/Monad.hs view
@@ -29,6 +29,10 @@     generateProgramStruct,     runCompilerM,     inNewFunction,+    failureJump,+    memoryFailure,+    errorLabel,+    withoutPendingFailures,     cachingMemory,     volQuals,     rawMem,@@ -116,7 +120,10 @@     compCtxFields :: DL.DList (C.Id, C.Type, Maybe C.Exp, Maybe (C.Stm, C.Stm)),     compClearItems :: DL.DList C.BlockItem,     compDeclaredMem :: [(VName, Space)],-    compItems :: DL.DList C.BlockItem+    compItems :: DL.DList C.BlockItem,+    -- | The label that the failures since the previous 'errorLabel' jump to, if+    -- there have been any.+    compErrorLabel :: Maybe VName   }  newCompilerState :: VNameSource -> s -> CompilerState s@@ -131,7 +138,8 @@       compCtxFields = mempty,       compClearItems = mempty,       compDeclaredMem = mempty,-      compItems = mempty+      compItems = mempty,+      compErrorLabel = Nothing     }  -- | In which part of the header file we put the declaration.  This is@@ -252,7 +260,7 @@ freeAllocatedMem :: CompilerM op s [C.BlockItem] freeAllocatedMem = collect $ mapM_ free =<< gets compDeclaredMem   where-    free (name, space) = unRefMem name space [C.cstm|err = 1;|]+    free (name, space) = unRefMem name space $ pure [C.cstm|err = 1;|]  declAllocatedMem :: CompilerM op s [C.BlockItem] declAllocatedMem = collect $ mapM_ f =<< gets compDeclaredMem@@ -383,12 +391,50 @@ inNewFunction m = do   old_mem <- gets compDeclaredMem   modify $ \s -> s {compDeclaredMem = mempty}-  x <- local noCached m+  x <- local noCached $ withoutPendingFailures m   modify $ \s -> s {compDeclaredMem = old_mem}   pure x   where     noCached env = env {envCachedMem = mempty} +-- | A statement that jumps to the next error label; see 'errorLabel'. Don't do+-- this without setting @err@ or similar first.+failureJump :: CompilerM op s C.Stm+failureJump = do+  label <- maybe (newVName "failure") pure =<< gets compErrorLabel+  modify $ \s -> s {compErrorLabel = Just label}+  pure [C.cstm|goto $id:label;|]++-- | What do do after failing a memory operation: set @err@ and jump as by+-- 'failureJump'.+memoryFailure :: CompilerM op s C.Stm+memoryFailure = do+  failure <- failureJump+  pure [C.cstm|{err = 1; $stm:failure}|]++-- | The label that the failures since the previous error label jump to, if+-- there have been any. A failure thus jumps to the first error label that+-- follows it in the program text. Every function that may fail must end with an+-- error label just before its @cleanup@ label.+errorLabel :: CompilerM op s [C.BlockItem]+errorLabel = do+  label <- gets compErrorLabel+  modify $ \s -> s {compErrorLabel = Nothing}+  pure $ case label of+    Nothing -> []+    Just label' -> [C.citems|$id:label': ;|]++-- | Run an action as if no failures had happened before it, and afterwards+-- restore the failures that had. The action must place the error labels for its+-- own failures.+withoutPendingFailures :: CompilerM op s a -> CompilerM op s a+withoutPendingFailures m = do+  old_label <- gets compErrorLabel+  modify $ \s -> s {compErrorLabel = Nothing}+  x <- m+  modify $ \s -> s {compErrorLabel = old_label}+  pure x+ -- | Insert a block item in the generated code at this point. item :: C.BlockItem -> CompilerM op s () item x = modify $ \s -> s {compItems = DL.snoc (compItems s) x}@@ -589,12 +635,14 @@       when refcount $         stm [C.cstm|$exp:mem.references = NULL;|] -setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> C.Stm -> CompilerM op s ()-setMem dest src space on_failure = do+-- | Set @dest@ to @src@.  A failure is handled by 'memoryFailure'.+setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> CompilerM op s ()+setMem dest src space = do   refcount <- fatMemory space   let src_s = T.unpack $ expText $ C.toExp src noLoc   if refcount-    then+    then do+      on_failure <- memoryFailure       stm         [C.cstm|if ($id:(fatMemSet space)(ctx, &$exp:dest, &$exp:src,                                                $string:src_s) != 0) {@@ -613,15 +661,17 @@                   }|]       _ -> stm [C.cstm|$exp:dest = $exp:src;|] -unRefMem :: (C.ToExp a) => a -> Space -> C.Stm -> CompilerM op s ()+-- | The failure code is only generated if it is needed.+unRefMem :: (C.ToExp a) => a -> Space -> CompilerM op s C.Stm -> CompilerM op s () unRefMem mem space on_failure = do   refcount <- fatMemory space   cached <- isJust <$> cacheMem mem   let mem_s = T.unpack $ expText $ C.toExp mem noLoc-  when (refcount && not cached) $+  when (refcount && not cached) $ do+    on_failure' <- on_failure     stm       [C.cstm|if ($id:(fatMemUnRef space)(ctx, &$exp:mem, $string:mem_s) != 0) {-                  $stm:on_failure+                  $stm:on_failure'                 }|]  allocMem ::@@ -629,22 +679,23 @@   a ->   b ->   Space ->-  C.Stm ->+  CompilerM op s C.Stm ->   CompilerM op s () allocMem mem size space on_failure = do   refcount <- fatMemory space   let mem_s = T.unpack $ expText $ C.toExp mem noLoc+  on_failure' <- on_failure   if refcount     then       stm         [C.cstm|if ($id:(fatMemAlloc space)(ctx, &$exp:mem, $exp:size,                                                  $string:mem_s)) {-                       $stm:on_failure+                       $stm:on_failure'                      }|]     else do       freeRawMem mem size space mem_s       alloc <- allocRawMem mem size space [C.cexp|desc|]-      stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure }|]+      stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure' }|]  copyMemoryDefaultSpace ::   C.Exp ->
src/Futhark/CodeGen/Backends/GenericC/Types.hs view
@@ -56,7 +56,7 @@     [C.cexp|$exp:arr->mem|]     [C.cexp|$exp:arr_size * $int:(primByteSize pt::Int)|]     space-    [C.cstm|err = 1;|]+    (pure [C.cstm|err = 1;|])  arrayLibraryFunctions ::   Publicness ->@@ -114,7 +114,7 @@       let dim_s = "dim" ++ show i        in stm [C.cstm|arr->shape[$int:i] = $id:dim_s;|] -  free_body <- collect $ unRefMem [C.cexp|arr->mem|] space [C.cstm|ret = 1;|]+  free_body <- collect $ unRefMem [C.cexp|arr->mem|] space $ pure [C.cstm|ret = 1;|]    values_body <-     collect $
src/Futhark/CodeGen/Backends/GenericPython.hs view
@@ -1355,6 +1355,10 @@   cond' <- compileExp $ Imp.untyped cond   body' <- collect $ compileCode body   stm $ While cond' body'+compileCode (Imp.OnFailure body handler) = do+  body' <- collect $ compileCode body+  handler' <- collect $ compileCode handler+  stm $ Try body' [Catch (Var "Exception") $ handler' ++ [Raise $ Var "e"]] compileCode (Imp.For i bound body) = do   bound' <- compileExp bound   let i' = compileName i
src/Futhark/CodeGen/Backends/GenericWASM.hs view
@@ -1,126 +1,22 @@ {-# LANGUAGE QuasiQuotes #-} +-- | Code generation shared by the WASM backends.  The JavaScript API is+-- in @rts/javascript/wrappers.js@, driven by the manifest. module Futhark.CodeGen.Backends.GenericWASM-  ( GC.CParts (..),-    GC.asLibrary,-    GC.asExecutable,-    GC.asServer,-    EntryPointType,-    JSEntryPoint (..),-    JSRecordField (..),-    JSOpaqueType (..),-    emccExportNames,-    manifestFunctions,+  ( emccExportNames,     javascriptWrapper,-    extToString,-    opaqueToJS,     runServer,     libraryExports,   ) where -import Data.List (intercalate) import Data.Map qualified as M import Data.Text qualified as T-import Futhark.CodeGen.Backends.GenericC qualified as GC-import Futhark.CodeGen.Backends.SimpleRep (isValidCName, opaqueName)-import Futhark.CodeGen.ImpCode.Sequential qualified as Imp import Futhark.CodeGen.RTS.JavaScript import Futhark.Manifest-import Futhark.Util (nubOrd, showText, zEncodeText)-import Language.Futhark.Core (nameToText)-import Language.Futhark.Primitive+import Futhark.Util (nubOrd) import NeatInterpolation (text) -extToString :: Imp.ExternalValue -> String-extToString (Imp.TransparentValue (Imp.ArrayValue vn _ pt s dimSize)) =-  concat (replicate (length dimSize) "[]") ++ extToString (Imp.TransparentValue (Imp.ScalarValue pt s vn))-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float16) _ _)) = "f16"-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float32) _ _)) = "f32"-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float64) _ _)) = "f64"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Signed _)) = "i8"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Signed _)) = "i16"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Signed _)) = "i32"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Signed _)) = "i64"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Unsigned _)) = "u8"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Unsigned _)) = "u16"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Unsigned _)) = "u32"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Unsigned _)) = "u64"-extToString (Imp.TransparentValue (Imp.ScalarValue Bool _ _)) = "bool"-extToString (Imp.TransparentValue (Imp.ScalarValue Unit _ _)) = error "extToString: Unit"-extToString (Imp.OpaqueValue oname _) = T.unpack $ opaqueName oname--type EntryPointType = String--data JSEntryPoint = JSEntryPoint-  { name :: String,-    parameters :: [EntryPointType],-    ret :: EntryPointType-  }---- | A field in a JavaScript record opaque type.-data JSRecordField = JSRecordField-  { -- | Original field name (used in the server protocol).-    jsrfName :: String,-    -- | JavaScript type string (e.g. @"i32"@, @"[]f64"@, @"opaque_foo"@).-    jsrfType :: EntryPointType,-    -- | JavaScript method name on @FutharkContext@ (e.g. @"project_opaque_foo_x"@).-    jsrfProjectFn :: String-  }---- | How an opaque type is represented for the JavaScript server.-data JSOpaqueType-  = -- | A record type with named, projectable fields.-    JSOpaqueRecord [JSRecordField]-  | -- | Any other opaque type (no project support).-    JSOpaqueOther---- | Convert the opaque-type table from an ImpCode program into the--- JavaScript-oriented representation used by the code generator.-opaqueToJS :: Imp.OpaqueTypes -> [(String, JSOpaqueType)]-opaqueToJS (Imp.OpaqueTypes types) = map convertOne types-  where-    convertOne (desc, (Imp.OpaqueRecord fields, _)) =-      (T.unpack (opaqueName desc), JSOpaqueRecord (map (convertField desc) fields))-    convertOne (desc, _) =-      (T.unpack (opaqueName desc), JSOpaqueOther)--    convertField desc (fname, etype) =-      JSRecordField-        { jsrfName = T.unpack (nameToText fname),-          jsrfType = entryTypeToJSString etype,-          jsrfProjectFn =-            "project_"-              ++ T.unpack (opaqueName desc)-              ++ "_"-              ++ T.unpack f'-        }-      where-        f'-          | isValidCName (opaqueName desc <> "_" <> nameToText fname) =-              nameToText fname-          | otherwise = zEncodeText (nameToText fname)---- | Convert an 'Imp.EntryPointType' to its JavaScript string representation.-entryTypeToJSString :: Imp.EntryPointType -> String-entryTypeToJSString (Imp.TypeOpaque desc) = T.unpack $ opaqueName desc-entryTypeToJSString (Imp.TypeTransparent (Imp.ValueType sign (Imp.Rank rank) pt)) =-  concat (replicate rank "[]") ++ primToJSString sign pt-  where-    primToJSString _ (FloatType Float16) = "f16"-    primToJSString _ (FloatType Float32) = "f32"-    primToJSString _ (FloatType Float64) = "f64"-    primToJSString Imp.Signed (IntType Int8) = "i8"-    primToJSString Imp.Signed (IntType Int16) = "i16"-    primToJSString Imp.Signed (IntType Int32) = "i32"-    primToJSString Imp.Signed (IntType Int64) = "i64"-    primToJSString Imp.Unsigned (IntType Int8) = "u8"-    primToJSString Imp.Unsigned (IntType Int16) = "u16"-    primToJSString Imp.Unsigned (IntType Int32) = "u32"-    primToJSString Imp.Unsigned (IntType Int64) = "u64"-    primToJSString _ Bool = "bool"-    primToJSString _ Unit = error "entryTypeToJSString: Unit"- -- | The names of the C functions that must be exported from the WebAssembly -- module. This is every function named in the manifest, as the server calls -- these directly, as well as the context management functions.@@ -129,6 +25,13 @@   map (\f -> "'_" ++ T.unpack f ++ "'") (nubOrd $ manifestFunctions manifest)     ++ [ "_futhark_context_config_new",          "_futhark_context_config_free",+         "_futhark_context_config_set_debugging",+         "_futhark_context_config_set_logging",+         "_futhark_context_config_set_profiling",+         "_futhark_context_config_set_tuning_param",+         "_futhark_get_tuning_param_count",+         "_futhark_get_tuning_param_name",+         "_futhark_get_tuning_param_class",          "_futhark_context_new",          "_futhark_context_free",          "_futhark_context_get_error",@@ -180,460 +83,29 @@      variantFunctions v = [sumVariantConstruct v, sumVariantDestruct v] -javascriptWrapper :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-javascriptWrapper entryPoints opaqueTypes =+-- | The JavaScript code passed to @emcc@ with @--extern-post-js@.+javascriptWrapper :: Manifest -> T.Text+javascriptWrapper manifest =   T.unlines     [ serverJs,       valuesJs,-      wrapperclassesJs,-      classFutharkContext entryPoints opaqueTypes+      wrappersJs,+      "const futhark_manifest = " <> manifestToJSON manifest <> ";"     ] --- Make FutharkModule the generated primary class, but keeps FutharkContext around for backwards compatibility.-classFutharkContext :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-classFutharkContext entryPoints opaqueTypes =-  T.unlines-    [ "class FutharkModule {",-      moduleConstructor,-      moduleInitFromWasm entryPoints opaqueTypes,-      moduleInit,-      getFreeFun,-      getEntryPointsFun,-      getTypesFun,-      getErrorFun,-      getUtilityFuns,-      T.unlines $ map toFutharkArray arrays,-      T.unlines $ concatMap (generateProjectMethods . snd) opaqueTypes,-      T.unlines $ map jsWrapEntryPoint entryPoints,-      "}",-      classFutharkContextCompat,-      [text|-      async function newFutharkContext(num_threads) {-        var wasm = await loadWASM();-        return new FutharkContext(wasm, num_threads);-      }-      |]-    ]-  where-    -- Collect array types from entry points AND from record fields so the-    -- array constructors are always available when returning projected values.-    arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)-    entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) entryPoints-    recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]--moduleConstructor :: T.Text-moduleConstructor =-  [text|-  constructor() {-    this.wasm = undefined;-    this.cfg = undefined;-    this.ctx = undefined;-    this.entry_points = {};-    this.types = {};-    this.entry = {};-  }-  |]--moduleInitFromWasm :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-moduleInitFromWasm jses opaqueTypes =-  [text|-  _init_from_wasm(wasm, num_threads) {-    this.wasm = wasm;-    this.cfg = this.wasm._futhark_context_config_new();-    if (num_threads) this.wasm._futhark_context_config_set_num_threads(this.cfg, num_threads);-    this.ctx = this.wasm._futhark_context_new(this.cfg);--    this.entry_points = {-      ${entries}-    };-    this.types = {-      ${type_entries}-    };-    this.entry = {};-    ${entry_aliases}-    ${array_aliases}-    ${array_type_aliases}-  }-  |]-  where-    entries = T.intercalate "," $ map dicEntry jses-    type_entries = T.intercalate "," $ map dicTypeEntry opaqueTypes-    entry_aliases = T.unlines $ map entryAlias jses-    array_aliases = T.unlines $ map arrayAlias arrays-    array_type_aliases = T.unlines $ map arrayTypeAlias arrays--    arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)-    entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) jses-    recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]--moduleInit :: T.Text-moduleInit =-  [text|-  async init(wasm, num_threads) {-    if (wasm === undefined) {-      throw new Error("FutharkModule.init() requires the generated backend runtime module");-    }--    this._init_from_wasm(wasm, num_threads);-  }-  |]--classFutharkContextCompat :: T.Text-classFutharkContextCompat =-  [text|-  class FutharkContext extends FutharkModule {-    constructor(wasm, num_threads) {-      super();--      if (wasm !== undefined) {-        this._init_from_wasm(wasm, num_threads);-      }-    }-  }-  |]--getFreeFun :: T.Text-getFreeFun =-  [text|-  free() {-    if (this.ctx !== undefined) {-      this.wasm._futhark_context_free(this.ctx);-      this.ctx = undefined;-    }--    if (this.cfg !== undefined) {-      this.wasm._futhark_context_config_free(this.cfg);-      this.cfg = undefined;-    }-  }-  |]--getEntryPointsFun :: T.Text-getEntryPointsFun =-  [text|-  get_entry_points() {-    return this.entry_points;-  }-  |]--getTypesFun :: T.Text-getTypesFun =-  [text|-  get_types() {-    return this.types;-  }-  |]--getErrorFun :: T.Text-getErrorFun =-  [text|-  get_error() {-    var ptr = this.wasm._futhark_context_get_error(this.ctx);-    var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);-    var str = String.fromCharCode(...this.wasm.HEAP8.subarray(ptr, ptr + len));-    this.wasm._free(ptr);-    return str;-  }-  |]--getUtilityFuns :: T.Text-getUtilityFuns =-  [text|-  async context_sync() {-    return this.wasm._futhark_context_sync(this.ctx);-  }--  async clear_caches() {-    return this.wasm._futhark_context_clear_caches(this.ctx);-  }--  async report() {-    var ptr = this.wasm._futhark_context_report(this.ctx);-    var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);-    var bytes = this.wasm.HEAPU8.subarray(ptr, ptr + len);-    var str = new TextDecoder().decode(bytes);-    this.wasm._free(ptr);-    return str;-  }--  async pause_profiling() {-    return this.wasm._futhark_context_pause_profiling(this.ctx);-  }--  async unpause_profiling() {-    return this.wasm._futhark_context_unpause_profiling(this.ctx);-  }-  |]--entryAlias :: JSEntryPoint -> T.Text-entryAlias jse =-  [text|this.entry["${ename}"] = this.${fname}.bind(this);|]-  where-    fname = GC.escapeName $ T.pack $ name jse-    ename = T.pack $ name jse--arrayAlias :: String -> T.Text-arrayAlias typ =-  [text|-  this.${signature} = {-    from_data: (data, ${dims}) => this.new_${signature}(data, ${dims}),-    from_jsarray: (data) => this.new_${signature}_from_jsarray(data)-  };-  |]-  where-    d = dim typ-    ftype = baseType typ-    signature = T.pack $ ftype ++ "_" ++ show d ++ "d"-    dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]--arrayTypeAlias :: String -> T.Text-arrayTypeAlias typ =-  [text|this.types["${typ_text}"] = this.${signature};|]-  where-    d = dim typ-    ftype = baseType typ-    signature = T.pack $ ftype ++ "_" ++ show d ++ "d"-    typ_text = T.pack typ--dicEntry :: JSEntryPoint -> T.Text-dicEntry jse =-  [text|-       "${ename}" : ["${fname}", ${params}, ${rets}]-  |]-  where-    fname = GC.escapeName $ T.pack $ name jse-    ename = T.pack $ name jse-    params = showText $ parameters jse-    rets = showText $ ret jse---- | Generate the @types@ dictionary entry for a single opaque type.-dicTypeEntry :: (String, JSOpaqueType) -> T.Text-dicTypeEntry (tname, JSOpaqueRecord fields) =-  T.pack $-    show tname-      ++ ": [\"record\", ["-      ++ intercalate ", " (map fieldEntry fields)-      ++ "]]"-  where-    fieldEntry rf =-      "["-        ++ show (jsrfName rf)-        ++ ", "-        ++ show (jsrfType rf)-        ++ ", \""-        ++ jsrfProjectFn rf-        ++ "\"]"-dicTypeEntry (tname, JSOpaqueOther) =-  T.pack $ show tname ++ ": [\"opaque\"]"---- | Generate @project_*@ wrapper methods for a record opaque type.-generateProjectMethods :: JSOpaqueType -> [T.Text]-generateProjectMethods (JSOpaqueRecord fields) = map generateProjectMethod fields-generateProjectMethods JSOpaqueOther = []--generateProjectMethod :: JSRecordField -> T.Text-generateProjectMethod rf =-  T.pack $-    unlines-      [ "  " ++ jsrfProjectFn rf ++ "(obj) {",-        "    var out = this.wasm._malloc(" ++ show (typeSize ftype) ++ ");",-        "    this.wasm._futhark_" ++ jsrfProjectFn rf ++ "(this.ctx, out, obj.ptr);",-        "    var result = " ++ readResult ++ ";",-        "    this.wasm._free(out);",-        "    return result;",-        "  }"-      ]-  where-    ftype = jsrfType rf-    readout = typeHeap ftype ++ "[out >> " ++ show (typeShift ftype) ++ "]"-    readResult-      | isArray ftype =-          "this.new_" ++ baseType ftype ++ "_" ++ show (dim ftype) ++ "d_from_ptr(" ++ readout ++ ")"-      | isOpaque ftype =-          "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ ftype ++ ")"-      | ftype == "bool" = readout ++ "!==0"-      | otherwise = readout--jsWrapEntryPoint :: JSEntryPoint -> T.Text-jsWrapEntryPoint jse =-  [text|-  ${func_name}(${inparams}) {-    var out = [${outparams}].map(n => this.wasm._malloc(n));-    var to_free = [];-    var do_free = () => { out.forEach(this.wasm._free); to_free.forEach(f => f.free()); };-    ${paramsToPtr}-    if (this.wasm._futhark_entry_${func_name}(this.ctx, ...out, ${ins}) > 0) {-      do_free();-      throw this.get_error();-    }-    ${results}-    do_free();-    return ${res};-  }-  |]-  where-    func_name = GC.escapeName $ T.pack $ name jse--    alp = [0 .. length (parameters jse) - 1]-    inparams = T.pack $ intercalate ", " ["in" ++ show i | i <- alp]-    ins = T.pack $ intercalate ", " [maybeDerefence ("in" ++ show i) $ parameters jse !! i | i <- alp]-    paramsToPtr = T.pack $ unlines $ filter ("" /=) [arrayPointer ("in" ++ show i) $ parameters jse !! i | i <- alp]--    outparams = showText $ typeSize $ ret jse-    results = T.pack $ makeResult 0 $ ret jse-    res = "result0"--maybeDerefence :: String -> String -> String-maybeDerefence arg typ =-  if isScalar typ then arg else arg ++ ".ptr"--arrayPointer :: String -> String -> String-arrayPointer arg typ =-  if isArray typ-    then "  if (" ++ arg ++ " instanceof Array) { " ++ reassign ++ "; to_free.push(" ++ arg ++ "); }"-    else ""-  where-    reassign = arg ++ " = this.new_" ++ signature ++ "_from_jsarray(" ++ arg ++ ")"-    signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"--makeResult :: Int -> String -> String-makeResult i typ =-  "  var result"-    ++ show i-    ++ " = "-    ++ if isArray typ-      then "this.new_" ++ signature ++ "_from_ptr(" ++ readout ++ ");"-      else-        if isOpaque typ-          then "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ typ ++ ");"-          else readout ++ if typ == "bool" then "!==0;" else ";"-  where-    res = "out[" ++ show i ++ "]"-    readout = typeHeap typ ++ "[" ++ res ++ " >> " ++ show (typeShift typ) ++ "]"-    signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"--baseType :: String -> String-baseType ('[' : ']' : end) = baseType end-baseType typ = typ--dim :: String -> Int-dim ('[' : ']' : end) = dim end + 1-dim _ = 0--isArray :: String -> Bool-isArray typ = take 2 typ == "[]"--isOpaque :: String -> Bool-isOpaque typ = take 6 typ == "opaque"--isScalar :: String -> Bool-isScalar typ = not (isArray typ || isOpaque typ)--typeSize :: String -> Integer-typeSize typ =-  case typ of-    "i8" -> 1-    "i16" -> 2-    "i32" -> 4-    "i64" -> 8-    "u8" -> 1-    "u16" -> 2-    "u32" -> 4-    "u64" -> 8-    "f16" -> 2-    "f32" -> 4-    "f64" -> 8-    "bool" -> 1-    _ -> 4--typeShift :: String -> Integer-typeShift typ =-  case typ of-    "i8" -> 0-    "i16" -> 1-    "i32" -> 2-    "i64" -> 3-    "u8" -> 0-    "u16" -> 1-    "u32" -> 2-    "u64" -> 3-    "f16" -> 1-    "f32" -> 2-    "f64" -> 3-    "bool" -> 0-    _ -> 2--typeHeap :: String -> String-typeHeap typ =-  case typ of-    "i8" -> "this.wasm.HEAP8"-    "i16" -> "this.wasm.HEAP16"-    "i32" -> "this.wasm.HEAP32"-    "i64" -> "this.wasm.HEAP64"-    "u8" -> "this.wasm.HEAPU8"-    "u16" -> "this.wasm.HEAPU16"-    "u32" -> "this.wasm.HEAPU32"-    "u64" -> "(new BigUint64Array(this.wasm.HEAP64.buffer))"-    "f16" -> "this.wasm.HEAPU16"-    "f32" -> "this.wasm.HEAPF32"-    "f64" -> "this.wasm.HEAPF64"-    "bool" -> "this.wasm.HEAP8"-    _ -> "this.wasm.HEAP32"--toFutharkArray :: String -> T.Text-toFutharkArray typ =-  [text|-  ${new}_from_jsarray(${arraynd_p}) {-    return this.${new}(${arraynd_flat_p}, ${arraynd_dims_p});-  }-  ${new}(array, ${dims}) {-    console.assert(array.length === ${dims_multiplied}, 'len=%s,dims=%s', array.length, [${dims}].toString());-      var copy = this.wasm._malloc(array.length << ${shift});-      ${heapType}.set(array, copy >> ${shift});-      var ptr = ${fnew}(this.ctx, copy, ${bigint_dims});-      this.wasm._free(copy);-      return this.${new}_from_ptr(ptr);-    }--    ${new}_from_ptr(ptr) {-      return new FutharkArray(this, ptr, ${args});-    }-    |]-  where-    d = dim typ-    ftype = baseType typ-    heap = typeHeap ftype-    signature = ftype ++ "_" ++ show d ++ "d"-    new = T.pack $ "new_" ++ signature-    fnew = T.pack $ "this.wasm._futhark_new_" ++ signature-    fshape = "this.wasm._futhark_shape_" ++ signature-    fvalues = "this.wasm._futhark_values_raw_" ++ signature-    ffree = "this.wasm._futhark_free_" ++ signature-    arraynd = "array" ++ show d ++ "d"-    shift = showText (typeShift ftype)-    heapType = T.pack heap-    arraynd_flat = if d > 1 then arraynd ++ ".flat()" else arraynd-    arraynd_dims = intercalate ", " [arraynd ++ mult i "[0]" ++ ".length" | i <- [0 .. d - 1]]-    dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]-    dims_multiplied = T.pack $ intercalate "*" ["Number(d" ++ show i ++ ")" | i <- [0 .. d - 1]]-    bigint_dims = T.pack $ intercalate ", " ["BigInt(d" ++ show i ++ ")" | i <- [0 .. d - 1]]-    mult i s = concat $ replicate i s-    (arraynd_p, arraynd_flat_p, arraynd_dims_p) = (T.pack arraynd, T.pack arraynd_flat, T.pack arraynd_dims)-    args = T.pack $ intercalate ", " ["'" ++ ftype ++ "'", show d, heap, fshape, fvalues, ffree]- -- | Javascript code that can be appended to the generated module to--- run a Futhark server instance on startup.-runServer :: Manifest -> T.Text-runServer manifest =+-- run a Futhark server instance when it is loaded.+runServer :: T.Text+runServer =   [text|-   Module.onRuntimeInitialized = () => {-     var context = new FutharkContext(Module);-     var server = new Server(context, ${manifest_json});-     server.run();+   // Worker threads (wasm-multicore) also load this module.+   const is_main_thread = globalThis.process?.versions?.node+     ? (await import("node:worker_threads")).isMainThread+     : globalThis.WorkerGlobalScope === undefined;+   if (is_main_thread) {+     await futhark_server_main();    }|]-  where-    manifest_json = manifestToJSON manifest  -- | The names exported by the generated module. libraryExports :: T.Text-libraryExports = "export {newFutharkContext, FutharkContext, FutharkModule, FutharkArray, FutharkOpaque};"+libraryExports = "export {FutharkContext, FutharkArray, FutharkOpaque};"
src/Futhark/CodeGen/Backends/MulticoreC.hs view
@@ -110,8 +110,8 @@ compileRetvalStructFields :: [VName] -> [(C.Type, C.Type, ValueType)] -> [C.FieldGroup] compileRetvalStructFields = zipWith field   where-    field name (ty, _, Prim _) =-      [C.csdecl|$ty:ty *$id:(closureRetvalStructField name);|]+    field name (_, storage_ty, Prim _) =+      [C.csdecl|$ty:storage_ty *$id:(closureRetvalStructField name);|]     field name (ty, _, ScalarMem) =       [C.csdecl|$ty:ty $id:(closureRetvalStructField name);|]     field name (_, _, _) =@@ -134,32 +134,55 @@     field name (_, _, ScalarMem) =       [C.cstm|$id:struct.$id:(closureFreeStructField name)=$id:name;|] +-- | Point the retval fields of a task struct at the variables that will receive+-- the results, and return the statements that must be executed after the task+-- has run. A scalar retval field points to the storage representation of its+-- type (see 'compileRetvalStructFields'). When that differs from the+-- representation of the variable itself, the field instead points to a+-- temporary in storage representation, which is converted back after the task+-- has run. This matters for ISPC, where the variable lives in ISPC code but is+-- written by C code, and the two may not agree on the representation of+-- 'Float16'. compileSetRetvalStructValues ::   (C.ToIdent a) =>   a ->   [VName] ->   [(C.Type, C.Type, ValueType)] ->-  [C.Stm]-compileSetRetvalStructValues struct vnames we = concat $ zipWith field vnames we+  GC.CompilerM op s [C.Stm]+compileSetRetvalStructValues struct vnames we = concat <$> zipWithM field vnames we   where-    field name (ct, _, Prim _) =-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ct*)&$id:name);-               $escstm:("#if defined(ISPC)")-               $id:struct.$id:(closureRetvalStructField name)+= programIndex;-               $escstm:("#endif")|]-    field name (_, _, MemBlock) =-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]-    field name (_, _, RawMem) =-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]-    field name (_, _, ScalarMem) =-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+    field name (ct, storage_ty, Prim pt)+      | ct == storage_ty = do+          point name storage_ty+          pure []+      | otherwise = do+          tmp <- newVName $ baseName name <> "_storage"+          GC.decl [C.cdecl|$ty:storage_ty $id:tmp = $exp:(toStorage pt (C.toExp name noLoc));|]+          point tmp storage_ty+          pure [[C.cstm|$id:name = $exp:(fromStorage pt (C.toExp tmp noLoc));|]]+      where+        point v ty =+          GC.stms+            [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ty*)&$id:v);+                     $escstm:("#if defined(ISPC)")+                     $id:struct.$id:(closureRetvalStructField name)+= programIndex;+                     $escstm:("#endif")|]+    field name (_, _, MemBlock) = do+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]+      pure []+    field name (_, _, RawMem) = do+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+      pure []+    field name (_, _, ScalarMem) = do+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+      pure []  compileGetRetvalStructVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.InitGroup] compileGetRetvalStructVals struct = zipWith field   where-    field name (ty, _storage_ty, Prim _) =+    field name (ty, _storage_ty, Prim pt) =       let inner = [C.cexp|*$id:struct->$id:(closureRetvalStructField name)|]-       in [C.cdecl|$ty:ty $id:name = $exp:inner;|]+       in [C.cdecl|$ty:ty $id:name = $exp:(fromStorage pt inner);|]     field name (ty, _, ScalarMem) =       [C.cdecl|$ty:ty $id:name = $id:struct->$id:(closureRetvalStructField name);|]     field name (ty, _, _) =@@ -190,8 +213,8 @@ compileWriteBackResVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.Stm] compileWriteBackResVals struct = zipWith field   where-    field name (_, _, Prim _) =-      [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(C.toExp name noLoc);|]+    field name (_, _, Prim pt) =+      [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(toStorage pt (C.toExp name noLoc));|]     field name (_, _, ScalarMem) =       [C.cstm|$id:struct->$id:(closureRetvalStructField name) = $id:name;|]     field name (_, _, _) =@@ -286,6 +309,7 @@         mapM_ GC.item =<< GC.declAllocatedMem         mapM_ GC.item code'         free_mem <- GC.freeAllocatedMem+        mapM_ GC.item =<< GC.errorLabel         GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|]     pure       [C.cedecl|int $id:s(void *args, typename int64_t iterations, int tid, struct scheduler_info info) {@@ -307,7 +331,7 @@   [(C.Type, C.Type, ValueType)] ->   [VName] ->   [(C.Type, C.Type, ValueType)] ->-  GC.CompilerM Multicore s Name+  GC.CompilerM Multicore s (Name, [C.Stm]) prepareTaskStruct def name free_args free_ctypes retval_args retval_ctypes = do   let makeStruct s =         pure@@ -321,8 +345,8 @@   GC.decl [C.cdecl|struct $id:fstruct $id:fstruct';|]   GC.stm [C.cstm|$id:fstruct'.ctx = ctx;|]   GC.stms [C.cstms|$stms:(compileSetStructValues fstruct' free_args free_ctypes)|]-  GC.stms [C.cstms|$stms:(compileSetRetvalStructValues fstruct' retval_args retval_ctypes)|]-  pure fstruct+  after <- compileSetRetvalStructValues fstruct' retval_args retval_ctypes+  pure (fstruct, after)  -- Generate a segop function for top_level and potentially nested SegOp code compileOp :: GC.OpCompiler Multicore s@@ -346,7 +370,7 @@    let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] params mempty seq_code -  fstruct <-+  (fstruct, read_retvals) <-     prepareTaskStruct multicoreDef "task" free_args free_ctypes retval_args retval_ctypes    fpar_task <- generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval@@ -375,22 +399,24 @@     Nothing ->       GC.stm [C.cstm|$id:ftask_name.nested_fn=NULL;|] +  failure <- GC.failureJump   let ftask_err = fpar_task <> "_err"       code =         [C.citems|int $id:ftask_err = scheduler_prepare_task(&ctx->scheduler, &$id:ftask_name);                   if ($id:ftask_err != 0) {                     err = $id:ftask_err;-                    goto cleanup;+                    $stm:failure                   }|]    mapM_ GC.item code+  GC.stms read_retvals compileOp (ParLoop s' body free) = do   free_ctypes <- mapM paramToCType free   let free_args = map paramName free    let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] free mempty body -  fstruct <-+  (fstruct, _) <-     prepareTaskStruct multicoreDef (s' <> "_parloop_struct") free_args free_ctypes mempty mempty    ftask <- multicoreDef (s' <> "_parloop") $ \s -> do@@ -406,6 +432,7 @@         mapM_ GC.item =<< GC.declAllocatedMem         free_mem <- GC.freeAllocatedMem         mapM_ GC.item body'+        mapM_ GC.item =<< GC.errorLabel         GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|]     pure       [C.cedecl|static int $id:s(void *args,@@ -432,6 +459,7 @@    let ftask_err = ftask <> "_err"       ftask_total = ftask <> "_total"+  failure <- GC.failureJump   code' <-     benchmarkCode       ftask_total@@ -439,7 +467,7 @@                                                            &$id:ftask_name);                if ($id:ftask_err != 0) {                  err = $id:ftask_err;-                 goto cleanup;+                 $stm:failure                }|]    mapM_ GC.item code'
src/Futhark/CodeGen/Backends/MulticoreISPC.hs view
@@ -707,6 +707,9 @@   GC.stm [C.cstm|if (!$exp:e') { $items:err }|]   where     stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs+compileCode OnFailure {} =+  -- ISPC does not allow the goto.+  error "compileCode: OnFailure in ISPC kernel." compileCode code =   GC.compileCode code @@ -790,6 +793,18 @@                      $id:name.size = 0;                      $id:name.references = NULL;|] +-- | Does this code contain an 'OnFailure', not counting those in the tasks of+-- nested 'SegOp's?+hasOnFailure :: MCCode -> Bool+hasOnFailure OnFailure {} = True+hasOnFailure (x :>>: y) = hasOnFailure x || hasOnFailure y+hasOnFailure (If _ x y) = hasOnFailure x || hasOnFailure y+hasOnFailure (For _ _ x) = hasOnFailure x+hasOnFailure (While _ x) = hasOnFailure x+hasOnFailure (Op (ForEach _ _ _ body)) = hasOnFailure body+hasOnFailure (Op (ForEachActive _ body)) = hasOnFailure body+hasOnFailure _ = False+ -- | Can the given code produce an error? If so, we can't use foreach -- loops, since they don't allow for early-outs in error handling. mayProduceError :: MCCode -> Bool@@ -800,6 +815,7 @@ mayProduceError (Op (ForEachActive _ body)) = mayProduceError body mayProduceError (Op (ForEach _ _ _ body)) = mayProduceError body mayProduceError (Op SegOp {}) = True+mayProduceError (OnFailure x y) = mayProduceError x || mayProduceError y mayProduceError Allocate {} = True mayProduceError Assert {} = True mayProduceError SetMem {} = True@@ -822,7 +838,7 @@    let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] params mempty seq_code -  fstruct <-+  (fstruct, read_retvals) <-     MC.prepareTaskStruct sharedDef "task" free_args free_ctypes retval_args retval_ctypes    fpar_task <- MC.generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval@@ -874,6 +890,7 @@                         struct $id:fstruct $tyqual:uniform * $tyqual:uniform args,                         $tyqual:uniform int iterations);|] +  failure <- GC.failureJump   aos_name <- newVName "aos"   GC.items     [C.citems|@@ -892,16 +909,23 @@     $escstm:("#else")     err = $id:schedn(ctx, &$id:(fstruct <> "_"), $exp:e');     if (err != 0) {-      goto cleanup;+      $stm:failure     }     $escstm:("#endif")|]+  GC.stms read_retvals+compileOp (ISPCKernel body free)+  | -- ISPC is too weak for our failure handling machinery.+    hasOnFailure body =+      local (\env -> env {GC.envOperations = MC.operations}) $+        MC.compileOp $+          ISPCKernel body free compileOp (ISPCKernel body free) = do   free_ctypes <- mapM MC.paramToCType free   let free_args = map paramName free    let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] free mempty body   -- Generate ISPC kernel-  fstruct <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] []+  (fstruct, _) <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] []   let fstruct' = fstruct <> "_"    ispcShim <- ispcDef "loop_ispc" $ \s -> do@@ -911,7 +935,9 @@           GC.collect $ do             GC.decl [C.cdecl|$tyqual:uniform struct futhark_context * $tyqual:uniform ctx = $id:fstruct'->ctx;|]             GC.items =<< compileGetStructVals fstruct free_args free_ctypes-            body' <- GC.collect $ compileCode body+            body' <- GC.collect $ do+              compileCode body+              GC.items =<< GC.errorLabel             mapM_ GC.item decl_cached             mapM_ GC.item =<< GC.declAllocatedMem @@ -962,11 +988,12 @@         }|]    -- Generate C code to call into ISPC kernel+  failure <- GC.failureJump   GC.items     [C.citems|     err = $id:ispcShim(start, end, & $id:fstruct');     if (err != 0) {-      goto cleanup;+      $stm:failure     }|] compileOp (ForEach i from bound body) = do   from' <- compileExp from@@ -1089,18 +1116,13 @@     free = freeIn bound findDeps (While cond x) = do   local (<> freeIn cond) $ findDeps x-findDeps (Op (SegOp _ free _ _ retvals _)) =-  mapM_-    ( \x ->-        addDeps (paramName x) $-          namesFromList $-            map paramName free-    )-    retvals findDeps (Op (ForEach _ _ _ body)) =   findDeps body findDeps (Op (ForEachActive _ body)) =   findDeps body+findDeps (OnFailure body handler) = do+  findDeps body+  findDeps handler findDeps (SetScalar name e) =   addDeps name $ freeIn e findDeps (Call tars _ args) =@@ -1130,15 +1152,20 @@     deps' = M.map grow deps  -- | Find roots of variance. These are memory blocks declared in--- the current scope as well as loop indices of foreach loops.+-- the current scope, loop indices of foreach loops, and the results of+-- nested SegOps. A SegOp in a kernel is run once per program instance,+-- each of which receives its result through a pointer to its own lane+-- of the variable (see 'MC.compileSetRetvalStructValues'). findVarying :: MCCode -> [VName] findVarying (x :>>: y) = findVarying x ++ findVarying y findVarying (If _ x y) = findVarying x ++ findVarying y findVarying (For _ _ x) = findVarying x findVarying (While _ x) = findVarying x findVarying (Op (ForEachActive _ body)) = findVarying body+findVarying (OnFailure body handler) = findVarying body ++ findVarying handler findVarying (Op (ForEach idx _ _ body)) = idx : findVarying body findVarying (DeclareMem mem _) = [mem]+findVarying (Op (SegOp _ _ _ _ retvals _)) = map paramName retvals findVarying _ = []  -- | Analyze variability in a body of code and run an action with@@ -1147,7 +1174,7 @@ analyzeVariability code m = do   let roots = findVarying code   let deps = depsFixedPoint $ execVariabilityM $ findDeps code-  let safelist = M.filter (\b -> all (`notNameIn` b) roots) deps+  let safelist = M.filterWithKey (\v b -> v `notElem` roots && all (`notNameIn` b) roots) deps   let safe = namesFromList $ M.keys safelist   pre_state <- GC.getUserState   GC.modifyUserState (\s -> s {sUniform = safe})
src/Futhark/CodeGen/Backends/MulticoreWASM.hs view
@@ -4,22 +4,14 @@ -- JavaScript wrapper. module Futhark.CodeGen.Backends.MulticoreWASM   ( compileProg,-    runServer,-    libraryExports,-    GC.CParts (..),-    GC.asLibrary,-    GC.asExecutable,-    GC.asServer,   ) where -import Data.Maybe import Data.Text qualified as T import Futhark.CodeGen.Backends.GenericC qualified as GC import Futhark.CodeGen.Backends.GenericWASM import Futhark.CodeGen.Backends.MulticoreC qualified as MC import Futhark.CodeGen.Backends.MulticoreC.Boilerplate (generateBoilerplate)-import Futhark.CodeGen.ImpCode.Multicore qualified as Imp import Futhark.CodeGen.ImpGen.Multicore qualified as ImpGen import Futhark.IR.MCMem import Futhark.MonadFreshNames@@ -32,10 +24,11 @@ -- * Generated C code (to be passed to Emscripten). -- -- * JavaScript wrapper code that presents a nicer interface to the---   Emscripten-produced code (this should be put in a @.class.js@---   file by itself).+--   Emscripten-produced code (passed to @emcc@ with+--   @--extern-post-js@). ----- * Options that should be passed to @emcc@.+-- * The names of the C functions that must be exported from the+--   WebAssembly module. compileProg ::   (MonadFreshNames m) =>   T.Text ->@@ -58,20 +51,7 @@   pure     ( ws,       ( prog'',-        javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')),+        javascriptWrapper (GC.cManifest prog''),         "_futhark_context_config_set_num_threads" : emccExportNames (GC.cManifest prog'')       )     )--fRepMyRep :: Imp.Definitions Imp.Multicore -> [JSEntryPoint]-fRepMyRep prog =-  let Imp.Functions fs = Imp.defFuns prog-      function fun = do-        Imp.EntryPoint n res args _ <- Imp.functionEntry fun-        Just $-          JSEntryPoint-            { name = nameToString n,-              parameters = map (extToString . snd) args,-              ret = extToString $ snd res-            }-   in mapMaybe (function . snd) fs
src/Futhark/CodeGen/Backends/PyOpenCL.hs view
@@ -233,6 +233,11 @@ callKernel (Imp.GetSizeMax v size_class) = do   v' <- compileVar v   stm $ Assign v' $ kernelConstToExp $ Imp.SizeMaxConst size_class+callKernel (Imp.GetBookkeeping v b) = do+  v' <- compileVar v+  stm . Assign v' . Var $ case b of+    Imp.BookkeepingCounters -> "self.bookkeeping_counters"+    Imp.BookkeepingLocks -> "self.bookkeeping_locks" callKernel (Imp.LaunchKernel safety name shared_memory args num_threadblocks workgroup_size) = do   num_threadblocks' <- mapM (fmap asLong . compileExp) num_threadblocks   workgroup_size' <- mapM compileBlockDim workgroup_size
src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs view
@@ -11,13 +11,15 @@ import Futhark.CodeGen.Backends.GenericPython qualified as Py import Futhark.CodeGen.Backends.GenericPython.AST import Futhark.CodeGen.ImpCode.OpenCL-  ( ErrorMsg (..),+  ( Bookkeeping (..),+    ErrorMsg (..),     ErrorMsgPart (..),     FailureMsg (..),     KernelConst (..),     KernelConstExp,     ParamMap,     PrimType (..),+    bookkeepingSize,     errorMsgArgTypes,     untyped,   )@@ -57,6 +59,8 @@   [text| size_heuristics=$size_heuristics self.global_failure_args_max = $max_num_args+self.bookkeeping_counters_size = $num_counters+self.bookkeeping_locks_size = $num_locks self.failure_msgs=$failure_msgs constants = $constants' program = initialise_opencl_object(self,@@ -84,6 +88,8 @@     types' = prettyText $ map (show . prettyString) types -- Looks enough like Python.     sizes' = prettyText $ Py.sizeClassesToPython sizes     max_num_args = prettyText $ foldl max 0 $ map (errorMsgNumArgs . failureError) failures+    num_counters = prettyText $ bookkeepingSize BookkeepingCounters+    num_locks = prettyText $ bookkeepingSize BookkeepingLocks     failure_msgs = prettyText $ List $ map formatFailure failures     onConstant (name, e) =       Tuple
src/Futhark/CodeGen/Backends/SequentialWASM.hs view
@@ -4,16 +4,9 @@ -- JavaScript wrapper. module Futhark.CodeGen.Backends.SequentialWASM   ( compileProg,-    runServer,-    libraryExports,-    GC.CParts (..),-    GC.asLibrary,-    GC.asExecutable,-    GC.asServer,   ) where -import Data.Maybe import Data.Text qualified as T import Futhark.CodeGen.Backends.GenericC qualified as GC import Futhark.CodeGen.Backends.GenericWASM@@ -31,10 +24,11 @@ -- * Generated C code (to be passed to Emscripten). -- -- * JavaScript wrapper code that presents a nicer interface to the---   Emscripten-produced code (this should be put in a @.class.js@---   file by itself).+--   Emscripten-produced code (passed to @emcc@ with+--   @--extern-post-js@). ----- * Options that should be passed to @emcc@.+-- * The names of the C functions that must be exported from the+--   WebAssembly module. compileProg :: (MonadFreshNames m) => T.Text -> Prog SeqMem -> m (ImpGen.Warnings, (GC.CParts, T.Text, [String])) compileProg version prog = do   (ws, prog') <- ImpGen.compileProg prog@@ -49,23 +43,10 @@       (DefaultSpace, [DefaultSpace])       []       prog'-  pure (ws, (prog'', javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')), emccExportNames (GC.cManifest prog'')))+  pure (ws, (prog'', javascriptWrapper (GC.cManifest prog''), emccExportNames (GC.cManifest prog'')))   where     operations :: GC.Operations Imp.Sequential ()     operations =       GC.defaultOperations         { GC.opsCompiler = const $ pure ()         }--fRepMyRep :: Imp.Program -> [JSEntryPoint]-fRepMyRep prog =-  let Imp.Functions fs = Imp.defFuns prog-      function (Imp.Function entry _ _ _ _) = do-        Imp.EntryPoint n res args _doc <- entry-        Just $-          JSEntryPoint-            { name = nameToString n,-              parameters = map (extToString . snd) args,-              ret = extToString $ snd res-            }-   in mapMaybe (function . snd) fs
src/Futhark/CodeGen/ImpCode.hs view
@@ -146,7 +146,7 @@ data Param   = MemParam VName Space   | ScalarParam VName PrimType-  deriving (Eq, Show)+  deriving (Eq, Ord, Show)  -- | The name of a parameter. paramName :: Param -> VName@@ -337,6 +337,12 @@     -- out not to be true, then report a failure along with     -- the given error message.     Assert Exp (ErrorMsg Exp) (Loc, [Loc])+  | -- | @OnFailure body handler@ executes @body@, and if it fails, executes+    -- @handler@ before the failure propagates as usual. This is for code that+    -- must still be executed after a failure, such as releasing a lock. The+    -- handler must not fail. In a GPU kernel, only the failing thread executes+    -- the handler, so @body@ must not contain barriers.+    OnFailure (Code a) (Code a)   | -- | Print the given value to the screen, somehow     -- annotated with the given string as a description.  If     -- no type/value pair, just print the string.  This has@@ -390,6 +396,7 @@     go f (If _ x y) = f x <> f y     go f (For _ _ x) = f x     go f (While _ x) = f x+    go f (OnFailure x y) = f x <> f y     go _ _ = mempty      declared (DeclareMem mem space) =@@ -415,6 +422,7 @@     listify (If cond c1 c2) = [If cond (declsFirst c1) (declsFirst c2)]     listify (For i e c) = [For i e (declsFirst c)]     listify (While cond c) = [While cond (declsFirst c)]+    listify (OnFailure c1 c2) = [OnFailure (declsFirst c1) (declsFirst c2)]     listify c = [c]     isDecl (DeclareScalar {}) = True     isDecl (DeclareMem {}) = True@@ -429,6 +437,7 @@ calledFuncs f (If _ x y) = calledFuncs f x <> calledFuncs f y calledFuncs f (For _ _ x) = calledFuncs f x calledFuncs f (While _ x) = calledFuncs f x+calledFuncs f (OnFailure x y) = calledFuncs f x <> calledFuncs f y calledFuncs _ _ = mempty  -- | Compute call graph, as per 'calledFuncs', but also include@@ -449,6 +458,7 @@ foldProvenances f (For _ _ c) = foldProvenances f c foldProvenances f (While _ c) = foldProvenances f c foldProvenances f (If _ x y) = foldProvenances f x <> foldProvenances f y+foldProvenances f (OnFailure x y) = foldProvenances f x <> foldProvenances f y foldProvenances f (Op x) = f x foldProvenances _ _ = mempty @@ -643,6 +653,13 @@     pretty dest <+> "<-" <+> pretty from <+> "@" <> pretty space   pretty (Assert e msg _) =     "assert" <> parens (commasep [pretty msg, pretty e])+  pretty (OnFailure body handler) =+    "on_failure"+      <+> "{"+      </> indent 2 (pretty body)+      </> "} handler {"+      </> indent 2 (pretty handler)+      </> "}"   pretty (Copy t shape (dst, dstspace) (dstoffset, dststrides) (src, srcspace) (srcoffset, srcstrides)) =     ("lmadcopy_" <> pretty (length shape) <> "d_" <> pretty t)       <> (parens . align)@@ -732,6 +749,8 @@     While cond <$> traverse f code   traverse f (If cond x y) =     If cond <$> traverse f x <*> traverse f y+  traverse f (OnFailure x y) =+    OnFailure <$> traverse f x <*> traverse f y   traverse f (Op kernel) =     Op <$> f kernel   traverse _ Skip =@@ -779,6 +798,7 @@ declaredIn (x :>>: y) = declaredIn x <> declaredIn y declaredIn (For i _ body) = oneName i <> declaredIn body declaredIn (While _ body) = declaredIn body+declaredIn (OnFailure body handler) = declaredIn body <> declaredIn handler declaredIn _ = mempty  instance FreeIn EntryPoint where@@ -837,6 +857,8 @@     freeIn' cond <> freeIn' t <> freeIn' f   freeIn' (Assert e msg _) =     freeIn' e <> foldMap freeIn' msg+  freeIn' (OnFailure body handler) =+    freeIn' body <> freeIn' handler   freeIn' (Op op) =     freeIn' op   freeIn' (Meta {}) =
src/Futhark/CodeGen/ImpCode/GPU.hs view
@@ -7,6 +7,8 @@     KernelConst (..),     KernelConstExp,     HostOp (..),+    Bookkeeping (..),+    bookkeepingSize,     KernelOp (..),     Fence (..),     AtomicOp (..),@@ -48,8 +50,29 @@   | GetSize VName Name SizeClass   | CmpSizeLe VName Name SizeClass Exp   | GetSizeMax VName SizeClass+  | -- | Bind the given device memory to one of the bookkeeping buffers of the+    -- context.+    GetBookkeeping VName Bookkeeping   deriving (Show) +-- | The bookkeeping buffers are device buffers of 32-bit integers that belong+-- to the context and are shared by all kernels. They are zero whenever no+-- kernel is running, and a kernel that uses them must leave them zeroed when it+-- succeeds, and the context zeroes them again after a failure. As kernels run+-- serially, they never compete for them. If we ever add support for multiple+-- streams, we must have a set of buffers per stream.+data Bookkeeping+  = -- | Counters, e.g. for detecting the last block to finish.+    BookkeepingCounters+  | -- | Locks for operations that cannot be done atomically.+    BookkeepingLocks+  deriving (Eq, Ord, Show)++-- | The number of 32-bit integers in a bookkeeping buffer.+bookkeepingSize :: Bookkeeping -> Integer+bookkeepingSize BookkeepingCounters = 1024+bookkeepingSize BookkeepingLocks = 100151+ -- | The size of one dimension of a block. type BlockDim = Either Exp KernelConstExp @@ -114,6 +137,10 @@       <> parens (commasep [pretty key, pretty size_class])   pretty (GetSizeMax dest size_class) =     pretty dest <+> "<-" <+> "get_size_max" <> parens (pretty size_class)+  pretty (GetBookkeeping dest BookkeepingCounters) =+    pretty dest <+> "<-" <+> "bookkeeping_counters()"+  pretty (GetBookkeeping dest BookkeepingLocks) =+    pretty dest <+> "<-" <+> "bookkeeping_locks()"   pretty (CmpSizeLe dest name size_class x) =     pretty dest       <+> "<-"@@ -133,6 +160,8 @@     freeIn' dest   freeIn' (GetSize dest _ _) =     freeIn' dest+  freeIn' (GetBookkeeping dest _) =+    freeIn' dest  instance FreeIn Kernel where   freeIn' kernel =@@ -181,13 +210,14 @@     -- warp. Only supported for CUDA and HIP.     WarpSync   | SharedAlloc VName (Count Bytes (TExp Int64))-  | -- | Perform a barrier and also check whether any-    -- threads have failed an assertion.  Make sure all-    -- threads would reach all 'ErrorSync's if any of them-    -- do.  A failing assertion will jump to the next-    -- following 'ErrorSync', so make sure it's not inside-    -- control flow or similar.-    ErrorSync Fence+  | -- | Perform a barrier and also check whether any threads have failed an+    -- assertion. If so, all threads of the block execute the given code and+    -- then stop. The code must not fail, and is usually empty. Make sure all+    -- threads would reach all 'ErrorSync's if any of them do. A failing+    -- assertion will jump to the next following 'ErrorSync' (unless it is in+    -- the body of an 'OnFailure'), so make sure it's not inside control flow or+    -- similar.+    ErrorSync Fence (Code KernelOp)   deriving (Show)  -- | Atomic operations return the value stored before the update. This@@ -257,10 +287,14 @@     "warp_sync()"   pretty (SharedAlloc name size) =     pretty name <+> equals <+> "shared_alloc" <> parens (pretty size)-  pretty (ErrorSync FenceLocal) =-    "error_sync_local()"-  pretty (ErrorSync FenceGlobal) =-    "error_sync_global()"+  pretty (ErrorSync f handler) =+    sync f <> handler'+    where+      sync FenceLocal = "error_sync_local()"+      sync FenceGlobal = "error_sync_global()"+      handler' = case handler of+        Skip -> mempty+        _ -> " on_failure" <> brace (pretty handler)   pretty (Atomic _ (AtomicAdd t old arr ind x)) =     pretty old       <+> "<-"@@ -335,6 +369,7 @@ instance FreeIn KernelOp where   freeIn' (Atomic _ op) = freeIn' op   freeIn' (SharedAlloc _ size) = freeIn' size+  freeIn' (ErrorSync _ handler) = freeIn' handler   freeIn' _ = mempty  brace :: Doc a -> Doc a
src/Futhark/CodeGen/ImpCode/Kernels.hs view
@@ -5,6 +5,8 @@     KernelArg (..),     HostCode,     HostOp (..),+    Bookkeeping (..),+    bookkeepingSize,     KernelSafety (..),     numFailureParams,     FailureMsg (..),@@ -17,7 +19,13 @@ where  import Futhark.CodeGen.ImpCode-import Futhark.CodeGen.ImpCode.GPU (BlockDim, KernelConst (..), KernelConstExp)+import Futhark.CodeGen.ImpCode.GPU+  ( BlockDim,+    Bookkeeping (..),+    KernelConst (..),+    KernelConstExp,+    bookkeepingSize,+  ) import Futhark.IR.GPU.Sizes import Futhark.Util.Pretty @@ -73,6 +81,7 @@   | GetSize VName Name   | CmpSizeLe VName Name Exp   | GetSizeMax VName SizeClass+  | GetBookkeeping VName Bookkeeping   deriving (Show)  instance Pretty HostOp where
src/Futhark/CodeGen/ImpCode/Multicore.hs view
@@ -206,6 +206,7 @@     go f (If _ x y) = f x <> f y     go f (For _ _ x) = f x     go f (While _ x) = f x+    go f (OnFailure x y) = f x <> f y     go f (Op op) = goOp f op     go _ _ = mempty 
src/Futhark/CodeGen/ImpGen.hs view
@@ -50,6 +50,7 @@     lookupAcc,     askAttrs,     askProvenance,+    bodyMayFail,      -- * Building Blocks     TV,@@ -126,6 +127,7 @@ import Control.Monad import Control.Monad.Reader import Control.Monad.State+import Control.Monad.Writer (execWriter, tell) import Control.Parallel.Strategies import Data.Bifunctor (first) import Data.DList qualified as DL@@ -134,6 +136,7 @@ import Data.List.NonEmpty (NonEmpty (..)) import Data.Map.Strict qualified as M import Data.Maybe+import Data.Monoid (Any (..)) import Data.Set qualified as S import Data.String import Data.Text qualified as T@@ -1354,6 +1357,17 @@ -- | The provenance of whatever we are currently generating code for. askProvenance :: ImpM rep r op Provenance askProvenance = asks envProvenance++-- | Can executing this body fail? This is the case if it contains an+-- assertion, or calls a non-builtin function, which we conservatively assume+-- may fail. Assertions inside 'Op's are not considered.+bodyMayFail :: (ASTRep rep) => Body rep -> Bool+bodyMayFail = any (expMayFail . stmExp) . bodyStms+  where+    expMayFail (BasicOp Assert {}) = True+    expMayFail (Apply fname _ _ _) = not $ isBuiltInFunction fname+    expMayFail e = getAny $ execWriter $ walkExpM walker e+    walker = identityWalker {walkOnBody = \_ -> tell . Any . bodyMayFail}  -- | Wrap any code emitted in the enclosed section with the current provenance, -- if any.
src/Futhark/CodeGen/ImpGen/GPU.hs view
@@ -239,9 +239,8 @@     locksForInputs atomics ((c, (_, _, op)) : inputs')       | Just (op_lam, _) <- op,         AtomicLocking _ <- atomicUpdateLocking atomics op_lam = do-          let num_locks = 100151-          locks_arr <- genZeroes "withacc_locks" num_locks-          let locks = Locks locks_arr num_locks+          locks_arr <- bookkeepingArray Imp.BookkeepingLocks+          let locks = Locks locks_arr $ fromInteger $ Imp.bookkeepingSize Imp.BookkeepingLocks               extend env = env {hostLocks = M.insert c locks $ hostLocks env}           localEnv extend $ locksForInputs atomics inputs'       | otherwise =
src/Futhark/CodeGen/ImpGen/GPU/Base.hs view
@@ -18,6 +18,7 @@     warpLocal,     blockScan,     blockLoop,+    peeledKernelLoop,     isActive,     sKernel,     sKernelThread,@@ -31,7 +32,7 @@     blockCoverSpace,     fenceForArrays,     updateAcc,-    genZeroes,+    bookkeepingArray,     isPrimParam,     kernelConstToExp,     getSize,@@ -107,9 +108,10 @@     -- | A mapping from dimensions of nested SegOps to already     -- computed local thread IDs.  Only valid in non-virtualised case.     kernelLocalIdMap :: M.Map [SubExp] [Imp.TExp Int32],-    -- | Mapping from dimensions of nested SegOps to how many-    -- iterations the virtualisation loop needs.-    kernelChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)+    -- | Mapping from dimensions of nested SegOps to how many full block-sized+    -- chunks the virtualisation loop has; see 'peeledKernelLoop'. Constant dimensions+    -- are not included.+    kernelFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32)   }  kernelGlobalThreadId, kernelLocalThreadId, kernelBlockId :: KernelConstants -> Imp.TExp Int32@@ -180,15 +182,16 @@             Nothing ->               error $ "Missing locks for " ++ prettyString acc --- | Generate a constant device array of 32-bit integer zeroes with--- the given number of elements.  Initialised with a replicate.-genZeroes :: Name -> Int -> CallKernelGen VName-genZeroes desc n = genConstants $ do-  counters_mem <- sAlloc (desc <> "_mem") (4 * fromIntegral n) (Space "device")-  let shape = Shape [intConst Int64 (fromIntegral n)]-  counters <- sArrayInMem desc int32 shape counters_mem-  sReplicate counters $ intConst Int32 0-  pure (namesFromList [counters_mem], counters)+-- | Retrieve a bookkeeping array.+bookkeepingArray :: Imp.Bookkeeping -> CallKernelGen VName+bookkeepingArray b = do+  mem <- sDeclareMem (desc <> "_mem") (Space "device")+  sOp $ Imp.GetBookkeeping mem b+  sArrayInMem desc int32 (Shape [intConst Int64 $ Imp.bookkeepingSize b]) mem+  where+    desc = case b of+      Imp.BookkeepingCounters -> "counters"+      Imp.BookkeepingLocks -> "locks"  compileThreadExp :: ExpCompiler GPUMem KernelEnv Imp.KernelOp compileThreadExp (Pat [pe]) (BasicOp (Opaque _ se)) =@@ -226,6 +229,28 @@           i <- dPrimVE "i" $ chunk_i * num_threads + tid           sWhen (i .<. n) $ f i +-- | @peeledKernelLoop tid num_threads num_full_chunks n f@ invokes @f@ on the+-- iterations @tid@, @tid+num_threads@, and so on, below @n@, where+-- @num_full_chunks@ must be @n `quot` num_threads@. Only the final partial+-- chunk is bounds checked, so the body is generated twice; see Note [Peeled+-- block loops]. The body must contain thread-level code.+peeledKernelLoop ::+  (IntExp t) =>+  Imp.TExp t ->+  Imp.TExp t ->+  Imp.TExp t ->+  Imp.TExp t ->+  (Imp.TExp t -> InKernelGen ()) ->+  InKernelGen ()+peeledKernelLoop tid num_threads num_full_chunks n f =+  localOps threadOperations $ do+    sFor "chunk_i" num_full_chunks $ \chunk_i ->+      f =<< dPrimVE "i" (sMinAs (chunk_i * num_threads + tid) (n - 1))+    i <- dPrimVE "i" $ num_full_chunks * num_threads + tid+    sWhen (i .<. n) $ f i+  where+    sMinAs x y = TPrimExp $ BinOpExp (SMin $ expIntType x) (untyped x) (untyped y)+ -- | Assign iterations of a for-loop to threads in the threadblock.  The -- passed-in function is invoked with the (symbolic) iteration.  For -- multidimensional loops, use 'blockCoverSpace'.@@ -236,11 +261,13 @@   InKernelGen () blockLoop n f = do   constants <- kernelConstants <$> askEnv-  kernelLoop-    (kernelLocalThreadId constants `sExtAs` n)-    (kernelBlockSize constants `sExtAs` n)-    n-    f+  let tid = kernelLocalThreadId constants `sExtAs` n+      num_threads = kernelBlockSize constants `sExtAs` n+  if n == num_threads+    then localOps threadOperations $ f tid+    else do+      num_full_chunks <- dPrimVE "num_full_chunks" $ n `quot` num_threads+      peeledKernelLoop tid num_threads num_full_chunks n f  -- | Iterate collectively though a multidimensional space, such that -- all threads in the block participate.  The passed-in function is@@ -322,18 +349,17 @@   Imp.TExp Int32 ->   Imp.TExp Bool ->   [VName] ->-  InKernelGen () ->+  Imp.Fence ->   Lambda GPUMem ->   InKernelGen ()-inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs barrier scan_lam = do+inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs fence scan_lam = do   skip_threads <- dPrim "skip_threads"   let actual_params = lambdaParams scan_lam       (x_params, y_params) =         splitAt (length actual_params `div` 2) actual_params       y_to_x =         forM_ (zip x_params y_params) $ \(x, y) ->-          when (isPrimParam x) $-            copyDWIM (paramName x) [] (Var (paramName y)) []+          copyDWIM (paramName x) [] (Var (paramName y)) []    -- Set initial y values   sComment "read input for in-block scan" $@@ -366,25 +392,32 @@               $ compileBody' x_params               $ lambdaBody scan_lam -      maybeBarrier =-        sWhen-          (lockstep_width .<=. tvExp skip_threads)-          barrier+      -- See Note [Failing block-level operators].+      maybeBarrier+        | may_fail = sOp $ Imp.ErrorSync fence mempty+        | otherwise =+            sWhen+              (lockstep_width .<=. tvExp skip_threads)+              barrier +      warpLocal'+        | may_fail = id+        | otherwise = warpLocal+   sComment "in-block scan (hopefully no barriers needed)" $ do     skip_threads <-- 1     sWhile (tvExp skip_threads .<. block_size) $ do       thread_active <-         dPrimVE "thread_active" $ tvExp skip_threads .<=. in_block_id .&&. active -      warpLocal $ do+      warpLocal' $ do         sWhen thread_active . sComment "read operands" $           zipWithM_ (readParam (sExt64 $ tvExp skip_threads)) x_params arrs         sComment "perform operation" $ op_to_x thread_active        maybeBarrier -      warpLocal . sWhen thread_active . sComment "write result" $+      warpLocal' . sWhen thread_active . sComment "write result" $         sequence_ $           zipWith3 writeResult x_params y_params arrs @@ -397,9 +430,15 @@     ltid32 = kernelLocalThreadId constants     ltid = sExt64 ltid32     array_scan = not $ all primType $ lambdaReturnType scan_lam+    barrier = sOp $ Imp.Barrier fence+    may_fail = bodyMayFail $ lambdaBody scan_lam -    readInitial p arr =+    readInitial p arr = do       copyDWIMFix (paramName p) [] (Var arr) [ltid]+      -- An array-typed element is also read @arrs_full_size@ elements along,+      -- by 'readParam', so put it there too.+      unless (isPrimParam p) $+        copyDWIMFix arr [ltid + arrs_full_size] (Var arr) [ltid]      readParam behind p arr       | isPrimParam p =@@ -408,10 +447,38 @@           copyDWIMFix (paramName p) [] (Var arr) [ltid - behind + arrs_full_size]      writeResult x y arr = do-      when (isPrimParam x) $-        copyDWIMFix arr [ltid] (Var $ paramName x) []+      copyDWIMFix arr [ltid] (Var $ paramName x) []+      unless (isPrimParam x) $+        copyDWIMFix arr [ltid + arrs_full_size] (Var $ paramName x) []       copyDWIM (paramName y) [] (Var $ paramName x) [] +-- | 'blockScan' splits the block into chunks of this size, which are scanned+-- separately. The only requirement is that it should not be less than the+-- square root of the block size. With 32, we will work on blocks of size 1024+-- or smaller, which fits every device Troels has seen. Still, it would be nicer+-- if it were a runtime parameter. Some day.+blockScanChunkSize :: (Num a) => a+blockScanChunkSize = 32++-- | Scan the @w@ first elements of the given arrays, in place. @w@ must be at+-- most the thread block size; you'll need to wrap this function in a loop if+-- you want to scan larger arrays.+--+-- An array-typed element is kept twice: at index @i@, where the result is+-- expected, and again at index @i + arrs_full_size@. It is the latter copy that+-- a thread reads as its x operand, because the operator writes its result+-- through its x parameter, which may well be the former. With @k@ chunks, the+-- first @k@ elements are overwritten while computing the per-chunk carry-ins,+-- so they are saved at indexes @arrs_full_size + k@ to @arrs_full_size + 2k@,+-- whose copies are not needed by then. Only the @w@ first elements need be+-- within the /shape/ of the arrays, but the indexes from @arrs_full_size@ to+-- @arrs_full_size + w@ must also be backed by memory; see+-- 'Futhark.CodeGen.ImpGen.GPU.Block.blockScanElems'.+--+-- The callers in "Futhark.CodeGen.ImpGen.GPU.SegScan.TwoPass" and+-- "Futhark.CodeGen.ImpGen.GPU.SegRed" pass arrays whose elements /are/ the x+-- and y parameters of the operator, so for them the copies that keep the two in+-- step are redundant, but harmless. blockScan ::   Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->   Imp.TExp Int64 ->@@ -419,7 +486,7 @@   Lambda GPUMem ->   [VName] ->   InKernelGen ()-blockScan seg_flag arrs_full_size w lam arrs = do+blockScan seg_flag arrs_full_size w lam arrs = localOps threadOperations $ do   constants <- kernelConstants <$> askEnv   renamed_lam <- renameLambda lam @@ -437,12 +504,7 @@   -- scanned separately. Typically, these chunks are at most the   -- lockstep width, which enables barrier-free execution inside them.   ---  -- We hardcode the chunk size here. The only requirement is that it-  -- should not be less than the square root of the block size. With-  -- 32, we will work on blocks of size 1024 or smaller, which fits-  -- every device Troels has seen. Still, it would be nicer if it were-  -- a runtime parameter. Some day.-  let chunk_size = 32+  let chunk_size = blockScanChunkSize       simd_width = kernelWaveSize constants       chunk_id = ltid32 `quot` chunk_size       in_chunk_id = ltid32 - chunk_id * chunk_size@@ -455,19 +517,18 @@           chunk_size           active           arrs-          barrier+          barrier_fence       array_scan = not $ all primType $ lambdaReturnType lam-      barrier-        | array_scan =-            sOp $ Imp.Barrier Imp.FenceGlobal-        | otherwise =-            sOp $ Imp.Barrier fence+      barrier_fence+        | array_scan = Imp.FenceGlobal+        | otherwise = fence+      barrier = sOp $ Imp.Barrier barrier_fence        errorsync         | array_scan =-            sOp $ Imp.ErrorSync Imp.FenceGlobal+            sOp $ Imp.ErrorSync Imp.FenceGlobal mempty         | otherwise =-            sOp $ Imp.ErrorSync Imp.FenceLocal+            sOp $ Imp.ErrorSync Imp.FenceLocal mempty        writeBlockResult p arr =         copyDWIMFix arr [sExt64 chunk_id] (Var $ paramName p) []@@ -478,19 +539,24 @@   doInChunkScan seg_flag ltid_in_bounds lam   barrier +  -- With more than one chunk, the first 'num_chunks' elements are overwritten+  -- by the chunk results, which are then scanned to produce the carry-ins.+  num_chunks <- dPrimVE "num_chunks" $ sExt32 $ w `ceilDiv` blockScanChunkSize   let is_first_block = chunk_id .==. 0+      several_chunks = num_chunks .>. 1+      scans_carries = several_chunks .&&. ltid32 .<. num_chunks   when array_scan $ do     sComment "save correct values for first block" $-      sWhen is_first_block $+      sWhen scans_carries $         forM_ (zip x_params arrs) $ \(x, arr) ->           unless (isPrimParam x) $-            copyDWIMFix arr [arrs_full_size + sExt64 chunk_size + ltid] (Var $ paramName x) []+            copyDWIMFix arr [arrs_full_size + sExt64 num_chunks + ltid] (Var $ paramName x) []      barrier    let last_in_block = in_chunk_id .==. chunk_size - 1   sComment "last thread of block 'i' writes its result to offset 'i'" $-    sWhen (last_in_block .&&. ltid_in_bounds) $+    sWhen (several_chunks .&&. last_in_block .&&. ltid_in_bounds) $       everythingVolatile $         zipWithM_ writeBlockResult x_params arrs @@ -502,20 +568,20 @@           flag_true (from * chunk_size + chunk_size - 1) (to * chunk_size + chunk_size - 1)   sComment     "scan the first block, after which offset 'i' contains carry-in for block 'i+1'"-    $ doInChunkScan first_block_seg_flag (is_first_block .&&. ltid_in_bounds) renamed_lam+    $ doInChunkScan first_block_seg_flag scans_carries renamed_lam    errorsync    when array_scan $ do     sComment "move correct values for first block back a block" $-      sWhen is_first_block $+      sWhen scans_carries $         forM_ (zip x_params arrs) $ \(x, arr) ->           unless (isPrimParam x) $             copyDWIMFix               arr               [arrs_full_size + ltid]               (Var arr)-              [arrs_full_size + sExt64 chunk_size + ltid]+              [arrs_full_size + sExt64 num_chunks + ltid]      barrier @@ -541,8 +607,7 @@        write_final_result =         forM_ (zip x_params arrs) $ \(p, arr) ->-          when (isPrimParam p) $-            copyDWIMFix arr [ltid] (Var $ paramName p) []+          copyDWIMFix arr [ltid] (Var $ paramName p) []    sComment "carry-in for every block except the first" $     localOps threadOperations $ do@@ -550,17 +615,20 @@       sComment "perform operation" op_to_x       sComment "write final result" $ sUnless no_carry_in write_final_result -  barrier+  -- See Note [Failing block-level operators].+  sOp $ Imp.ErrorSync barrier_fence mempty    sComment "restore correct values for first block" $-    sWhen (is_first_block .&&. ltid_in_bounds) $-      forM_ (zip3 x_params y_params arrs) $ \(x, y, arr) ->+    sWhen scans_carries $+      forM_ (zip3 x_params y_params arrs) $ \(_x, y, arr) ->         if isPrimParam y           then copyDWIMFix arr [ltid] (Var $ paramName y) []-          else copyDWIMFix (paramName x) [] (Var arr) [arrs_full_size + ltid]+          else copyDWIMFix arr [ltid] (Var arr) [arrs_full_size + ltid]    barrier +-- | Reduce the @w@ first elements of the given arrays, leaving the result in+-- element zero of each. The others are clobbered. blockReduce ::   Imp.TExp Int32 ->   Lambda GPUMem ->@@ -580,26 +648,18 @@   constants <- kernelConstants <$> askEnv    let local_tid = kernelLocalThreadId constants--      barrier-        | all primType $ lambdaReturnType lam = sOp $ Imp.Barrier Imp.FenceLocal-        | otherwise = sOp $ Imp.Barrier Imp.FenceGlobal+      may_fail = bodyMayFail $ lambdaBody lam        errorsync-        | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal-        | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal+        | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal mempty+        | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal mempty        readReduceArgument param arr = do         let i = local_tid + tvExp offset         copyDWIMFix (paramName param) [] (Var arr) [sExt64 i]        writeReduceOpResult param arr =-        when (isPrimParam param) $-          copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []--      writeArrayOpResult param arr =-        unless (isPrimParam param) $-          copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []+        copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []    let (reduce_acc_params, reduce_arr_params) =         splitAt (length arrs) $ lambdaParams lam@@ -622,7 +682,10 @@         sComment "write result of operation" $           zipWithM_ writeReduceOpResult reduce_acc_params arrs -      wave_size = kernelWaveSize constants+      -- See Note [Failing block-level operators].+      wave_size+        | may_fail = 1+        | otherwise = kernelWaveSize constants       tblock_size = kernelBlockSize constants       wave_id = local_tid `quot` wave_size       in_wave_id = local_tid - wave_id * wave_size@@ -652,20 +715,14 @@         arg_in_bounds .&&. is_first_thread_in_wave .&&. wave_not_skipped       cross_wave_reductions =         sWhile doing_cross_wave_reductions $ do-          barrier           offset <-- tvExp skip_waves * wave_size           sWhen apply_in_cross_wave_iteration do_reduce+          errorsync           skip_waves <-- tvExp skip_waves * 2 -  in_wave_reductions-  cross_wave_reductions+  unless may_fail in_wave_reductions   errorsync--  unless (all isPrimParam reduce_acc_params) $-    sComment "Copy array-typed operands to result array" $-      sWhen (local_tid .==. 0) $-        localOps threadOperations $-          zipWithM_ writeArrayOpResult reduce_acc_params arrs+  cross_wave_reductions  compileThreadOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp compileThreadOp pat (Alloc size space) =@@ -864,8 +921,11 @@       acquire_fence       dLParams acc_params       bind_acc_params-      op_body-      do_hist+      -- A thread whose operator fails must still release the lock, or other+      -- threads would wait for it forever.+      update <- collect $ op_body >> do_hist+      on_failure <- collect release_lock+      emit $ Imp.OnFailure update on_failure       release_lock       break_loop   where@@ -1005,7 +1065,7 @@             kernelNumThreads = sExt32 (tblock_size' * num_tblocks'),             kernelWaveSize = Imp.le32 wave_size,             kernelLocalIdMap = mempty,-            kernelChunkItersMap = mempty+            kernelFullChunksMap = mempty           }    let set_constants = do@@ -1095,7 +1155,7 @@             kernelNumThreads = sExt32 (tblock_size' * num_tblocks'),             kernelWaveSize = 0,             kernelLocalIdMap = mempty,-            kernelChunkItersMap = mempty+            kernelFullChunksMap = mempty           }        wrapKernel m = do@@ -1142,7 +1202,7 @@         (tvExp phys_tblock_id + i * sExt32 (kernelNumBlocks constants))     -- Make sure the virtual block is actually done before we let     -- another virtual block have its way with it.-    sOp $ Imp.ErrorSync Imp.FenceGlobal+    sOp $ Imp.ErrorSync Imp.FenceGlobal mempty virtualiseBlocks _ _ m =   m . tvExp . kernelBlockIdVar . kernelConstants =<< askEnv @@ -1477,3 +1537,41 @@   copyDWIMFix (patElemName pe) is what [] compileThreadResult _ _ TileReturns {} =   compilerBugS "compileThreadResult: TileReturns unhandled."++-- Note [Peeled block loops]+--+-- When the threads of a block cover an iteration space larger than the block+-- (virtualised SegOps), each thread handles one iteration per block-sized+-- chunk, and only the final chunk can be partial. A bounds check in every chunk+-- keeps the kernel compiler from overlapping the memory accesses of consecutive+-- chunks, which makes memory-bound loops considerably slower.+-- 'peeledKernelLoop' therefore runs the full chunks without a check, and+-- generates the body again for the final chunk, with a bounds check.+--+-- A bounds check also tells the kernel compiler that the iteration is in+-- bounds, which lets it remove checks in the body that follow from it, such as+-- those of accumulator updates, which likewise keep memory accesses from+-- overlapping. To retain that information, the iteration is clamped to @n-1@ in+-- the full chunks, which does not change its value. This is a bit of a hack,+-- but it seems robust.+--+-- For the same reason, the number of full chunks should be computed in the+-- kernel when it is a constant, rather than be passed in as a kernel argument,+-- so that the kernel compiler can unroll the loop. This is why+-- 'kernelFullChunksMap' does not contain constant dimensions.++-- Note [Failing block-level operators]+--+-- Nothing prevents reduction/scan operators from having arbitrary assertions+-- inside them, which is a bit complicated to handle due to the intricate+-- synchronisation we perform during collective block-level reductions and+-- scans.+--+-- The problem is that we implement failures by jumping to the next+-- 'Imp.ErrorSync' in the program text, so no other barrier, nor the warp-level+-- synchronisation of 'warpLocal', must occur in between (see "Bounds Checking+-- on GPU", HLPP '20). In 'blockReduce' and 'blockScan', we handle operators+-- that may fail ('bodyMayFail') by skipping the warp-level phase (which is just+-- an optimisation anyway) and following each application with an+-- 'Imp.ErrorSync' in the same iteration. For operators that cannot fail, an+-- 'Imp.ErrorSync' is just a barrier.
src/Futhark/CodeGen/ImpGen/GPU/Block.hs view
@@ -29,7 +29,7 @@ import Futhark.IR.Mem.LMAD qualified as LMAD import Futhark.Transform.Rename import Futhark.Util (chunks, mapAccumLM, takeLast)-import Futhark.Util.IntegralExp (ceilDiv, rem)+import Futhark.Util.IntegralExp (ceilDiv, quot, rem) import Prelude hiding (quot, rem)  -- | @flattenArray k flat arr@ flattens the outer @k@ dimensions of@@ -103,6 +103,8 @@   (Imp.TExp Int32 -> TV Int64 -> InKernelGen ()) ->   InKernelGen () blockChunkLoop w m = do+  -- The loop body contains error syncs, and failures must not jump into it.+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty   constants <- kernelConstants <$> askEnv   let max_chunk_size = sExt32 $ kernelBlockSize constants   num_chunks <- dPrimVE "num_chunks" $ w `ceilDiv` max_chunk_size@@ -123,6 +125,27 @@ crossesSegment segment_size from to =   (to - from) .>. (to `rem` sExt32 segment_size) +-- | Given a chunk offset, adjust a segment predicate ('crossesSegment') defined+-- in terms of whole-array indexes to instead be defined in terms of chunk-local+-- indexes. Also returns whether the chunk begins a new segment. This is used to+-- process segmented reductions and scans that are chunked.+chunkSegFlags ::+  Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->+  Imp.TExp Int32 ->+  ( Imp.TExp Bool,+    Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool)+  )+chunkSegFlags seg_flag chunk_start =+  ( maybe false (\flag_true -> flag_true (chunk_start - 1) chunk_start) seg_flag,+    fmap shifted seg_flag+  )+  where+    shifted flag_true from to =+      flag_true (chunk_start + from) (chunk_start + to)++-- | Scan the @w@ first elements of the given arrays in-place. The scan is done+-- one block-sized chunk at a time, but @seg_flag@ is passed indexes relative to+-- the whole array. virtualisedBlockScan ::   Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->   Imp.TExp Int32 ->@@ -131,14 +154,9 @@   InKernelGen () virtualisedBlockScan seg_flag w lam arrs = do   blockChunkLoop w $ \chunk_start chunk_size -> do-    constants <- kernelConstants <$> askEnv-    let ltid = kernelLocalThreadId constants-        crosses_segment =-          case seg_flag of-            Nothing -> false-            Just flag_true ->-              flag_true (sExt32 (chunk_start - 1)) (sExt32 chunk_start)-    sComment "possibly incorporate carry" $+    ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+    let (crosses_segment, shifted) = chunkSegFlags seg_flag chunk_start+    sComment "possibly incorporate carry" . localOps threadOperations $       sWhen (chunk_start .>. 0 .&&. ltid .==. 0 .&&. bNot crosses_segment) $ do         carry_idx <- dPrimVE "carry_idx" $ sExt64 chunk_start - 1         applyRenamedLambda@@ -150,10 +168,21 @@      arrs_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) arrs -    sOp $ Imp.ErrorSync Imp.FenceLocal+    sOp $ Imp.ErrorSync Imp.FenceLocal mempty -    blockScan seg_flag (sExt64 w) (tvExp chunk_size) lam arrs_chunks+    -- The chunks past this one are still live, so the second copy that+    -- 'blockScan' keeps of an array-typed element must go beyond the whole+    -- space, not merely beyond the chunk.+    blockScan shifted (sExt64 w) (tvExp chunk_size) lam arrs_chunks +-- | Semantically, a block-level statement finishes before the next one. This+-- means that any statement that writes memory must be enclosed in barriers, as+-- sibling statements may reuse the same memory. Our current approach to this is+-- a bit naive and may result in duplicate barriers. If this ever turns out to+-- be a performance problem, we'll do something more sophisticated.+beforeBlockWrite :: InKernelGen ()+beforeBlockWrite = sOp $ Imp.Barrier Imp.FenceLocal+ copyInBlock :: CopyCompiler GPUMem KernelEnv Imp.KernelOp copyInBlock pt destloc srcloc = do   dest_space <- entryMemSpace <$> lookupMemory (memLocName destloc)@@ -179,6 +208,7 @@         (sliceMemLoc destloc destslice')         (sliceMemLoc srcloc srcslice')     _ -> do+      beforeBlockWrite       blockCoverSpace (map sExt32 dims) $ \is ->         lmadCopy           pt@@ -244,6 +274,19 @@            pure (Just l', f l' (Space "shared") local_subhistos) +-- | The virtualisation that 'blockCoverSegSpace' will actually use for this+-- index space.  Maybe we can statically detect that the requested one is really+-- a 'SegNoVirtFull', and generate ever-so-slightly simpler code.  Covering an+-- index space twice has to know this, as it decides whether the cover loops -+-- and so introduces a scope of its own for the indexes.+blockCoverVirt :: SegVirt -> SegSpace -> InKernelGen SegVirt+blockCoverVirt virt space = do+  tblock_size <- kernelBlockSize . kernelConstants <$> askEnv+  pure $+    if map pe64 (segSpaceDims space) == [tblock_size]+      then SegNoVirtFull (SegSeqDims [])+      else virt+ blockCoverSegSpace :: SegVirt -> SegSpace -> InKernelGen () -> InKernelGen () blockCoverSegSpace virt space m = do   let (ltids, dims) = unzip $ unSegSpace space@@ -251,24 +294,23 @@    constants <- kernelConstants <$> askEnv   let tblock_size = kernelBlockSize constants-  -- Maybe we can statically detect that this is actually a-  -- SegNoVirtFull and generate ever-so-slightly simpler code.-  let virt' = if dims' == [tblock_size] then SegNoVirtFull (SegSeqDims []) else virt+  virt' <- blockCoverVirt virt space   case virt' of     SegVirt -> do-      iters <- M.lookup dims . kernelChunkItersMap . kernelConstants <$> askEnv-      case iters of-        Nothing -> do-          iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'-          blockLoop iterations $ \i -> do+      iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'+      let onIteration i = do             dIndexSpace (zip ltids dims') $ sExt64 i             m-        Just num_chunks -> localOps threadOperations $ do-          let ltid = kernelLocalThreadId constants-          sFor "chunk_i" num_chunks $ \chunk_i -> do-            i <- dPrimVE "i" $ chunk_i * sExt32 tblock_size + ltid-            dIndexSpace (zip ltids dims') $ sExt64 i-            sWhen (inBounds (Slice (map (DimFix . le64) ltids)) dims') m+      full_chunks <- M.lookup dims . kernelFullChunksMap . kernelConstants <$> askEnv+      case full_chunks of+        Nothing -> blockLoop iterations onIteration+        Just num_full_chunks ->+          peeledKernelLoop+            (kernelLocalThreadId constants)+            (sExt32 tblock_size)+            num_full_chunks+            iterations+            onIteration     SegNoVirt -> localOps threadOperations $ do       zipWithM_ dPrimV_ ltids =<< localThreadIDs dims       sWhen (isActive $ zip ltids dims) m@@ -288,7 +330,8 @@ -- The static arrays stuff does not work inside kernels. compileBlockExp (Pat [dest]) (BasicOp (ArrayVal vs t)) =   compileBlockExp (Pat [dest]) (BasicOp (ArrayLit (map Constant vs) (Prim t)))-compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) =+compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) = do+  beforeBlockWrite   forM_ (zip [0 ..] es) $ \(i, e) ->     copyDWIMFix (patElemName dest) [fromIntegral (i :: Int64)] e [] compileBlockExp _ (BasicOp (UpdateAcc safety acc is vs)) = do@@ -296,6 +339,7 @@   sWhen (ltid .==. 0) $ updateAcc safety acc is vs   sOp $ Imp.Barrier Imp.FenceLocal compileBlockExp (Pat [dest]) (BasicOp (Replicate ds se)) | ds /= mempty = do+  beforeBlockWrite   flat <- newVName "rep_flat"   is <- replicateM (arrayRank dest_t) (newVName "rep_i")   let is' = map le64 is@@ -305,6 +349,7 @@   where     dest_t = patElemType dest compileBlockExp (Pat [dest]) (BasicOp (Iota n e s it)) = do+  beforeBlockWrite   n' <- toExp n   e' <- toExp e   s' <- toExp s@@ -339,7 +384,7 @@   -- Avoid that by always doing an error sync here.  Potential   -- improvement: only do this if any errors are pending (this could   -- also be handled in later codegen).-  when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal+  when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal mempty   defCompileExp dest e   where     doSync Loop {} = True@@ -362,192 +407,401 @@ blockAlloc dest _ _ =   error $ "Invalid target for in-kernel allocation: " ++ show dest -compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp-compileBlockOp pat (Alloc size space) =-  blockAlloc pat size space-compileBlockOp pat (Inner (SegOp (SegMap lvl space _ body))) = do-  compileFlatId space+-- | How many elements an operand array for 'blockScan' must be backed by, given+-- the number of elements actually scanned and the return types of the operator.+-- Array-typed elements are kept twice over; see 'blockScan'.+blockScanElems :: Imp.TExp Int64 -> [Type] -> Imp.TExp Int64+blockScanElems n ts+  | all primType ts = n+  | otherwise = 2 * n -  blockCoverSegSpace (segVirt lvl) space $-    compileStms mempty (bodyStms body) $-      zipWithM_ (compileThreadResult space) (patElems pat) $-        bodyResult body-  sOp $ Imp.ErrorSync Imp.FenceLocal-compileBlockOp pat (Inner (SegOp (SegScan lvl space _ body scans post_op))) = do-  compileFlatId space+-- | A shared memory array of @elems@ values of type @elem_t@, staging the+-- operands of a 'blockScan' or 'blockReduce'. Note that 'blockScan' needs+-- memory past the index space it scans, and that excess is deliberately left+-- outside the shape (but is given by @elems@), as nothing but 'blockScan' may+-- touch it.+sharedStagingArray ::+  Name -> Type -> Imp.TExp Int64 -> Shape -> InKernelGen VName+sharedStagingArray desc elem_t elems shape = do+  mem <-+    sAlloc (desc <> "_mem") (Imp.bytes $ elems * unCount (typeSize elem_t)) $+      Space "shared"+  sArrayInMem desc (elemType elem_t) shape mem -  let (ltids, dims) = unzip $ unSegSpace space-      dims' = map pe64 dims-      [scan] = scans-      shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)-      scan_ts = concatMap shpT scans-      shpOfT t s =-        arrayShape $-          foldr (flip arrayOfRow) (arrayOfShape t s) $-            segSpaceDims space-      (scan_pars, map_pars) =-        splitAt (length $ segBinOpNeutral scan) $ lambdaParams $ segPostOpLambda post_op-      (body_scan_res, body_map_res) =-        splitAt (length $ segBinOpNeutral scan) $ bodyResult body+-- | Will a single thread be responsible for more than one element of this index+-- space? If so, a value computed for one element in one phase of a multi-phase+-- operation cannot be kept in a scalar variable until a later phase, as the+-- thread will by then have overwritten it with the value belonging to some+-- other element.+severalElemsPerThread :: SegVirt -> Bool+severalElemsPerThread SegVirt = True+severalElemsPerThread SegNoVirt = False+severalElemsPerThread (SegNoVirtFull (SegSeqDims seq_dims)) = not $ null seq_dims -  scan_out <- forM scan_ts $ \(s, t) ->-    sAllocArray "scan_out" (elemType t) (shpOfT t s) $ Space "shared"+-- | Compile a block-level 'SegScan'. This proceeds in three phases, separated+-- by barriers:+--+-- 1. The kernel body is run to produce the scan inputs as distinct arrays.+--+-- 2. The scan itself is performed.+--+-- 3. Finally the post operator is applied to the scan results.+--+-- Everything that has to survive a barrier lives in shared memory, and because+-- the index space may be virtualised (such that a thread handles more than one+-- element), that includes the map results of the kernel body.+compileBlockScan ::+  Pat LetDecMem ->+  SegLevel ->+  SegSpace ->+  KernelBody GPUMem ->+  [SegBinOp GPUMem] ->+  SegPostOp GPUMem ->+  InKernelGen ()+compileBlockScan pat lvl space body scans post_op = do+  compileFlatId space -  dScope Nothing $ scopeOfLParams $ lambdaParams $ segPostOpLambda post_op+  -- Whether a thread ends up handling several elements of the index space+  -- decides both where the map results have to live, and how the post operator+  -- gets at the indexes again.+  virt <- blockCoverVirt (segVirt lvl) space -  blockCoverSegSpace (segVirt lvl) space $-    compileStms mempty (bodyStms body) $ do-      forM_ (zip scan_out body_scan_res) $ \(dest, res) ->-        copyDWIMFix-          dest-          (map Imp.le64 ltids)-          (kernelResultSubExp res)-          []+  scan_arrs <- mapM scanArray scan_ts+  map_dests <- mapM (mapResDest virt) map_ts -      sComment "bind map results to post lambda params" $-        forM_ (zip map_pars body_map_res) $ \(par, res) ->-          copyDWIMFix-            (paramName par)-            []-            (kernelResultSubExp res)-            []+  dScope Nothing $ scopeOfLParams $ lambdaParams post_lam -  sOp $ Imp.ErrorSync Imp.FenceLocal+  computeScanInputs virt scan_arrs map_dests+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty+  doScan virt scan_arrs+  applyPostOp virt scan_arrs map_dests -  let seg_flag = case dims' of-        [_] -> Nothing -- Nonsegmented.-        _ -> Just $ crossesSegment $ last dims'+  sOp $ Imp.Barrier Imp.FenceGlobal+  where+    (ltids, dims) = unzip $ unSegSpace space+    dims' = map pe64 dims+    [scan] = scans+    post_lam = segPostOpLambda post_op+    num_scan_res = length $ segBinOpNeutral scan+    (scan_pars, map_pars) = splitAt num_scan_res $ lambdaParams post_lam+    (body_scan_res, body_map_res) = splitAt num_scan_res $ bodyResult body+    scan_ts = concatMap shpT scans+    shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)+    map_ts = map ((mempty,) . paramType) map_pars -  -- blockScan needs to treat the scan output as a one-dimensional-  -- array of scan elements, so we invent some new flattened arrays-  -- here.-  dims_flat <- dPrimV "dims_flat" $ product dims'-  arrs_flat <--    mapM (flattenArray (length dims') dims_flat) scan_out+    shapeFor s t =+      arrayShape $ foldr (flip arrayOfRow) (arrayOfShape t s) dims -  case segVirt lvl of-    SegVirt ->-      virtualisedBlockScan-        seg_flag-        (sExt32 $ tvExp dims_flat)-        (segBinOpLambda scan)-        arrs_flat-    _ ->-      blockScan-        seg_flag-        (product dims')+    -- One value per element of the index space, plus whatever 'blockScan'+    -- needs past that.+    scanArray (s, t) =+      sharedStagingArray+        "scan_out"+        (t `arrayOfShape` s)+        (blockScanElems (product dims') (lambdaReturnType (segBinOpLambda scan)))+        (shapeFor s t)++    -- Map results are not operands of the scan, so one value per element of+    -- the index space is all they need.+    mapArray (s, t) =+      sharedStagingArray+        "map_out"+        (t `arrayOfShape` s)         (product dims')-        (segBinOpLambda scan)-        arrs_flat+        (shapeFor s t) -  -- FIXME: we actually need something like blockCoverSegSpacee here, although in-  -- practice we currently do not generate virtualised scans.-  sWhen (isActive $ zip ltids dims) . localOps threadOperations $ do-    sComment "bind scan results to post lambda params" $-      forM_ (zip scan_pars scan_out) $ \(par, acc) ->-        copyDWIMFix (paramName par) [] (Var acc) (map Imp.le64 ltids)+    -- Where does a map result produced by the kernel body live until the post+    -- operator needs it? A shared memory array when a thread handles several+    -- elements, as a scalar would by then have been overwritten; otherwise the+    -- post operator parameter itself. Accumulators are phantoms in ImpGen, so+    -- they need no storage at all.+    mapResDest virt (s, t)+      | isAcc t = pure Nothing+      | severalElemsPerThread virt = Just <$> mapArray (s, t)+      | otherwise = pure Nothing -    let res = fmap resSubExp $ bodyResult $ lambdaBody $ segPostOpLambda post_op-    sComment "compute post op." $-      compileStms mempty (bodyStms $ lambdaBody $ segPostOpLambda post_op) $ do-        sComment "write values" $-          forM_ (zip (patElems pat) res) $ \(pe, subexp) ->-            copyDWIMFix (patElemName pe) (map le64 ltids) subexp []+    seg_flag = case dims' of+      [_] -> Nothing -- Nonsegmented.+      _ -> Just $ crossesSegment $ last dims' -  sOp $ Imp.Barrier Imp.FenceGlobal-compileBlockOp pat (Inner (SegOp (SegRed lvl space _ body ops))) = do-  compileFlatId space+    -- Run the kernel body for every element of the index space, stashing the+    -- scan inputs - and the map results, if they cannot be kept in scalars -+    -- in shared memory.+    computeScanInputs virt scan_arrs map_dests =+      blockCoverSegSpace virt space $+        compileStms mempty (bodyStms body) $ do+          let ltids' = map Imp.le64 ltids+          forM_ (zip scan_arrs body_scan_res) $ \(arr, res) ->+            copyDWIMFix arr ltids' (kernelResultSubExp res) [] -  let dims' = map pe64 dims-      mkTempArr t =-        sAllocArray "red_arr" (elemType t) (Shape dims <> arrayShape t) $ Space "shared"+          sComment "stash map results for the post operator" $+            forM_ (zip3 map_pars map_dests body_map_res) $ \(par, dest, res) ->+              case dest of+                Just arr -> copyDWIMFix arr ltids' (kernelResultSubExp res) []+                Nothing -> copyDWIMFix (paramName par) [] (kernelResultSubExp res) [] -  tmp_arrs <- mapM mkTempArr $ concatMap (lambdaReturnType . segBinOpLambda) ops-  blockCoverSegSpace (segVirt lvl) space $-    compileStms mempty (bodyStms body) $ do-      let (red_res, map_res) =-            splitAt (segBinOpResults ops) $ bodyResult body-      forM_ (zip tmp_arrs red_res) $ \(dest, res) ->-        copyDWIMFix dest (map Imp.le64 ltids) (kernelResultSubExp res) []-      zipWithM_ (compileThreadResult space) map_pes map_res+    -- blockScan needs to treat the scan input as a one-dimensional array of+    -- scan elements, so we invent some new flattened arrays here.+    doScan virt scan_arrs = do+      dims_flat <- dPrimV "dims_flat" $ product dims'+      arrs_flat <- mapM (flattenArray (length dims') dims_flat) scan_arrs+      case virt of+        SegVirt ->+          virtualisedBlockScan+            seg_flag+            (sExt32 $ tvExp dims_flat)+            (segBinOpLambda scan)+            arrs_flat+        _ ->+          blockScan+            seg_flag+            (product dims')+            (product dims')+            (segBinOpLambda scan)+            arrs_flat -  sOp $ Imp.ErrorSync Imp.FenceLocal+    applyPostOp virt scan_arrs map_dests+      | severalElemsPerThread virt =+          blockCoverSegSpace virt space postOp+      | otherwise =+          -- In this case the ltids from computeScanInputs are still in scope.+          sWhen (isActive $ zip ltids dims) $ localOps threadOperations postOp+      where+        ltids' = map Imp.le64 ltids+        res = fmap resSubExp $ bodyResult $ lambdaBody post_lam+        postOp = do+          sComment "bind scan results to post lambda params" $+            forM_ (zip scan_pars scan_arrs) $ \(par, arr) ->+              copyDWIMFix (paramName par) [] (Var arr) ltids' -  let tmps_for_ops = chunks (map (length . segBinOpNeutral) ops) tmp_arrs-  case segVirt lvl of-    SegVirt -> virtCase dims' tmps_for_ops-    _ -> nonvirtCase dims' tmps_for_ops+          sComment "bind map results to post lambda params" $+            forM_ (zip map_pars map_dests) $ \(par, dest) ->+              forM_ dest $ \arr ->+                copyDWIMFix (paramName par) [] (Var arr) ltids'++          sComment "compute post op." $+            compileStms mempty (bodyStms $ lambdaBody post_lam) $+              sComment "write values" $+                forM_ (zip (patElems pat) res) $ \(pe, subexp) ->+                  copyDWIMFix (patElemName pe) ltids' subexp []++isEmptySpace :: SegSpace -> Imp.TExp Bool+isEmptySpace = (.==. 0) . product . map pe64 . segSpaceDims++-- | Compile a block-level 'SegRed'. Unless the index space is virtualised, the+-- kernel body is first run for the entire index space, with the results stashed+-- in shared memory, which is then reduced. A virtualised index space is instead+-- handled one block-sized chunk at a time - chunks of the /flattened/ space,+-- when there are several segments - so that we need no more shared memory than+-- there are threads in the block.+compileBlockRed ::+  Pat LetDecMem ->+  SegLevel ->+  SegSpace ->+  KernelBody GPUMem ->+  [SegBinOp GPUMem] ->+  InKernelGen ()+compileBlockRed pat lvl space body ops = do+  compileFlatId space++  sIf (isEmptySpace space) emptyCase $ do+    virt <- blockCoverVirt (segVirt lvl) space+    case (virt, dims') of+      (SegVirt, [dim']) -> chunkedCase dim'+      (SegVirt, _) -> segmentedChunkedCase+      (_, [dim']) -> stageWholeSpace (product dims') $ singleSegmentCase dim'+      _ -> stageWholeSpace (blockScanElems (product dims') red_ts) segmentedCase   where     (ltids, dims) = unzip $ unSegSpace space+    dims' = map pe64 dims     (red_pes, map_pes) = splitAt (segBinOpResults ops) $ patElems pat+    (body_red_res, body_map_res) =+      splitAt (segBinOpResults ops) $ bodyResult body+    red_ts = concatMap (lambdaReturnType . segBinOpLambda) ops+    forOps :: [a] -> [[a]]+    forOps = chunks $ map (length . segBinOpNeutral) ops -    virtCase [dim'] tmps_for_ops = do+    -- Shared memory for @elems@ values per reduction operand, declared with+    -- the given shape.+    sharedArrays elems shape =+      forM red_ts $ \t ->+        sharedStagingArray "red_arr" t elems (shape <> arrayShape t)++    -- Write the reduction results, read at @src@ in the staging arrays, to+    -- @dest@ in the destinations.+    saveResults dest tmps_for_ops src =+      sComment "Save result of reduction." $+        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->+          copyDWIM (patElemName pe) dest (Var arr) src++    -- The result of the chunks so far accumulates at @dest_is@ in the+    -- destination, so the carry is incorporated by folding the destination into+    -- the chunk's first element; the reduction or scan that follows propagates+    -- it through the rest of the chunk.+    incorporateCarry due dest_is tmps_for_ops = do       ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+      sComment "possibly incorporate carry" . localOps threadOperations $+        sWhen (due .&&. ltid .==. 0) $+          forM_ (zip3 ops tmps_for_ops (forOps red_pes)) $ \(op, tmps, pes) ->+            applyRenamedLambda+              (segBinOpLambda op)+              (map (,[DimFix 0]) tmps)+              ( map ((,dest_is) . Var . patElemName) pes+                  ++ map ((,[DimFix 0]) . Var) tmps+              )++    -- Run the segop body for the point of the index space that the thread's+    -- index variables are currently bound to, saving the reduction inputs at+    -- @is@ in the given @tmp_arrs@. Map results go straight to their+    -- destinations.+    runBody tmp_arrs is =+      compileStms mempty (bodyStms body) $ do+        forM_ (zip tmp_arrs body_red_res) $ \(dest, res) ->+          copyDWIMFix dest is (kernelResultSubExp res) []+        zipWithM_ (compileThreadResult space) map_pes body_map_res++    emptyCase = do+      -- This is the result if there are no chunks at all.+      sComment "set result to neutral element" $+        blockCoverSpace (map pe64 $ init dims) $ \is ->+          forM_ (zip red_pes $ concatMap segBinOpNeutral ops) $ \(pe, ne) ->+            copyDWIMFix (patElemName pe) is ne []++    -- Run the kernel body for the entire index space, staging the results in+    -- shared memory, then reduce them with the passed-in continuation.+    stageWholeSpace elems k = do+      tmp_arrs <- sharedArrays elems $ Shape dims+      blockCoverSegSpace (segVirt lvl) space $+        runBody tmp_arrs (map Imp.le64 ltids)+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty+      k $ forOps tmp_arrs++    -- A single segment, virtualised. We run the kernel body for one block-sized+    -- chunk at a time, reduce that chunk, and carry the result into the next+    -- chunk.+    chunkedCase dim' = do+      tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv+      ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+      -- A chunk is never larger than the block, nor than the index space+      -- itself, so that is all the shared memory we need.+      tmp_arrs <-+        sharedArrays (sMin64 dim' (pe64 (unCount tblock_size))) $+          Shape [unCount tblock_size]+      let tmps_for_ops = forOps tmp_arrs+          [ltid_name] = ltids+       blockChunkLoop (sExt32 dim') $ \chunk_start chunk_size -> do-        sComment "possibly incorporate carry" $-          sWhen (chunk_start .>. 0 .&&. ltid .==. 0) $-            forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->-              applyRenamedLambda-                (segBinOpLambda op)-                (map (,[DimFix $ sExt64 chunk_start]) tmps)-                ( map ((,[]) . Var . patElemName) red_pes-                    ++ map ((,[DimFix $ sExt64 chunk_start]) . Var) tmps-                )+        sComment "run the kernel body for this chunk" . localOps threadOperations $ do+          dPrimV_ ltid_name $ sExt64 chunk_start + sExt64 ltid+          sWhen (Imp.le64 ltid_name .<. dim') $+            runBody tmp_arrs [sExt64 ltid] -        sOp $ Imp.ErrorSync Imp.FenceLocal+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty -        forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do-          tmps_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) tmps-          blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps_chunks+        incorporateCarry (chunk_start .>. 0) [] tmps_for_ops -        sOp $ Imp.ErrorSync Imp.FenceLocal+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty -        sComment "Save result of reduction." $-          forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->-            copyDWIMFix (patElemName pe) [] (Var arr) [sExt64 chunk_start]+        forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->+          blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps +        sOp $ Imp.ErrorSync Imp.FenceLocal mempty++        saveResults [] tmps_for_ops [DimFix 0]++        -- Everyone must be done reading the result before the next chunk+        -- overwrites it.+        sOp $ Imp.Barrier Imp.FenceLocal++    -- Multiple segments, virtualised. We go through the flattened index space+    -- one block-sized chunk at a time: run the segop body for the chunk, scan+    -- the chunk with the segment flags, then let every thread whose element is+    -- the last of its segment write that segment's result. The running value of+    -- a segment that continues past the end of the chunk is carried into the+    -- next chunk.     ---    virtCase dims' tmps_for_ops = do-      dims_flat <- dPrimV "dims_flat" $ product dims'+    -- Chunking the flattened space rather than taking whole segments at a time+    -- keeps every thread busy no matter how the segment size relates to the+    -- block size, and needs no separate strategy for segments larger than the+    -- block.+    segmentedChunkedCase = do+      tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv+      ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+      let w = product dims'+          -- A chunk is never larger than the block, nor than the index space+          -- itself, so that is all the shared memory we need.+          chunk_elems = sMin64 w $ pe64 $ unCount tblock_size+          seg_flag = crossesSegment $ last dims'+      tmp_arrs <-+        sharedArrays (blockScanElems chunk_elems red_ts) $+          Shape [unCount tblock_size]+      let tmps_for_ops = forOps tmp_arrs -      forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do-        tmps_flat <- mapM (flattenArray (length dims') dims_flat) tmps-        virtualisedBlockScan-          (Just $ crossesSegment $ last dims')-          (sExt32 $ tvExp dims_flat)-          (segBinOpLambda op)-          tmps_flat+      blockChunkLoop (sExt32 w) $ \chunk_start chunk_size -> do+        flat_idx <- dPrimVE "flat_idx" $ sExt64 chunk_start + sExt64 ltid+        in_bounds <- dPrimVE "in_bounds" $ flat_idx .<. w+        dIndexSpace (zip ltids dims') flat_idx -      sOp $ Imp.ErrorSync Imp.FenceLocal+        sComment "run segred body for this chunk" . localOps threadOperations $+          sWhen in_bounds $+            runBody tmp_arrs [sExt64 ltid] -      sComment "Save result of reduction." $-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->-          copyDWIM-            (patElemName pe)-            []-            (Var arr)-            (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty -      sOp $ Imp.Barrier Imp.FenceLocal+        let (crosses_segment, shifted) =+              chunkSegFlags (Just seg_flag) chunk_start -    -- Nonsegmented case (or rather, a single segment) - this we can-    -- handle directly with a block-level reduction.-    nonvirtCase [dim'] tmps_for_ops = do+        -- Thread zero is at @chunk_start@, so its segment index is that of the+        -- segment the carry belongs to - the one the preceding chunk ended in.+        incorporateCarry+          (chunk_start .>. 0 .&&. bNot crosses_segment)+          (map (DimFix . Imp.le64) (init ltids))+          tmps_for_ops++        sOp $ Imp.ErrorSync Imp.FenceLocal mempty++        -- Scan the currente chunk.+        forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->+          blockScan shifted chunk_elems (tvExp chunk_size) (segBinOpLambda op) tmps++        sOp $ Imp.ErrorSync Imp.FenceLocal mempty++        -- A thread whose element is the last of its segment holds that+        -- segment's final result. The thread holding the last element of the+        -- chunk holds the running result of a segment that continues into the+        -- next chunk, which that chunk then reads back as its carry.+        ends_segment <-+          dPrimVE "ends_segment" $ Imp.le64 (last ltids) .==. last dims' - 1+        ends_chunk <-+          dPrimVE "ends_chunk" $ sExt64 ltid .==. tvExp chunk_size - 1+        localOps threadOperations $+          sWhen (in_bounds .&&. (ends_segment .||. ends_chunk)) $+            saveResults+              (map (DimFix . Imp.le64) (init ltids))+              tmps_for_ops+              [DimFix $ sExt64 ltid]++        -- Everyone must be done reading the chunk before the next one+        -- overwrites it, and the carry must be visible to the next chunk.+        fence <- fenceForArrays $ map patElemName red_pes+        sOp $ Imp.Barrier fence++    -- A single segment - this we can handle directly with a block-level+    -- reduction.+    singleSegmentCase dim' tmps_for_ops = do       forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->         blockReduce (sExt32 dim') (segBinOpLambda op) tmps-      sOp $ Imp.ErrorSync Imp.FenceLocal-      sComment "Save result of reduction." $-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->-          copyDWIMFix (patElemName pe) [] (Var arr) [0]-      sOp $ Imp.ErrorSync Imp.FenceLocal+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty+      saveResults [] tmps_for_ops [DimFix 0]+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty -    -- Segmented intra-block reductions are turned into (regular)-    -- segmented scans.  It is possible that this can be done+    -- Multiple segments, with the whole index space staged in shared memory.+    -- These are turned into (regular) segmented scans, of which we then pick+    -- the last element of each segment. It is possible that this can be done     -- better, but at least this approach is simple.-    nonvirtCase dims' tmps_for_ops = do-      -- blockScan operates on flattened arrays.  This does not-      -- involve copying anything; merely playing with the index-      -- function.+    segmentedCase tmps_for_ops = do+      -- The scans operate on flattened arrays. This does not involve copying+      -- anything; merely playing with the index function.       dims_flat <- dPrimV "dims_flat" $ product dims'        forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do@@ -559,19 +813,39 @@           (segBinOpLambda op)           tmps_flat -      sOp $ Imp.ErrorSync Imp.FenceLocal+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty -      sComment "Save result of reduction." $-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->-          copyDWIM-            (patElemName pe)-            []-            (Var arr)-            (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])+      saveResults [] tmps_for_ops lastInSegment        sOp $ Imp.Barrier Imp.FenceLocal-compileBlockOp pat (Inner (SegOp (SegHist lvl space _ kbody ops))) = do++    lastInSegment =+      map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1]++compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp+compileBlockOp pat (Alloc size space) =+  blockAlloc pat size space+compileBlockOp pat (Inner (SegOp op)) = do+  beforeBlockWrite+  compileBlockSegOp pat op+compileBlockOp pat _ =+  compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat++compileBlockSegOp :: Pat LetDecMem -> SegOp SegLevel GPUMem -> InKernelGen ()+compileBlockSegOp pat (SegMap lvl space _ body) = do   compileFlatId space++  blockCoverSegSpace (segVirt lvl) space $+    compileStms mempty (bodyStms body) $+      zipWithM_ (compileThreadResult space) (patElems pat) $+        bodyResult body+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty+compileBlockSegOp pat (SegScan lvl space _ body scans post_op) =+  compileBlockScan pat lvl space body scans post_op+compileBlockSegOp pat (SegRed lvl space _ body ops) =+  compileBlockRed pat lvl space body ops+compileBlockSegOp pat (SegHist lvl space _ kbody ops) = do+  compileFlatId space   let (ltids, dims) = unzip $ unSegSpace space    -- We don't need the red_pes, because it is guaranteed by our type@@ -613,9 +887,7 @@                   copyDWIMFix (paramName p) [] v is                 do_op (bin_is ++ is) -  sOp $ Imp.ErrorSync Imp.FenceLocal-compileBlockOp pat _ =-  compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty  -- | The operations for block-level kernels. blockOperations :: Operations GPUMem KernelEnv Imp.KernelOp@@ -744,7 +1016,7 @@ -- | Various useful precomputed information for block-level SegOps. data Precomputed = Precomputed   { pcSegOpSizes :: SegOpSizes,-    pcChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)+    pcFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32)   }  -- | Find the sizes of nested parallelism in a t'SegOp' body.@@ -773,13 +1045,16 @@ precomputeConstants :: Count BlockSize (Imp.TExp Int64) -> Stms GPUMem -> CallKernelGen Precomputed precomputeConstants tblock_size stms = do   let sizes = segOpSizes stms-  iters_map <- M.fromList <$> mapM mkMap (S.toList sizes)-  pure $ Precomputed sizes iters_map+  -- Constant sizes are left to the kernel; see Note [Peeled block loops].+  full_chunks_map <-+    M.fromList <$> mapM mkMap (filter (any (isJust . subExpVar)) (S.toList sizes))+  pure $ Precomputed sizes full_chunks_map   where     mkMap dims = do       let n = product $ map Imp.pe64 dims-      num_chunks <- dPrimVE "num_chunks" $ sExt32 $ n `ceilDiv` unCount tblock_size-      pure (dims, num_chunks)+      num_full_chunks <-+        dPrimVE "num_full_chunks" $ sExt32 $ n `quot` unCount tblock_size+      pure (dims, num_full_chunks)  -- | Make use of various precomputed constants. precomputedConstants :: Precomputed -> InKernelGen a -> InKernelGen a@@ -791,7 +1066,7 @@           { kernelConstants =               (kernelConstants env)                 { kernelLocalIdMap = new_ids,-                  kernelChunkItersMap = pcChunkItersMap pre+                  kernelFullChunksMap = pcFullChunksMap pre                 }           }   localEnv f m
src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs view
@@ -196,21 +196,19 @@       [Imp.TExp Int64] -> InKernelGen ()     ) prepareAtomicUpdateGlobal l segments dests slug =-  -- We need a separate lock array if the operators are not all of a-  -- particularly simple form that permits pure atomic operations.+  -- We need locks if the operators are not all of a particularly+  -- simple form that permits pure atomic operations.   case (l, slugAtomicUpdate slug) of     (_, AtomicPrim f) -> pure (l, f (Space "global") dests)     (_, AtomicCAS f) -> pure (l, f (Space "global") dests)     (Just l', AtomicLocking f) -> pure (l, f l' (Space "global") dests)     (Nothing, AtomicLocking f) -> do-      -- The number of locks used here is too low, but since we are-      -- currently forced to inline a huge list, I'm keeping it down-      -- for now.  Some quick experiments suggested that it has little-      -- impact anyway (maybe the locking case is just too slow).-      ---      -- A fun solution would also be to use a simple hashing-      -- algorithm to ensure good distribution of locks.-      let num_locks = 100151+      -- The lock for a bucket is chosen by its flat index modulo the+      -- number of locks.  More locks, or a simple hashing algorithm+      -- that distributes the buckets better, might reduce contention,+      -- but quick experiments suggested that it has little impact+      -- (maybe the locking case is just too slow).+      let num_locks = Imp.bookkeepingSize Imp.BookkeepingLocks           dims =             map pe64 $               shapeDims segments@@ -218,7 +216,7 @@                 ++ [tvSize (slugNumSubhistos slug)]                 ++ shapeDims (histShape (slugOp slug)) -      locks <- genZeroes "hist_locks" num_locks+      locks <- bookkeepingArray Imp.BookkeepingLocks       let l' = Locking locks 0 1 0 (pure . (`rem` fromIntegral num_locks) . flattenIndex dims)       pure (Just l', f l' (Space "global") dests) @@ -781,7 +779,7 @@                           copyDWIMFix (paramName p) [] v is                         do_op (bucket_is ++ is) -        sOp $ Imp.ErrorSync Imp.FenceGlobal+        sOp $ Imp.ErrorSync Imp.FenceGlobal mempty          sComment "Compact the multiple shared memory subhistograms to result in global memory" $           onSlugs $ \slug dests hist_H_chk histo_dims _histo_size bins_per_thread -> do
src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs view
@@ -9,13 +9,8 @@ --   kernel, with the final threadblock to write a result (tracked via --   an atomic counter) performing the final reduction as well. ----- * Instead of depending on storage layout transformations to handle---   non-commutative reductions efficiently, we slide a---   @tblocksize@-sized window over the input, and perform a parallel---   reduction for each window.  This sacrifices the notion of---   efficient sequentialisation, but is sometimes faster and---   definitely simpler and more predictable (and uses less auxiliary---   storage).+-- * We have various strategies for handling non-commutative reductions. See+--   Note [Stage one strategies]. -- -- For segmented reductions we use the approach from "Strategies for -- Regular Segmented Reductions on GPU" (FHPC '17).  This involves@@ -30,10 +25,8 @@ --   value per segment. -- --   Each thread /can/ read multiple elements, which will greatly---   increase performance; however, if the reduction is---   non-commutative we will have to use a less efficient traversal---   (with interim block-wide reductions) to enable coalesced memory---   accesses, just as in the non-segmented case.+--   increase performance.  Non-commutative reductions use the same+--   strategies as in the non-segmented case. -- -- * Small: is used to let each block process *multiple* segments --   within a block. We will only use this approach when we can@@ -41,13 +34,6 @@ --   cases, we would allocate a /whole/ block per segment with the --   large strategy, but at most 50% of the threads in the block would --   have any element to read, which becomes highly inefficient.------ An optimization specfically targeted at non-segmented and large-segments--- segmented reductions with non-commutative is made: The stage one main loop is--- essentially stripmined by a factor *chunk*, inserting collective copies via--- shared memory of each reduction parameter going into the intra-block (partial)--- reductions. This saves a factor *chunk* number of intra-block reductions at--- the cost of some overhead in collective copies. module Futhark.CodeGen.ImpGen.GPU.SegRed   ( compileSegRed,     compileSegRed',@@ -56,7 +42,6 @@ where  import Control.Monad-import Data.List (zip4) import Data.Map qualified as M import Data.Maybe import Futhark.CodeGen.ImpCode.GPU qualified as Imp@@ -72,6 +57,11 @@ forM2_ :: (Monad m) => [a] -> [b] -> (a -> b -> m c) -> m () forM2_ xs ys f = forM_ (zip xs ys) (uncurry f) +-- | The total size of values of the given types in 32-bit words, where+-- smaller types occupy a full word.+primWords :: [PrimType] -> Int+primWords = sum . map ((`div` 4) . max 4 . primByteSize)+ -- | Given available register and a list of parameter types, compute -- the largest available chunk size given the parameters for which we -- want chunking and the available resources.@@ -91,7 +81,7 @@       sizes = map primByteSize types'        sum_sizes = sum sizes-      sum_sizes' = sum (map (sMax64 4 . primByteSize) types') `quot` 4+      sum_sizes' = fromIntegral $ primWords types'       max_size = maximum sizes        mem_constraint = max k_mem sum_sizes `quot` max_size@@ -153,6 +143,45 @@   all primType (lambdaReturnType $ segBinOpLambda segbinop)     && shapeRank (segBinOpShape segbinop) == 0 +-- | How the first stage of a reduction distributes the input among the threads.+-- For elaboration on the strategies, see Note [Stage one strategies].+data StageOne+  = -- | Thread @t@ of @T@ handles elements @t@, @t+T@, @t+2T@, and so on, not+    -- combining the per-thread results until all input has been processed.+    -- Requires a commutative operator.+    StageOneStrided+  | -- | Each block handles a sequence of contiguous tiles, using collective+    -- copies through shared memory and a block-wide reduction per tile.+    -- Requires primitive (non-array) operands.+    StageOneTiled+  | -- | Each thread handles a contiguous range of the input.+    StageOneContiguous+  | -- | Each block slides a window of @tblock_size@ elements across a+    -- contiguous range of the input, with a block-wide reduction per window.+    StageOneWindow+  deriving (Eq)++-- | The largest total size of the reduction operands, in 32-bit words, for+-- which we use 'StageOneTiled'. Heuristically chosen. It is difficult to make+-- this a tuning parameter, as the strategy is a compile-time choice.+maxTiledOperandWords :: Int+maxTiledOperandWords = 16++stageOneStrategy :: [SegBinOp GPUMem] -> StageOne+stageOneStrategy segbinops+  | all ((== Commutative) . segBinOpComm) segbinops =+      StageOneStrided+  | not $ all isPrimSegBinOp segbinops =+      StageOneWindow+  | operand_words <= maxTiledOperandWords =+      StageOneTiled+  | otherwise =+      StageOneContiguous+  where+    operand_words =+      primWords . map elemType $+        concatMap (lambdaReturnType . segBinOpLambda) segbinops+ -- | Like 'compileSegRed', but where the body is a monadic action. compileSegRed' ::   Pat LetDecMem ->@@ -175,19 +204,19 @@         (compileReduction (chunk_v, chunk_const) largeSegmentsReduction)   where     compileReduction chunk f =-      f pat num_tblocks tblock_size chunk space segbinops map_body_cont+      f pat num_tblocks tblock_size strategy chunk space segbinops map_body_cont      param_types = map paramType $ concatMap paramOf segbinops      num_tblocks = gridNumBlocks grid     tblock_size = gridBlockSize grid -    chunk_const =-      if Noncommutative `elem` map segBinOpComm segbinops-        && all isPrimSegBinOp segbinops-        then getRedChunkSize param_types-        else Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)+    strategy = stageOneStrategy segbinops +    chunk_const+      | strategy == StageOneTiled = getRedChunkSize param_types+      | otherwise = Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)+ -- | Prepare intermediate arrays for the reduction.  Prim-typed -- arguments go in shared memory (so we need to do the allocation of -- those arrays inside the kernel), while array-typed arguments go in@@ -198,14 +227,14 @@ -- For more info about the intermediate arrays used for the different reduction -- kernels, see note [IntermArrays]. makeIntermArrays ::+  StageOne ->   Imp.TExp Int64 ->   SubExp ->   SubExp ->   [SegBinOp GPUMem] ->   InKernelGen [SegRedIntermediateArrays]-makeIntermArrays tblock_id tblock_size chunk segbinops-  | Noncommutative <- mconcat (map segBinOpComm segbinops),-    all isPrimSegBinOp segbinops =+makeIntermArrays strategy tblock_id tblock_size chunk segbinops+  | strategy == StageOneTiled =       noncommPrimSegRedInterms   | otherwise =       generalSegRedInterms tblock_id tblock_size segbinops@@ -282,10 +311,10 @@ -- | Arrays for storing block results. -- -- The block-result arrays have an extra dimension because they are--- also used for keeping vectorised accumulators for first-stage--- reduction, if necessary.  If necessary, this dimension has size--- tblock_size, and otherwise 1.  When actually storing block results,--- the first index is set to 0.+-- also used for keeping the per-thread accumulators for first-stage+-- reduction of non-primitive operands.  In that case, this dimension+-- has size tblock_size, and otherwise 1.  When actually storing block+-- results, the first index is set to 0. groupResultArrays ::   SubExp ->   SubExp ->@@ -308,6 +337,7 @@   Pat LetDecMem ->   Count NumBlocks SubExp ->   Count BlockSize SubExp ->+  StageOne ->   (TV Int64, Imp.KernelConstExp) ->   SegSpace ->   [SegBinOp GPUMem] ->@@ -315,7 +345,7 @@   CallKernelGen ()  nonsegmentedReduction :: DoCompileSegRed-nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do+nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do   let (gtids, dims) = unzip $ unSegSpace space       chunk = tvExp chunk_v       num_tblocks_se = unCount num_tblocks@@ -324,7 +354,7 @@       global_tid = Imp.le64 $ segFlat space       n = pe64 $ last dims -  counters <- genZeroes "counters" $ length segbinops+  counters <- bookkeepingArray Imp.BookkeepingCounters    reds_block_res_arrs <- groupResultArrays num_tblocks_se tblock_size_se segbinops @@ -341,7 +371,7 @@     let ltid = kernelLocalThreadId constants     let tblock_id = kernelBlockId constants -    interms <- makeIntermArrays (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops+    interms <- makeIntermArrays strategy (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops     sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared"      -- Since this is the nonsegmented case, all outer segment IDs must@@ -355,6 +385,7 @@         zip3 segbinops interms reds_block_res_arrs     new_lambdas <-       reductionStageOne+        strategy         gtids         n         global_tid@@ -369,22 +400,18 @@             (map (length . segBinOpNeutral) segbinops)             segred_pes -    forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $-      \(pes, slug, new_lambda, i) ->-        reductionStageTwo-          pes-          tblock_id-          [0]-          0-          (sExt64 $ kernelNumBlocks constants)-          slug-          new_lambda-          counters-          sync_arr-          (fromInteger i)+    reductionStageTwo+      tblock_id+      [0]+      0+      (sExt64 $ kernelNumBlocks constants)+      (zip3 segred_pess slugs new_lambdas)+      counters+      sync_arr+      0  smallSegmentsReduction :: DoCompileSegRed-smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ space segbinops map_body_cont = do+smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ _ space segbinops map_body_cont = do   let (gtids, dims) = unzip $ unSegSpace space       dims' = map pe64 dims       segment_size = last dims'@@ -453,7 +480,7 @@           in_bounds           out_of_bounds -      sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier.       let crossesSegment from to =             (sExt64 to - sExt64 from) .>. (sExt64 to `rem` segment_size)       sWhen (segment_size .>. 0) $@@ -495,7 +522,7 @@       sOp $ Imp.Barrier Imp.FenceLocal  largeSegmentsReduction :: DoCompileSegRed-largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do+largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do   let (gtids, dims) = unzip $ unSegSpace space       dims' = map pe64 dims       num_segments = product $ init dims'@@ -537,13 +564,13 @@   -- However, we exploit the fact that the number of segments being   -- reduced at any point in time is limited by the number of   -- threadblocks. If we bound the number of threadblocks, we can get away-  -- with using that many counters.  FIXME: Is this limit checked-  -- anywhere?  There are other places in the compiler that will fail-  -- if the block count exceeds the maximum block size, which is at-  -- most 1024 anyway.-  let counters_per_op = 1024-  counters <--    genZeroes "counters" $ fromIntegral $ length segbinops * counters_per_op+  -- with using that many counters, so we use the bookkeeping counters (see+  -- 'Imp.Bookkeeping'), indexed by the segment modulo their number.  FIXME:+  -- Is this limit checked anywhere?  There are other places in the compiler+  -- that will fail if the block count exceeds the maximum block size, which+  -- is at most 1024 anyway.+  counters <- bookkeepingArray Imp.BookkeepingCounters+  let num_counters = Imp.bookkeepingSize Imp.BookkeepingCounters    let attrs =         (defKernelAttrs num_tblocks tblock_size)@@ -555,7 +582,7 @@     let tblock_id = sExt64 $ kernelBlockId constants         ltid = kernelLocalThreadId constants -    interms <- makeIntermArrays tblock_id tblock_size_se (tvSize chunk_v) segbinops+    interms <- makeIntermArrays strategy tblock_id tblock_size_se (tvSize chunk_v) segbinops     sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared"      -- We probably do not have enough actual threadblocks to cover the@@ -583,6 +610,7 @@           zip3 segbinops interms reds_block_res_arrs       new_lambdas <-         reductionStageOne+          strategy           gtids           n           global_tid@@ -598,23 +626,15 @@               segred_pes            multiple_blocks_per_segment =-            forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $-              \(pes, slug, new_lambda, i) -> do-                let counter_idx =-                      fromIntegral (i * counters_per_op)-                        + flat_segment_id-                          `rem` fromIntegral counters_per_op-                reductionStageTwo-                  pes-                  virttblock_id-                  (map Imp.le64 segment_gtids)-                  first_block_for_segment-                  blocks_per_segment-                  slug-                  new_lambda-                  counters-                  sync_arr-                  counter_idx+            reductionStageTwo+              virttblock_id+              (map Imp.le64 segment_gtids)+              first_block_for_segment+              blocks_per_segment+              (zip3 segred_pess slugs new_lambdas)+              counters+              sync_arr+              (flat_segment_id `rem` fromIntegral num_counters)            one_block_per_segment =             sComment "first thread in block saves final result to memory" $@@ -633,7 +653,9 @@   { slugOp :: SegBinOp GPUMem,     -- | Intermediate arrays needed for the given reduction.     slugInterms :: SegRedIntermediateArrays,-    -- | Place(s) to store block accumulator(s) in stage 1 reduction.+    -- | Per-thread accumulator(s) for the stage one reduction.  After+    -- each block-wide reduction, the accumulator(s) of thread 0 hold+    -- the result for the block.     slugAccs :: [(VName, [Imp.TExp Int64])],     -- | Global memory destination(s) for the final result(s) for this     -- particular reduction.@@ -652,10 +674,10 @@     mkAcc p block_res_arr       | Prim t <- paramType p,         shapeRank (segBinOpShape op) == 0 = do-          block_res_acc <- dPrimS (baseName (paramName p) <> "_block_res_acc") t-          pure (block_res_acc, [])+          thread_acc <- dPrimS (baseName (paramName p) <> "_thread_acc") t+          pure (thread_acc, [])       -- if this is a non-primitive reduction, the global mem result array will-      -- double as accumulator.+      -- double as the per-thread accumulators.       | otherwise =           pure (block_res_arr, [sExt64 ltid, sExt64 tblock_id]) @@ -674,9 +696,6 @@ slugShape :: SegBinOpSlug -> Shape slugShape = segBinOpShape . slugOp -slugsComm :: [SegBinOpSlug] -> Commutativity-slugsComm = mconcat . map (segBinOpComm . slugOp)- slugSplitParams :: SegBinOpSlug -> ([LParam GPUMem], [LParam GPUMem]) slugSplitParams slug = splitAt (length (slugNeutral slug)) $ slugParams slug @@ -689,7 +708,43 @@ slugCollCopyArrs :: SegBinOpSlug -> [VName] slugCollCopyArrs = collCopyArrs . slugInterms +-- | When 'sequentialStageOneBody' performs the block-wide reduction.+data BlockReduceAt = ReducePerIteration | ReduceAfterLoop+  deriving (Eq)++-- | Each thread sequentially reduces @q@ elements, with block-wide reductions+-- as determined by 'BlockReduceAt'. The function maps the iteration number to+-- the index of the element.+sequentialStageOneBody ::+  [SegBinOpSlug] ->+  DoSegBody ->+  TV Int64 ->+  Imp.TExp Int64 ->+  Imp.TExp Int64 ->+  BlockReduceAt ->+  InKernelGen () ->+  (Imp.TExp Int64 -> Imp.TExp Int64) ->+  InKernelGen ()+sequentialStageOneBody slugs body_cont glb_ind_var q n reduce_at doBlockReduce index = do+  sFor "i" q $ \i -> do+    glb_ind_var <-- index i++    sWhen (tvExp glb_ind_var .<. n) $+      sComment "apply map function(s)" $+        body_cont $ \all_red_res -> do+          let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res++          forM2_ slugs maps_res $ \slug map_res ->+            sLoopNest (slugShape slug) $ \vec_is ->+              applyRedOp slug vec_is map_res++    when (reduce_at == ReducePerIteration) doBlockReduce++  sOp $ Imp.ErrorSync Imp.FenceLocal mempty+  when (reduce_at == ReduceAfterLoop) doBlockReduce+ reductionStageOne ::+  StageOne ->   [VName] ->   Imp.TExp Int64 ->   Imp.TExp Int64 ->@@ -699,39 +754,39 @@   [SegBinOpSlug] ->   DoSegBody ->   InKernelGen [Lambda GPUMem]-reductionStageOne gtids n global_tid q chunk threads_per_segment slugs body_cont = do+reductionStageOne strategy gtids n global_tid q chunk threads_per_segment slugs body_cont = do   constants <- kernelConstants <$> askEnv   let glb_ind_var = mkTV (last gtids)       ltid = sExt64 $ kernelLocalThreadId constants    dScope Nothing $ scopeOfLParams $ concatMap slugParams slugs -  sComment "ne-initialise the outer (per-block) accumulator(s)" $ do+  sComment "ne-initialise the per-thread accumulator(s)" $ do     forM_ slugs $ \slug ->       forM2_ (slugAccs slug) (slugNeutral slug) $ \(acc, acc_is) ne ->         sLoopNest (slugShape slug) $ \vec_is ->           copyDWIMFix acc (acc_is ++ vec_is) ne []    new_lambdas <- mapM (renameLambda . slugLambda) slugs-  let tblock_size = sExt32 $ kernelBlockSize constants   let doBlockReduce =         forM2_ slugs new_lambdas $ \slug new_lambda -> do           let accs = slugAccs slug           let params = slugParams slug+          let tblock_size = sExt32 $ kernelBlockSize constants           sLoopNest (slugShape slug) $ \vec_is -> do             let block_red_arrs = slugBlockRedArrs slug-            sComment "store accs. prims go in lmem; non-prims in params (in global mem)" $+            sComment "store reduction accs" $               forM_ (zip3 block_red_arrs accs params) $                 \(arr, (acc, acc_is), p) ->                   if isPrimParam p                     then copyDWIMFix arr [ltid] (Var acc) (acc_is ++ vec_is)                     else copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is) -            sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.+            sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier.             blockReduce tblock_size new_lambda block_red_arrs             sOp $ Imp.Barrier Imp.FenceLocal -            sComment "thread 0 updates per-block acc(s); rest reset to ne" $ do+            sComment "thread 0 keeps the block result in its accumulator; rest reset to neutral" $ do               sIf                 (ltid .==. 0)                 ( forM2_ accs (lambdaParams new_lambda) $@@ -743,8 +798,8 @@                       copyDWIMFix acc (acc_is ++ vec_is) ne []                 ) -  case (slugsComm slugs, all (isPrimSegBinOp . slugOp) slugs) of-    (Noncommutative, True) ->+  case strategy of+    StageOneTiled ->       noncommPrimParamsStageOneBody         slugs         body_cont@@ -754,74 +809,43 @@         n         chunk         doBlockReduce-    _ ->-      generalStageOneBody-        slugs-        body_cont-        glb_ind_var-        global_tid-        q-        n-        threads_per_segment-        doBlockReduce+    StageOneStrided ->+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $+        \i -> global_tid + threads_per_segment * i+    StageOneContiguous ->+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $+        \i -> global_tid * q + i+    StageOneWindow -> do+      let window_size = kernelBlockSize constants+      block_base_offset <-+        dPrimVE "block_base_offset" $ (global_tid `quot` window_size) * q * window_size+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReducePerIteration doBlockReduce $+        \i -> block_base_offset + i * window_size + ltid    pure new_lambdas -generalStageOneBody ::-  [SegBinOpSlug] ->-  DoSegBody ->-  TV Int64 ->-  Imp.TExp Int64 ->-  Imp.TExp Int64 ->-  Imp.TExp Int64 ->-  Imp.TExp Int64 ->-  InKernelGen () ->+-- | Combine the accumulators of a reduction with the given next values. The+-- vector indexes select the elements of a vectorised reduction, and are+-- appended to the indexes of both.+applyRedOp ::+  SegBinOpSlug ->+  [Imp.TExp Int64] ->+  [(SubExp, [Imp.TExp Int64])] ->   InKernelGen ()-generalStageOneBody slugs body_cont glb_ind_var global_tid q n threads_per_segment doBlockReduce = do-  let is_comm = slugsComm slugs == Commutative--  constants <- kernelConstants <$> askEnv-  let tblock_size = kernelBlockSize constants-      ltid = sExt64 $ kernelLocalThreadId constants--  -- this block's id within its designated segment, and this block's initial-  -- global offset.-  tblock_id_in_segment <- dPrimVE "tblock_id_in_segment" $ global_tid `quot` tblock_size-  block_base_offset <- dPrimVE "block_base_offset" $ tblock_id_in_segment * q * tblock_size--  sFor "i" q $ \i -> do-    block_offset <- dPrimVE "block_offset" $ block_base_offset + i * tblock_size-    glb_ind_var-      <-- if is_comm-        then global_tid + threads_per_segment * i-        else block_offset + ltid--    sWhen (tvExp glb_ind_var .<. n) $-      sComment "apply map function(s)" $-        body_cont $ \all_red_res -> do-          let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res--          forM2_ slugs maps_res $ \slug map_res ->-            sLoopNest (slugShape slug) $ \vec_is -> do-              let (acc_params, next_params) = slugSplitParams slug-              sComment "load accumulator(s)" $-                forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->-                  copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)-              sComment "load next value(s)" $-                forM2_ next_params map_res $ \p (res, res_is) ->-                  copyDWIMFix (paramName p) [] res (res_is ++ vec_is)-              sComment "apply reduction operator(s)"-                $ compileStms mempty (bodyStms $ slugBody slug)-                $ sComment "store in accumulator(s)"-                $ forM2_-                  (slugAccs slug)-                  (map resSubExp $ bodyResult $ slugBody slug)-                $ \(acc, acc_is) se ->-                  copyDWIMFix acc (acc_is ++ vec_is) se []--    unless is_comm doBlockReduce-  sOp $ Imp.ErrorSync Imp.FenceLocal-  when is_comm doBlockReduce+applyRedOp slug vec_is next_values = do+  let (acc_params, next_params) = slugSplitParams slug+  sComment "load accumulator(s)" $+    forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->+      copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)+  sComment "load next value(s)" $+    forM2_ next_params next_values $ \p (v, v_is) ->+      copyDWIMFix (paramName p) [] v (v_is ++ vec_is)+  sComment "apply reduction operator(s)" $+    compileStms mempty (bodyStms $ slugBody slug) $+      sComment "store in accumulator(s)" $+        forM2_ (slugAccs slug) (map resSubExp $ bodyResult $ slugBody slug) $+          \(acc, acc_is) se ->+            copyDWIMFix acc (acc_is ++ vec_is) se []  noncommPrimParamsStageOneBody ::   [SegBinOpSlug] ->@@ -870,7 +894,7 @@                 copyDWIMFix priv_chunk [k] ne []         ) -    sOp $ Imp.ErrorSync Imp.FenceLocal+    sOp $ Imp.ErrorSync Imp.FenceLocal mempty      sComment "effectualize collective copies in shared memory" $ do       forM_ slugs $ \slug -> do@@ -890,58 +914,42 @@            sOp $ Imp.Barrier Imp.FenceLocal -    sComment "per-thread sequential reduction of private chunk(s)" $ do+    sComment "per-thread sequential reduction of private chunk(s)" $       chunkLoop $ \k ->-        forM_ slugs $ \slug -> do-          let accs = map fst $ slugAccs slug-          let (acc_ps, next_ps) = slugSplitParams slug-          let ps_accs_chunks = zip4 acc_ps next_ps accs (slugPrivChunks slug)--          sComment "load params for all reductions" $ do-            forM_ ps_accs_chunks $ \(acc_p, next_p, acc, priv_chunk) -> do-              copyDWIM (paramName acc_p) [] (Var acc) []-              copyDWIMFix (paramName next_p) [] (Var priv_chunk) [k]--          sComment "apply reduction operator(s)" $ do-            let binop_ress = map resSubExp $ bodyResult $ slugBody slug-            compileStms mempty (bodyStms $ slugBody slug) $-              forM2_ accs binop_ress $ \acc binop_res ->-                copyDWIM acc [] binop_res []+        forM_ slugs $ \slug ->+          applyRedOp slug [] [(Var c, [k]) | c <- slugPrivChunks slug]     doLMemBlockReduce-  sOp $ Imp.ErrorSync Imp.FenceLocal+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty +-- | The second stage of a reduction where each segment is handled by more than+-- one block. Each block writes its partial results for all the operators, and+-- the last block of the segment to do so combines them into the final results.+-- The blocks find out which of them is last by incrementing a counter for the+-- segment, which the last block then resets. reductionStageTwo ::-  [PatElem LetDecMem] ->   Imp.TExp Int32 ->   [Imp.TExp Int64] ->   Imp.TExp Int64 ->   Imp.TExp Int64 ->-  SegBinOpSlug ->-  Lambda GPUMem ->+  [([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem)] ->   VName ->   VName ->   Imp.TExp Int64 ->   InKernelGen ()-reductionStageTwo segred_pes tblock_id segment_gtids first_block_for_segment blocks_per_segment slug new_lambda counters sync_arr counter_idx = do+reductionStageTwo tblock_id segment_gtids first_block_for_segment blocks_per_segment reds counters sync_arr counter_idx = do   constants <- kernelConstants <$> askEnv    let ltid32 = kernelLocalThreadId constants-      ltid = sExt64 ltid32-      tblock_size = kernelBlockSize constants -  let (acc_params, next_params) = slugSplitParams slug-      nes = slugNeutral slug-      red_arrs = slugBlockRedArrs slug-      block_res_arrs = blockResArrs slug-   old_counter <- dPrim "old_counter"   (counter_mem, _, counter_offset) <- fullyIndexArray counters [counter_idx]-  sComment "first thread in block saves block result to global memory" $+  sComment "first thread in block saves block results to global memory" $     sWhen (ltid32 .==. 0) $ do-      forM_ (take (length nes) $ zip block_res_arrs (slugAccs slug)) $ \(v, (acc, acc_is)) ->-        writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is-      -- Increment the counter, thus stating that our result is-      -- available.+      forM_ reds $ \(_, slug, _) ->+        forM_ (take (length (slugNeutral slug)) $ zip (blockResArrs slug) (slugAccs slug)) $+          \(v, (acc, acc_is)) ->+            writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is+      -- Increment the counter, thus declaring that our results are available.       sOp         $ Imp.Atomic DefaultSpace         $ Imp.AtomicAdd@@ -950,8 +958,8 @@           counter_mem           counter_offset         $ untyped (1 :: Imp.TExp Int32)-      -- Now check if we were the last block to write our result.  If-      -- so, it is our responsibility to produce the final result.+      -- Now check if we were the last block to write our results.  If+      -- so, it is our responsibility to produce the final results.       sWrite sync_arr [0] $ untyped $ tvExp old_counter .==. sExt32 (blocks_per_segment - 1)    sOp $ Imp.Barrier Imp.FenceGlobal@@ -959,7 +967,7 @@   is_last_block <- dPrim "is_last_block"   copyDWIMFix (tvVar is_last_block) [] (Var sync_arr) [0]   sWhen (tvExp is_last_block) $ do-    -- The final block has written its result (and it was+    -- The final block has written its results (and it was     -- us!), so read in all the block results and perform the     -- final stage of the reduction.  But first, we reset the     -- counter so it is ready for next time.  This is done@@ -972,74 +980,93 @@             untyped $               sExt32 (negate blocks_per_segment) -    sLoopNest (slugShape slug) $ \vec_is -> do-      unless (null $ slugShape slug) $-        sOp (Imp.Barrier Imp.FenceLocal)+    mapM_ (finalReduction segment_gtids first_block_for_segment blocks_per_segment) reds -      -- There is no guarantee that the number of threadblocks for the-      -- segment is less than the threadblock size, so each thread may-      -- have to read multiple elements.  We do this in a sequential-      -- way that may induce non-coalesced accesses, but the total-      -- number of accesses should be tiny here.-      ---      -- TODO: here we *could* insert a collective copy of the num_tblocks-      -- per-block results. However, it may not be beneficial, since num_tblocks-      -- is not necessarily larger than tblock_size, meaning the number of-      -- uncoalesced reads here may be insignificant. In fact, if we happen to-      -- have a num_tblocks < tblock_size, then the collective copy would add-      -- unnecessary overhead. Also, this code is only executed by a single-      -- block.-      sComment "read in the per-block-results" $ do-        read_per_thread <--          dPrimVE "read_per_thread" $-            blocks_per_segment `ceilDiv` sExt64 tblock_size+-- | Combine the partial results that the blocks of a segment have written for+-- one operator. Executed by a single block.+finalReduction ::+  [Imp.TExp Int64] ->+  Imp.TExp Int64 ->+  Imp.TExp Int64 ->+  ([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem) ->+  InKernelGen ()+finalReduction segment_gtids first_block_for_segment blocks_per_segment (segred_pes, slug, new_lambda) = do+  constants <- kernelConstants <$> askEnv -        forM2_ acc_params nes $ \p ne ->-          copyDWIM (paramName p) [] ne []+  let ltid32 = kernelLocalThreadId constants+      ltid = sExt64 ltid32+      tblock_size = kernelBlockSize constants -        sFor "i" read_per_thread $ \i -> do-          block_res_id <--            dPrimVE "block_res_id" $-              ltid * read_per_thread + i-          index_of_block_res <--            dPrimVE "index_of_block_res" $-              first_block_for_segment + block_res_id+  let (acc_params, next_params) = slugSplitParams slug+      nes = slugNeutral slug+      red_arrs = slugBlockRedArrs slug+      block_res_arrs = blockResArrs slug -          sWhen (block_res_id .<. blocks_per_segment) $ do-            forM2_ next_params block_res_arrs $-              \p block_res_arr ->-                copyDWIMFix-                  (paramName p)-                  []-                  (Var block_res_arr)-                  ([0, index_of_block_res] ++ vec_is)+  sLoopNest (slugShape slug) $ \vec_is -> do+    unless (null $ slugShape slug) $+      sOp (Imp.Barrier Imp.FenceLocal) -            compileStms mempty (bodyStms $ slugBody slug) $-              forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->-                copyDWIMFix (paramName p) [] se []+    -- There is no guarantee that the number of threadblocks for the segment is+    -- less than the threadblock size, so each thread may have to read multiple+    -- elements. We do this in a sequential way that may induce non-coalesced+    -- accesses, but the total number of accesses should be tiny here.+    --+    -- TODO: here we *could* insert a collective copy of the num_tblocks+    -- per-block results. However, it may not be beneficial, since num_tblocks+    -- is not necessarily larger than tblock_size, meaning the number of+    -- uncoalesced reads here may be insignificant. In fact, if we happen to+    -- have a num_tblocks < tblock_size, then the collective copy would add+    -- unnecessary overhead. Also, this code is only executed by a single block.+    sComment "read in the per-block-results" $ do+      read_per_thread <-+        dPrimVE "read_per_thread" $+          blocks_per_segment `ceilDiv` sExt64 tblock_size -      forM2_ acc_params red_arrs $ \p arr ->-        when (isPrimParam p) $-          copyDWIMFix arr [ltid] (Var $ paramName p) []+      forM2_ acc_params nes $ \p ne ->+        copyDWIM (paramName p) [] ne [] -      sOp $ Imp.ErrorSync Imp.FenceLocal+      sFor "i" read_per_thread $ \i -> do+        block_res_id <-+          dPrimVE "block_res_id" $+            ltid * read_per_thread + i+        index_of_block_res <-+          dPrimVE "index_of_block_res" $+            first_block_for_segment + block_res_id -      sComment "reduce the per-block results" $ do-        blockReduce (sExt32 tblock_size) new_lambda red_arrs+        sWhen (block_res_id .<. blocks_per_segment) $ do+          forM2_ next_params block_res_arrs $ \p block_res_arr ->+            copyDWIMFix+              (paramName p)+              []+              (Var block_res_arr)+              ([0, index_of_block_res] ++ vec_is) -        sComment "and back to memory with the final result" $-          sWhen (ltid32 .==. 0) $-            forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->-              copyDWIMFix-                (patElemName pe)-                (segment_gtids ++ vec_is)-                (Var $ paramName p)-                []+          compileStms mempty (bodyStms $ slugBody slug) $+            forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->+              copyDWIMFix (paramName p) [] se [] +    forM2_ acc_params red_arrs $ \p arr ->+      when (isPrimParam p) $+        copyDWIMFix arr [ltid] (Var $ paramName p) []++    sOp $ Imp.ErrorSync Imp.FenceLocal mempty++    sComment "reduce the per-block results" $ do+      blockReduce (sExt32 tblock_size) new_lambda red_arrs++      sComment "and back to memory with the final result" $+        sWhen (ltid32 .==. 0) $+          forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->+            copyDWIMFix+              (patElemName pe)+              (segment_gtids ++ vec_is)+              (Var $ paramName p)+              []+ -- Note [IntermArrays] ----- Intermediate memory for the nonsegmented and large segments non-commutative--- reductions with all primitive parameters:+-- Intermediate memory for the nonsegmented and large segments reductions using+-- 'StageOneTiled' (see Note [Stage one strategies]): -- --   These kernels need shared memory for 1) the initial collective copy, 2) the --   (virtualized) block reductions, and (TODO: this one not implemented yet!)@@ -1078,9 +1105,63 @@ --   Aside from shared memory, these kernels also require a CHUNK-sized array of --   thread-private register memory per reduction operator. ----- For all other reductions, ie. commutative reductions, reductions with at--- least one non-primitive operator, and small segments reductions:+-- For all other reductions, ie. those using 'StageOneStrided',+-- 'StageOneContiguous', or 'StageOneWindow', and small segments+-- reductions: -- --   These kernels use shared memory only for the intra-block reductions, and --   since they do not use chunking or CHUNK, they all require onlly `tblock_size --   * max elem_sizes` bytes of shared memory and no thread-private register mem.++-- Note [Stage one strategies]+--+-- In the first stage of a nonsegmented or large-segments reduction, each+-- thread reduces some of the input elements, after which the per-thread+-- results are combined by a block-wide reduction.  The strategy decides which+-- elements a thread reduces.+--+-- A commutative operator allows thread @t@ of @T@ to reduce elements @t@,+-- @t+T@, @t+2T@, and so on ('StageOneStrided'). This gives coalesced memory+-- accesses and a single block-wide reduction at the end, and we always pick+-- this strategy for any commutative operator.+--+-- A non-commutative operator requires that each thread reduces a contiguous+-- range of elements.  There are three ways of achieving this:+--+-- - 'StageOneContiguous': each thread reduces a contiguous range of @q@+--   elements, followed by a single block-wide reduction. Order is preserved+--   because consecutive threads have consecutive ranges, the block-wide+--   reduction combines the threads in order, and stage two combines the blocks+--   in order. This is similar to 'StageOneStrided', but results in noncoalesced+--   reads. However, this strategy can still be a good choice for complex+--   noncommutative reductions that need fairly little input from global memory,+--   but have large operands.+--+-- - 'StageOneTiled': each block processes a sequence of contiguous tiles of+--   @tblock_size * chunk@ elements. The map function is applied to the elements+--   of a tile with coalesced accesses, its results are stored in thread-private+--   chunks, and the chunks are transposed through shared memory (the collective+--   copies), such that each thread obtains @chunk@ consecutive elements, which+--   it reduces sequentially. Since the elements a thread obtains from+--   consecutive tiles are not adjacent, each tile ends with a block-wide+--   reduction, whose result thread 0 keeps in its accumulator. The chunk+--   size amortises the block-wide reductions, but is bounded by the private+--   storage of @chunk@ copies of the operands (see 'getRedChunkSize'). This+--   requires non-array operands.+--+-- - 'StageOneWindow': each block slides a window of @tblock_size@ elements+--   across a contiguous range of @q * tblock_size@ elements, with coalesced+--   accesses and a block-wide reduction per window, whose result is kept in+--   the accumulator of thread 0.  This is like 'StageOneTiled' with a chunk+--   size of 1, but needs no private copies of the operands, and so works for+--   non-primitive operands.  It shares its code with 'StageOneStrided',+--   except for the indexing and the placement of the block-wide reduction.+--+-- 'StageOneTiled' is preferable for cheap operators, where the reduction is+-- bound by memory bandwidth.  'StageOneContiguous' is preferable for expensive+-- operators, where the block-wide reductions (which apply the operator several+-- times per element in each wave, often divergently) dominate.  We have no+-- good static measure of operator cost, so we use the size of the operands as+-- a crude proxy: we use 'StageOneTiled' when the operands are primitive and+-- take up at most 'maxTiledOperandWords' 32-bit words in total.  The threshold+-- is a heuristic.
src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs view
@@ -3,8 +3,6 @@ -- the scan and the chosen abckend. module Futhark.CodeGen.ImpGen.GPU.SegScan (compileSegScan) where -import Control.Monad-import Data.Maybe import Futhark.CodeGen.ImpCode.GPU qualified as Imp import Futhark.CodeGen.ImpGen hiding (compileProg) import Futhark.CodeGen.ImpGen.GPU.Base@@ -37,17 +35,6 @@               (concatMap (bodyResult . lambdaBody) lams)         } -bodyHas :: (Exp GPUMem -> Bool) -> Body GPUMem -> Bool-bodyHas f = any (f' . stmExp) . bodyStms-  where-    f' e-      | f e = True-      | otherwise = isNothing $ walkExpM walker e-    walker =-      identityWalker-        { walkOnBody = const $ guard . not . bodyHas f-        }- canBeSinglePass :: [SegBinOp GPUMem] -> Maybe (SegBinOp GPUMem) canBeSinglePass scan_ops =   if all ok scan_ops@@ -57,9 +44,7 @@     ok op =       segBinOpShape op == mempty         && all primType (lambdaReturnType (segBinOpLambda op))-        && not (bodyHas isAssert (lambdaBody (segBinOpLambda op)))-    isAssert (BasicOp Assert {}) = True-    isAssert _ = False+        && not (bodyMayFail (lambdaBody (segBinOpLambda op)))  -- | Compile 'SegScan' instance to host-level code with calls to -- various kernels.
src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs view
@@ -136,14 +136,17 @@ statusA = 1 statusP = 2 +-- | Scan the look-back window held in the given flag and value arrays, using+-- the first 32 threads, which form a single warp. Basically a kind of segmented+-- scan, optimised for only the cases handled by single pass scan (in+-- particular, no array operands). inBlockScanLookback ::   KernelConstants ->-  Imp.TExp Int64 ->   VName ->   [VName] ->   Lambda GPUMem ->   InKernelGen ()-inBlockScanLookback constants arrs_full_size flag_arr arrs scan_lam = do+inBlockScanLookback constants flag_arr arrs scan_lam = do   flg_x :: TV Int8 <- dPrim "flg_x"   flg_y :: TV Int8 <- dPrim "flg_y"   let flg_param_x = Param mempty (tvVar flg_x) (MemPrim p_int8)@@ -155,43 +158,26 @@   dLParams (lambdaParams scan_lam)    skip_threads <- dPrim "skip_threads"-  let in_block_thread_active =-        tvExp skip_threads .<=. in_block_id-      actual_params = lambdaParams scan_lam+  let thread_active = tvExp skip_threads .<=. ltid32       (x_params, y_params) =-        splitAt (length actual_params `div` 2) actual_params+        splitAt (length (lambdaParams scan_lam) `div` 2) $ lambdaParams scan_lam       y_to_x =-        forM_ (zip x_params y_params) $ \(x, y) ->-          when (primType (paramType x)) $-            copyDWIM (paramName x) [] (Var (paramName y)) []-      y_to_x_flg =-        copyDWIM (tvVar flg_x) [] (Var (tvVar flg_y)) []--  -- Set initial y values-  sComment "read input for in-block scan" $ do-    zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)-    -- Since the final result is expected to be in x_params, we may-    -- need to copy it there for the first thread in the block.-    sWhen (in_block_id .==. 0) $ do-      y_to_x-      y_to_x_flg--  when array_scan barrier--  let op_to_x = do+        forM_ (zip (flg_param_x : x_params) (flg_param_y : y_params)) $ \(x, y) ->+          copyDWIM (paramName x) [] (Var (paramName y)) []+      op_to_x =         sIf           (flg_y_exp .==. statusP_e .||. flg_y_exp .==. statusX_e)-          ( do-              y_to_x_flg-              y_to_x-          )+          y_to_x           (compileBody' x_params $ lambdaBody scan_lam) -  sComment "in-block scan (hopefully no barriers needed)" $ do+  sComment "read input for warp scan" $+    zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)++  sComment "warp scan" $ do     skip_threads <-- 1 -    sWhile (tvExp skip_threads .<. block_size) $ do-      warpLocal . sWhen in_block_thread_active $ do+    sWhile (tvExp skip_threads .<. warp_size) $ do+      warpLocal . sWhen thread_active $ do         sComment "read operands" $           zipWithM_             (readParam (sExt64 $ tvExp skip_threads))@@ -199,7 +185,7 @@             (flag_arr : arrs)         sComment "perform operation" op_to_x -      warpLocal . sWhen in_block_thread_active . sComment "write result" $+      warpLocal . sWhen thread_active . sComment "write result" $         sequence_ $           zipWith3             writeResult@@ -210,33 +196,18 @@       skip_threads <-- tvExp skip_threads * 2   where     p_int8 = IntType Int8-    block_size = 32-    block_id = ltid32 `quot` block_size-    in_block_id = ltid32 - block_id * block_size+    warp_size = 32     ltid32 = kernelLocalThreadId constants     ltid = sExt64 ltid32-    gtid = sExt64 $ kernelGlobalThreadId constants-    array_scan = not $ all primType $ lambdaReturnType scan_lam-    barrier-      | array_scan =-          sOp $ Imp.Barrier Imp.FenceGlobal-      | otherwise =-          sOp $ Imp.Barrier Imp.FenceLocal -    readInitial p arr-      | primType $ paramType p =-          copyDWIMFix (paramName p) [] (Var arr) [ltid]-      | otherwise =-          copyDWIMFix (paramName p) [] (Var arr) [gtid]-    readParam behind p arr-      | primType $ paramType p =-          copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]-      | otherwise =-          copyDWIMFix (paramName p) [] (Var arr) [gtid - behind + arrs_full_size]+    readInitial p arr =+      copyDWIMFix (paramName p) [] (Var arr) [ltid] +    readParam behind p arr =+      copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]+     writeResult x y arr = do-      when (isPrimParam x) $-        copyDWIMFix arr [ltid] (Var $ paramName x) []+      copyDWIMFix arr [ltid] (Var $ paramName x) []       copyDWIM (paramName y) [] (Var $ paramName x) []  -- | Calculate the number of u64 words needed to store n bits@@ -371,9 +342,6 @@     tvSize <$> dPrimV "num_virt_blocks" (n `ceilDiv` (tblock_size_e * chunk))   let num_virt_blocks_e = pe64 num_virt_blocks -  num_virt_threads <--    dPrimVE "num_virt_threads" $ num_virt_blocks_e * tblock_size_e-   let (gtids, dims) = unzip $ unSegSpace space       dims' = map pe64 dims       segmented = length dims' > 1@@ -392,7 +360,8 @@           <$> sAllocArray "aggregates" ty (Shape [num_virt_blocks]) (Space "device")           <*> sAllocArray "incprefixes" ty (Shape [num_virt_blocks]) (Space "device") -  global_id <- genZeroes "global_dynid" 1+  -- The first counter hands out the dynamic IDs.+  global_id <- bookkeepingArray Imp.BookkeepingCounters    let attrs' =         attrs@@ -555,7 +524,16 @@            sIf (virt_tid .<. n) in_bounds out_of_bounds -      sOp $ Imp.ErrorSync Imp.FenceLocal+      -- See Note [Failures in single-pass scan].+      publish_on_failure <- collect . sWhen (ltid32 .==. 0) $ do+        everythingVolatile $+          forM_ (zip incprefixArrays scanop_nes) $ \(incprefixArray, ne) ->+            copyDWIMFix incprefixArray [tvExp dyn_id] ne []+        sOp global_fence+        everythingVolatile $+          copyDWIMFix statusFlags [tvExp dyn_id] (intConst Int8 statusP) []+      sOp $ Imp.ErrorSync Imp.FenceLocal publish_on_failure+       sComment "Transpose scan inputs" $ do         forM_ (zip transposedArrays scan_private_chunks) $ \(trans, priv) -> do           sFor "i" chunk $ \i -> do@@ -611,11 +589,11 @@         blockScan           crossesSegment           tblock_size_e-          num_virt_threads+          tblock_size_e           scan_op1           prefixArrays -        sOp $ Imp.ErrorSync Imp.FenceLocal+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty          let firstThread acc prefixes =               copyDWIMFix (tvVar acc) [] (Var prefixes) [sExt64 tblock_size_e - 1]@@ -725,7 +703,6 @@                     lam' <- renameLambda scan_op1                     inBlockScanLookback                       constants-                      num_virt_threads                       warpscan                       exchanges                       lam'@@ -846,7 +823,7 @@                 lambdaParams $                   segPostOpLambda post_op -      sComment "Compute post op and write to global memory." $ do+      sComment "Compute post op and write to global memory." $         sFor "i" chunk $ \i -> do           flat_idx <- dPrimVE "flat_idx" $ thd_offset + i * tblock_size_e           sWhen (flat_idx .<. n) $ do@@ -883,5 +860,28 @@                       then pure ()                       else copyDWIMFix (patElemName pe) (map le64 gtids) subexp [] -      sOp local_barrier+      -- The block has published its prefix, so it may stop here.+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty {-# NOINLINE compileSegScan #-}++-- Note [Failures in single-pass scan]+--+-- Normally, when a thread fails, its whole thread block stops at the next+-- 'Imp.ErrorSync'. The single-pass scan cannot allow that before the block has+-- published its prefix, as each block waits in the lookback for the status+-- flags of the blocks with smaller dynamic IDs, so a block that stops before+-- publishing its flag leaves every later block waiting forever. Stopping after+-- publishing is fine, no block waits for the larger dynamic IDs that are then+-- skipped. The block that fetches the last ID normally resets the dynamic ID+-- counter, so after a failure the counter may be left nonzero. This is+-- harmless, as the runtime system resets the bookkeeping buffers when it+-- reports the failure.+--+-- The only failures before publishing are in the segop body (the "map part"),+-- as 'canBeSinglePass' checks that the scan operator cannot fail. The+-- 'Imp.ErrorSync' after the map therefore has a handler that publishes the+-- neutral element as the block's inclusive prefix before the block stops. The+-- later blocks then compute meaningless results, but as the failure is recorded+-- globally, the results are never used. Failures in the post op reach the+-- 'Imp.ErrorSync' at the end of the virtual block, after the prefix has been+-- published.
src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs view
@@ -257,7 +257,7 @@                           to' = sExt64 to + tvExp chunk_offset                        in f from' to' -              sOp $ Imp.ErrorSync fence+              sOp $ Imp.ErrorSync fence mempty                -- We need to avoid parameter name clashes.               scan_op_renamed <- renameLambda scan_op@@ -386,7 +386,7 @@               -- reload carry (xParams) from the scan output written by the               -- previous chunk.  Thread 0 reads the last element of the               -- previous chunk, unless a segment boundary falls between the-              -- chunks.+              -- chunks, or that element is out of bounds.               when use_global_carry $ do                 crosses_seg <-                   dPrimVE "crosses_seg" $@@ -394,7 +394,11 @@                       Nothing -> false                       Just f -> f prev_chunk_last (prev_chunk_last + 1)                 sIf-                  (chunk_id .>. 0 .&&. kernelLocalThreadId constants .==. 0 .&&. bNot crosses_seg)+                  ( (chunk_id .>. 0)+                      .&&. (kernelLocalThreadId constants .==. 0)+                      .&&. (prev_chunk_last .<. product dims')+                      .&&. bNot crosses_seg+                  )                   ( do                       let carry_is = unflattenIndex dims' prev_chunk_last                       forM_ (zip scan_x_params pes) $ \(p, pe) ->@@ -425,7 +429,7 @@                   \(arr, se) ->                     copyDWIMFix arr [localArrayIndex constants] se [] -              sOp $ Imp.ErrorSync fence+              sOp $ Imp.ErrorSync fence mempty                -- crossesSegment' maps block-local thread IDs to element               -- indices, adjusting for the current chunk offset.@@ -575,8 +579,9 @@                     (Var $ paramName p)                     [] -      sOp $ Imp.Barrier Imp.FenceLocal-+      -- Each thread reads only the elements it wrote itself, so no barrier is+      -- needed, and a failure must be able to reach the 'Imp.ErrorSync' at the+      -- end of the iteration without passing one.       if isIdentityPostOp post_op         then pure ()         else do@@ -607,8 +612,6 @@                   forM_ (zip (patElems pat) res) $ \(pe, subexp) ->                     copyDWIMFix (patElemName pe) (map Imp.le64 gtids) subexp [] -      sOp $ Imp.Barrier Imp.FenceGlobal- isIdentityPostOp :: SegPostOp rep -> Bool isIdentityPostOp post_op =   map resSubExp (bodyResult (lambdaBody lam))@@ -636,12 +639,15 @@    -- The stage-2 block size is a tunable/user-settable parameter.  It   -- is independent of stage1_num_tblocks so the autotuner can treat-  -- it as a fixed knob.+  -- it as a fixed knob.  It may not exceed the number of stage-1+  -- threads, because the memory of array-typed operator parameters+  -- is allocated for that many threads.   stage2_tblock_size_param <- getSize "segscan_stage2_tblock_size" SizeThreadBlock   stage2_tblock_size <-     fmap (Imp.Count . tvSize) $       dPrimV "stage2_tblock_size" $-        tvExp stage2_tblock_size_param+        sMin64 (tvExp stage2_tblock_size_param) $+          pe64 (unCount (kAttrNumBlocks attrs)) * pe64 (unCount (kAttrBlockSize attrs))    let shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)       scan_ts = concatMap shpT scans
src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs view
@@ -126,6 +126,7 @@     getSize (ImpOpenCL.CmpSizeLe _ v _) = Just v     getSize (ImpOpenCL.GetSizeMax {}) = Nothing     getSize (ImpOpenCL.LaunchKernel {}) = Nothing+    getSize (ImpOpenCL.GetBookkeeping {}) = Nothing     directUseInFun fun = mapMaybe getSize $ toList $ functionBody fun     direct_uses = map (second directUseInFun) $ unFunctions $ defFuns defs @@ -158,18 +159,11 @@ data KernelState = KernelState   { kernelSharedMemory :: [SharedMemoryUse],     kernelFailures :: [FailureMsg],-    kernelNextSync :: Int,-    -- | Has a potential failure occurred sine the last-    -- ErrorSync?-    kernelSyncPending :: Bool,     kernelHasBarriers :: Bool   }  newKernelState :: [FailureMsg] -> KernelState-newKernelState failures = KernelState mempty failures 0 False False--errorLabel :: KernelState -> String-errorLabel = ("error_" ++) . show . kernelNextSync+newKernelState failures = KernelState mempty failures False  data ToOpenCL = ToOpenCL   { clGPU :: M.Map KernelName (KernelSafety, T.Text),@@ -195,6 +189,7 @@ codeMayFail f (For _ _ x) = codeMayFail f x codeMayFail f (While _ x) = codeMayFail f x codeMayFail f (If _ x y) = codeMayFail f x || codeMayFail f y+codeMayFail f (OnFailure x y) = codeMayFail f x || codeMayFail f y codeMayFail _ _ = False  hostOpMayFail :: ImpGPU.HostOp -> Bool@@ -235,6 +230,8 @@   pure $ ImpOpenCL.CmpSizeLe v key x onHostOp _ (ImpGPU.GetSizeMax v size_class) =   pure $ ImpOpenCL.GetSizeMax v size_class+onHostOp _ (ImpGPU.GetBookkeeping v b) =+  pure $ ImpOpenCL.GetBookkeeping v b  genGPUCode ::   Env ->@@ -303,7 +300,8 @@ calledInHostOp _ = mempty  calledInKernelOp :: KernelOp -> S.Set Name-calledInKernelOp = const mempty+calledInKernelOp (ErrorSync _ handler) = calledFuncs calledInKernelOp handler+calledInKernelOp _ = mempty  ensureDeviceFuns :: ImpGPU.KernelCode -> OnKernelM [Name] ensureDeviceFuns code = do@@ -347,6 +345,7 @@           -- No need to free, as we cannot allocate memory in kernels.           mapM_ GC.item =<< GC.declAllocatedMem           mapM_ GC.item body+          GC.items =<< GC.errorLabel       kstate = GC.compUserState cstate        (kernel_consts, (const_defs, const_undefs)) =@@ -359,17 +358,32 @@   let (use_params, unpack_params) =         unzip $ mapMaybe useAsParam $ kernelUses kernel -  -- The local_failure variable is an int despite only really storing-  -- a single bit of information, as some OpenCL implementations-  -- (e.g. AMD) does not like byte-sized shared memory (and the others-  -- likely pad to a whole word anyway).-  let (safety, error_init)+  -- The local_failure variable is an int despite only really storing a single+  -- bit of information, as some OpenCL implementations (e.g. AMD) does not like+  -- byte-sized shared memory (and the others likely pad to a whole word+  -- anyway). The CUDA/HIP prelude #defines __local to nothing, so we need+  -- special handling here to get a shared-memory local variable. Ugly...+  let cuda_local_failure_decl =+        [C.citem|$escstm:("__shared__ volatile int local_failure;")|]+      local_failure_decl =+        case target of+          TargetOpenCL -> [C.citem|volatile __local int local_failure;|]+          TargetCUDA -> cuda_local_failure_decl+          TargetHIP -> cuda_local_failure_decl++      -- No thread may write a failure before all threads have cleared it.+      local_failure_barrier+        | kernelHasBarriers kstate = [C.citems|barrier(CLK_LOCAL_MEM_FENCE);|]+        | otherwise = []++      (safety, error_init)         -- We conservatively assume that any called function can fail.         | not $ null called =             ( SafetyFull,-              [C.citems|volatile __local int local_failure;+              [C.citems|$item:local_failure_decl                         // Harmless for all threads to write this.-                        local_failure = 0;|]+                        local_failure = 0;+                        $items:local_failure_barrier|]             )         | length (kernelFailures kstate) == length failures =             if kernelFailureTolerant kernel@@ -388,14 +402,14 @@               else                 ( SafetyFull,                   [C.citems|-                     volatile __local int local_failure;+                     $item:local_failure_decl                      if (failure_is_an_option) {                        int failed = *global_failure >= 0;                        if (failed) {                          return;                        }                      }-                     // All threads write this value - it looks like CUDA has a compiler bug otherwise.+                     // Harmless for all threads to write this.                      local_failure = 0;                      barrier(CLK_LOCAL_MEM_FENCE);                   |]@@ -457,8 +471,6 @@                     $items:error_init                     $items:kernel_body -                    $id:(errorLabel kstate): return;-                     $items:const_undefs                 }|]   modify $ \s ->@@ -565,18 +577,6 @@     useToArg (ScalarUse v pt) = Just $ ValueKArg (LeafExp v pt) pt     useToArg ConstUse {} = Nothing -nextErrorLabel :: GC.CompilerM KernelOp KernelState String-nextErrorLabel =-  errorLabel <$> GC.getUserState--incErrorLabel :: GC.CompilerM KernelOp KernelState ()-incErrorLabel =-  GC.modifyUserState $ \s -> s {kernelNextSync = kernelNextSync s + 1}--pendingError :: Bool -> GC.CompilerM KernelOp KernelState ()-pendingError b =-  GC.modifyUserState $ \s -> s {kernelSyncPending = b}- hasCommunication :: ImpGPU.KernelCode -> Bool hasCommunication = any communicates   where@@ -642,16 +642,15 @@       GC.modifyUserState $ \s ->         s {kernelSharedMemory = (name', size) : kernelSharedMemory s}       GC.stm [C.cstm|$id:name = (__local unsigned char*) $id:name';|]-    kernelOps (ErrorSync f) = do-      label <- nextErrorLabel-      pending <- kernelSyncPending <$> GC.getUserState-      when pending $ do-        pendingError False-        GC.stm [C.cstm|$id:label: barrier($exp:(fence f));|]-        GC.stm [C.cstm|if (local_failure) { return; }|]+    kernelOps (ErrorSync f handler) = do+      error_label <- GC.errorLabel+      unless (null error_label) $ do+        handler' <- GC.collect $ GC.compileCode handler+        GC.items error_label+        GC.stm [C.cstm|barrier($exp:(fence f));|]+        GC.stm [C.cstm|if (local_failure) { $items:handler' return; }|]       GC.stm [C.cstm|barrier($exp:(fence f));|]       GC.modifyUserState $ \s -> s {kernelHasBarriers = True}-      incErrorLabel     kernelOps (Atomic space aop) = atomicOps space aop      atomicCast s t = do@@ -814,16 +813,20 @@     kernelReadScalar =       GC.readScalarPointerWithQuals pointerQuals -    whatNext = do-      label <- nextErrorLabel-      pendingError True-      pure $-        if has_communication-          then [C.citems|local_failure = 1; goto $id:label;|]-          else-            if mode == FunMode-              then [C.citems|return 1;|]-              else [C.citems|return;|]+    -- A failing thread jumps to the next error label, which is either at an+    -- 'ErrorSync', before the handler of an 'OnFailure', or at the end of the+    -- kernel. Since 'local_failure' is set before the jump, the handler of an+    -- 'OnFailure' can propagate the failure with a plain jump. Device functions+    -- instead return on failure, which would skip such a handler, so they must+    -- not contain an 'OnFailure'.+    whatNext+      | mode == FunMode = pure [C.citems|return 1;|]+      | has_communication = do+          failure <- GC.failureJump+          pure [C.citems|local_failure = 1; $stm:failure|]+      | otherwise = do+          failure <- GC.failureJump+          pure [C.citems|$stm:failure|]      callInKernel dests fname args       | functionMayFail fname env = do@@ -897,10 +900,12 @@ typesInCode (If (TPrimExp e) c1 c2) =   typesInExp e <> typesInCode c1 <> typesInCode c2 typesInCode (Assert e _ _) = typesInExp e+typesInCode (OnFailure body handler) = typesInCode body <> typesInCode handler typesInCode (Meta _) = mempty typesInCode (DebugPrint _ v) = maybe mempty typesInExp v typesInCode (TracePrint msg) = foldMap typesInExp msg typesInCode (GetUserParam _ _ def) = typesInExp (untyped def)+typesInCode (Op (ErrorSync _ handler)) = typesInCode handler typesInCode Op {} = mempty  typesInExp :: Exp -> S.Set PrimType
src/Futhark/CodeGen/ImpGen/Multicore/Base.hs view
@@ -38,6 +38,7 @@ import Futhark.Error import Futhark.IR.MCMem import Futhark.Transform.Rename+import Futhark.Util (nubOrd) import Prelude hiding (quot, rem)  -- | Is there an atomic t'BinOp' corresponding to this t'BinOp'?@@ -107,13 +108,15 @@     _ -> pure $ product $ init ns_64  -- When the SegRed's return value is a scalar--- we perform a call by value-result in the segop function+-- we perform a call by value-result in the segop function.  Several+-- array results may reside in the same memory block (e.g. after+-- short-circuiting), so the parameters are deduplicated. getReturnParams :: Pat LetDecMem -> SegOp () MCMem -> MulticoreGen [Imp.Param] getReturnParams pat SegRed {} =   -- It's a good idea to make sure any prim values are initialised, as   -- we will load them (redundantly) in the task code, and   -- uninitialised values are UB.-  fmap concat . forM (patElems pat) $ \pe -> do+  fmap (nubOrd . concat) . forM (patElems pat) $ \pe -> do     case patElemType pe of       Prim pt -> patElemName pe <~~ ValueExp (blankPrimValue pt)       _ -> pure ()@@ -149,6 +152,7 @@ isLoadBalanced (a Imp.:>>: b) = isLoadBalanced a && isLoadBalanced b isLoadBalanced (Imp.For _ _ a) = isLoadBalanced a isLoadBalanced (Imp.If _ a b) = isLoadBalanced a && isLoadBalanced b+isLoadBalanced (Imp.OnFailure a b) = isLoadBalanced a && isLoadBalanced b isLoadBalanced Imp.While {} = False isLoadBalanced (Imp.Op (Imp.ParLoop _ code _)) = isLoadBalanced code isLoadBalanced (Imp.Op (Imp.ForEachActive _ a)) = isLoadBalanced a@@ -464,8 +468,11 @@     sUnless (lock_acquired .==. 0) $ do       dLParams acc_params       bind_acc_params-      op_body-      do_hist+      -- A thread whose operator fails must still release the lock, or other+      -- threads would wait for it forever.+      update <- collect $ op_body >> do_hist+      on_failure <- collect release_lock+      emit $ Imp.OnFailure update on_failure       release_lock   where     writeArray bucket arr val = copyDWIMFix arr bucket val []
src/Futhark/CodeGen/RTS/JavaScript.hs view
@@ -4,7 +4,7 @@ module Futhark.CodeGen.RTS.JavaScript   ( serverJs,     valuesJs,-    wrapperclassesJs,+    wrappersJs,   ) where @@ -19,6 +19,6 @@ valuesJs :: T.Text valuesJs = $(embedStringFile "rts/javascript/values.js") --- | @rts/javascript/wrapperclasses.js@-wrapperclassesJs :: T.Text-wrapperclassesJs = $(embedStringFile "rts/javascript/wrapperclasses.js")+-- | @rts/javascript/wrappers.js@+wrappersJs :: T.Text+wrappersJs = $(embedStringFile "rts/javascript/wrappers.js")
src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs view
@@ -1326,8 +1326,12 @@                in -- It is not safe to short circuit if some other pattern                   -- element is aliased to this one, as this indicates the                   -- two pattern elements reference the same physical-                  -- value somehow.+                  -- value somehow.  Nor is it safe if the destination is+                  -- itself bound by the pattern (e.g. the existential memory+                  -- of a 'Match' or 'Loop'), as then it is not in scope where+                  -- @b@ is created, although it is in @scope td_env@.                   if any ((`nameIn` aliasesOf patel) . patElemName) pes+                    || x_mem `elem` map patElemName pes                     then failed                     else case M.lookup b vtab of                       Nothing ->
src/Futhark/Pass/ExpandAllocations.hs view
@@ -161,7 +161,7 @@           -- allocations inside the update function.           lvl = SegThread SegNoVirt Nothing           (op_lam', lam_allocs) =-            extractLambdaAllocations (lvl, [0]) bound_outside mempty op_lam+            extractLambdaAllocations (lvl, [0], Nothing) bound_outside mempty op_lam           variantAlloc (_, Var v, _) = v `notNameIn` bound_outside           variantAlloc _ = False           (variant_allocs, invariant_allocs) = M.partition variantAlloc lam_allocs@@ -178,7 +178,9 @@       let num_is = shapeRank shape           is = map paramName $ take num_is $ lambdaParams op_lam       (alloc_stms, alloc_offsets) <--        genericExpandedInvariantAllocations (const $ const (shape, map le64 is)) invariant_allocs+        genericExpandedInvariantAllocations+          (const $ const (map pe64 $ shapeDims shape, map le64 is))+          invariant_allocs        scope <- askScope       let scope' = scopeOf op_lam <> scope <> scopeOf alloc_stms@@ -217,7 +219,7 @@   ExpandM (Stms GPUMem, (SegLevel, [Lambda GPUMem], KernelBody GPUMem)) transformScanRed lvl space ops kbody = do   bound_outside <- asks $ namesFromList . M.keys-  let user = (lvl, [le64 $ segFlat space])+  let user = (lvl, [le64 $ segFlat space], Nothing)       (kbody', kbody_allocs) =         extractKernelBodyAllocations user bound_outside bound_in_kernel kbody       (ops', ops_allocs) =@@ -343,8 +345,9 @@ type Exp64 = TPrimExp Int64 VName  -- | Identifying the spot where an allocation occurs in terms of its--- level and unique thread ID.-type User = (SegLevel, [Exp64])+-- level and unique thread ID. For a 'SegThreadInBlock' user, also the+-- size of its index space, if that is bound outside the kernel.+type User = (SegLevel, [Exp64], Maybe Exp64)  -- | A description of allocations that have been extracted, and how -- much memory (and which space) is needed.@@ -404,7 +407,7 @@    in (set_stms (stmsFromList stms) body, allocs)  expandable :: User -> Space -> Bool-expandable (SegBlock {}, _) (Space "shared") = False+expandable (SegBlock {}, _, _) (Space "shared") = False expandable _ ScalarSpace {} = False expandable _ _ = True @@ -447,13 +450,22 @@       tell allocs       pure body' -    onOp (_, user_ids) (Inner (SegOp op)) =+    onOp (_, user_ids, _) (Inner (SegOp op)) =       Inner . SegOp <$> mapSegOpM (opMapper user'') op       where+        dims = segSpaceDims $ segSpace op         user'' =-          (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))])+          (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))], extent)+        extent+          | SegThreadInBlock {} <- segLevel op,+            all boundOutside dims =+              Just $ product $ map pe64 dims+          | otherwise = Nothing     onOp _ op = pure op +    boundOutside (Var v) = v `nameIn` bound_outside+    boundOutside Constant {} = True+     opMapper user' =       identitySegOpMapper         { mapOnSegBinOpLambda = onLambda user',@@ -472,7 +484,7 @@       pure lam {lambdaBody = body}  genericExpandedInvariantAllocations ::-  (User -> Space -> (Shape, [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap)+  (User -> Space -> ([Exp64], [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap) genericExpandedInvariantAllocations getNumUsers invariant_allocs = do   -- We expand the invariant allocations by adding an inner dimension   -- equal to the number of kernel threads.@@ -485,22 +497,20 @@           allocpat = Pat [PatElem mem $ MemMem space]       total_size <-         letExp "total_size" <=< toExp . product $-          pe64 per_thread_size : map pe64 (shapeDims num_users)+          pe64 per_thread_size : num_users       letBind allocpat $ Op $ Alloc (Var total_size) space       pure $ M.singleton mem $ newBase user space      newBaseThread user space _old_shape =-      let (users_shape, user_ids) = getNumUsers user space-          dims = map pe64 (shapeDims users_shape)+      let (dims, user_ids) = getNumUsers user space        in ( flattenIndex dims user_ids,             product dims           ) -    newBase user@(SegThreadInBlock {}, _) space = newBaseThread user space-    newBase user@(SegThread {}, _) space = newBaseThread user space-    newBase user@(SegBlock {}, _) space = \_old_shape ->-      let (users_shape, user_ids) = getNumUsers user space-          dims = map pe64 (shapeDims users_shape)+    newBase user@(SegThreadInBlock {}, _, _) space = newBaseThread user space+    newBase user@(SegThread {}, _, _) space = newBaseThread user space+    newBase user@(SegBlock {}, _, _) space = \_old_shape ->+      let (dims, user_ids) = getNumUsers user space        in ( flattenIndex dims user_ids,             product dims           )@@ -514,16 +524,20 @@ expandedInvariantAllocations num_threads (Count num_tblocks) (Count tblock_size) =   genericExpandedInvariantAllocations getNumUsers   where-    getNumUsers (SegThread {}, [gtid]) _ = (Shape [num_threads], [gtid])-    getNumUsers (SegThread {}, [gid, ltid]) _ = (Shape [num_tblocks, tblock_size], [gid, ltid])-    getNumUsers (SegThreadInBlock {}, [gtid]) _ = (Shape [num_threads], [gtid])-    getNumUsers (SegThreadInBlock {}, [_gid, ltid]) (Space "shared") =-      (Shape [tblock_size], [ltid])-    getNumUsers (SegThreadInBlock {}, [gid, ltid]) (Space "device") =-      (Shape [num_tblocks, tblock_size], [gid, ltid])-    getNumUsers (SegBlock {}, [gid]) _ = (Shape [num_tblocks], [gid])+    getNumUsers (SegThread {}, [gtid], _) _ = ([pe64 num_threads], [gtid])+    getNumUsers (SegThread {}, [gid, ltid], _) _ =+      ([pe64 num_tblocks, pe64 tblock_size], [gid, ltid])+    getNumUsers (SegThreadInBlock {}, [gtid], _) _ = ([pe64 num_threads], [gtid])+    getNumUsers (SegThreadInBlock {}, [_gid, ltid], extent) (Space "shared") =+      ([inBlockUsers extent], [ltid])+    getNumUsers (SegThreadInBlock {}, [gid, ltid], extent) (Space "device") =+      ([pe64 num_tblocks, inBlockUsers extent], [gid, ltid])+    getNumUsers (SegBlock {}, [gid], _) _ = ([pe64 num_tblocks], [gid])     getNumUsers user space = error $ "getNumUsers: unhandled " ++ show (user, space) +    -- The number of threads in a block that use an in-block allocation.+    inBlockUsers = maybe (pe64 tblock_size) (sMin64 (pe64 tblock_size))+ expandedVariantAllocations ::   SubExp ->   SegSpace ->@@ -770,9 +784,9 @@  offsetMemoryInLambda :: RebaseMap -> Lambda GPUMem -> OffsetM (Lambda GPUMem) offsetMemoryInLambda offsets lam = do-  body <- inScopeOf lam $ offsetMemoryInBody offsets $ lambdaBody lam-  let params = map (offsetMemoryInParam offsets) $ lambdaParams lam-  pure $ lam {lambdaBody = body, lambdaParams = params}+  let lam' = lam {lambdaParams = map (offsetMemoryInParam offsets) (lambdaParams lam)}+  body <- inScopeOf lam' $ offsetMemoryInBody offsets $ lambdaBody lam'+  pure $ lam' {lambdaBody = body}  -- A loop may have memory parameters, and those memory blocks may -- be expanded.  We assume (but do not check - FIXME) that if the
src/Futhark/Pass/ExtractMulticore.hs view
@@ -123,6 +123,9 @@   where     transformInput (shape, arrs, op) =       (shape,arrs,) <$> traverse (bitraverse transformLambda pure) op+transformStm stm@(Let _ aux (Op _))+  | "sequential" `inAttrs` stmAuxAttrs aux =+      pure $ oneStm $ runIdentity $ rephraseStm toMC stm transformStm (Let pat aux (Op op)) =   fmap (certify (stmAuxCerts aux)) <$> transformSOAC pat (stmAuxAttrs aux) op @@ -145,8 +148,10 @@  sequentialiseBody :: Body SOACS -> ExtractM (Body MC) sequentialiseBody = pure . runIdentity . rephraseBody toMC-  where-    toMC = injectSOACS OtherOp++-- | Keep SOACs as they are, to be compiled to sequential code.+toMC :: Rephraser Identity SOACS MC+toMC = injectSOACS OtherOp  transformFunDef :: FunDef SOACS -> ExtractM (FunDef MC) transformFunDef (FunDef entry attrs name rettype params body) = do
src/Futhark/Pass/Flatten.hs view
@@ -170,7 +170,9 @@           (\i pe -> DistResult (ResTag i) (DistType [] (Rank 0) (patElemType pe)) (patElemName pe))           [0 ..]           (patElems nested_pat)-  env <- flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form)+  env <-+    freshenSOAC res $+      flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form)   forM_ (zip (patNames pat) res) $ \(pat_v, r) ->     case resVar (distResTag r) env of       Regular v ->@@ -321,7 +323,8 @@           ~[pe] = patElems pat       flattenBasicOp ops segments env (inps, res', pe, aux, op)     Op (Screma w arrs form) ->-      flattenScrema ops segments env inps res (pat, aux) (w, arrs, form)+      freshenSOAC res $+        flattenScrema ops segments env inps res (pat, aux) (w, arrs, form)     Match scrutinees cases defaultCase rt ->       flattenMatch ops segments env inps res aux scrutinees cases defaultCase rt     Apply name args rettype s ->@@ -333,9 +336,11 @@     WithAcc inputs lam ->       flattenWithAcc ops segments env inps res pat aux inputs lam     Op (Hist w hist_inputs hist_ops bucket_fun) ->-      flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun)+      freshenSOAC res $+        flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun)     Op (FlatMap w arrs lam) ->-      flattenFlatMapNested ops segments env inps res aux w arrs lam+      freshenSOAC res $+        flattenFlatMapNested ops segments env inps res aux w arrs lam     Op (Stream {}) -> error "transformDistStm: Stream should have been removed"     Op (JVP {}) -> error "Unhandled JVP"     Op (VJP {}) -> error "Unhandled VJP"@@ -378,6 +383,39 @@   env' <- foldM (flattenDistStm ops segments) env dstms   concat <$> zipWithM (liftRegResult lvl segments w inputs env') rettype result +-- | Flatten a SOAC, copying those of its results that would otherwise alias+-- something. This is because a SOAC result is fresh, and so must be the results+-- of applying flattening to the SOAC, which is not the case if the body is e.g.+-- a slice or a transpose.+freshenSOAC :: [DistResult] -> FlattenM DistEnv -> FlattenM DistEnv+freshenSOAC res m = do+  (env, stms) <- collectStms m+  addStms stms+  let vs = map (repArray . (`resVar` env) . distResTag) res+      -- Transitive aliases of each result, including names bound in 'stms'.+      als = bodyAliases $ analyseBody mempty $ Body () stms (varsRes vs)+      bound = boundByStms stms+  snd <$> foldM (freshen bound) (mempty, env) (zip3 res vs als)+  where+    -- Only the data array of an irregular representation can be consumed.+    repArray (Regular v) = v+    repArray (Irregular irreg) = irregularD irreg++    freshen bound (taken, env) (r, v, v_als) = do+      v_t <- lookupType v+      let v_all = oneName v <> v_als+          aliases_outside = v_all `namesSubtract` bound /= mempty+      case v_t of+        Array {}+          | aliases_outside || v_all `namesIntersect` taken -> do+              v' <- letExp (baseName v <> "_fresh") $ BasicOp $ Replicate mempty $ Var v+              env' <- insertRepM (distResTag r) (replaceArray v' $ resVar (distResTag r) env) env+              pure (taken, env')+        _ -> pure (taken <> v_all, env)++    replaceArray v' (Regular _) = Regular v'+    replaceArray v' (Irregular irreg) = Irregular irreg {irregularD = v'}+ -- | A lifted function must return fresh, non-aliasing arrays (as it -- corresponds to 'map f'; see 'liftRetType').  This is not -- automatically the case: a result may alias a parameter (when a value@@ -570,11 +608,6 @@   pure $ Lambda params ret body'  transformStm :: Attrs -> FunHasParallelism -> FunSizeParams -> Stm SOACS -> FlattenM ()-transformStm attrs funHasParallelism funSizeParams (Let pat aux (Op soac))-  | "sequential_outer" `inAttrs` stmAuxAttrs aux = do-      scope <- askScope-      stms <- runBuilderT_ (FOT.transformSOAC pat soac) (castScope scope)-      transformStms attrs funHasParallelism funSizeParams $ fmap (certify (stmAuxCerts aux)) stms transformStm _ _ _ stm   | "sequential" `inAttrs` stmAuxAttrs (stmAux stm) = addStm $ soacsStmToGPU stm transformStm _ _ _ (Let pat aux (Op (Hist w arrs ops bucket_fun))) =
src/Futhark/Pass/Flatten/Builtins.hs view
@@ -4,6 +4,7 @@   ( BuiltinFn (..),     builtinFunDef,     determineReduceOp,+    getGridSize,     genUniformSegHist,     mkSegSpace,     segMap,@@ -62,24 +63,17 @@  data ThreadRecommendation = ManyThreads | NoRecommendation SegVirt -numberOfBlocks ::+-- | Compute a sensible grid size given this many (virtual) threads.+getGridSize ::   (MonadBuilder m, Op (Rep m) ~ HostOp inner (Rep m)) =>   Name ->   SubExp ->-  SubExp ->-  m (SubExp, SubExp)-numberOfBlocks desc w tblock_size = do+  Count BlockSize SubExp ->+  m (Count NumBlocks SubExp)+getGridSize desc num_threads tblock_size = do   max_num_tblocks_key <- nameFromText . prettyText <$> newVName (desc <> "_num_tblocks")-  num_tblocks <--    letSubExp "num_tblocks" $-      Op $-        SizeOp $-          CalcNumBlocks w max_num_tblocks_key tblock_size-  num_threads <--    letSubExp "num_threads" $-      BasicOp $-        BinOp (Mul Int64 OverflowUndef) num_tblocks tblock_size-  pure (num_tblocks, num_threads)+  fmap Count . letSubExp "num_tblocks" . Op . SizeOp $+    CalcNumBlocks num_threads max_num_tblocks_key (unCount tblock_size)  -- | Like 'segThread', but cap the thread count to the input size. -- This is more efficient for small kernels, e.g. summing a small@@ -104,8 +98,8 @@       let grid = KernelGrid (Count usable_groups) (Count tblock_size)       pure $ SegThread SegNoVirt (Just grid)     NoRecommendation v -> do-      (num_tblocks, _) <- numberOfBlocks desc w tblock_size-      let grid = KernelGrid (Count num_tblocks) (Count tblock_size)+      num_tblocks <- getGridSize desc w $ Count tblock_size+      let grid = KernelGrid num_tblocks (Count tblock_size)       pure $ SegThread v (Just grid)  -- FIXME: We use segThreadCapped here because otherwise we may get
src/Futhark/Pass/Flatten/Incremental.hs view
@@ -55,7 +55,6 @@      -- * Levels     defaultSegLevel,-    inBlockSegLevel,     allowVersioning,      -- * Various queries@@ -84,9 +83,6 @@ defaultSegLevel :: SegLevel defaultSegLevel = SegThread SegVirt Nothing -inBlockSegLevel :: SegLevel-inBlockSegLevel = SegThreadInBlock SegNoVirt- allowVersioning :: SegLevel -> Bool allowVersioning SegThreadInBlock {} = False allowVersioning _ = True@@ -191,18 +187,30 @@ onlyExploitIntra attrs =   "only_intra" `inAttrs` flatteningAttrs attrs +onlyExploitOuter :: Attrs -> Bool+onlyExploitOuter attrs =+  "only_outer" `inAttrs` flatteningAttrs attrs+ mayExploitOuter :: Attrs -> Bool mayExploitOuter attrs =-  not $ "no_outer" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'+  not $ "no_outer" `inAttrs` attrs' || "only_full" `inAttrs` attrs'   where     attrs' = flatteningAttrs attrs  mayExploitIntra :: Attrs -> Bool mayExploitIntra attrs =-  not $ "no_intra" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'+  not $ "no_intra" `inAttrs` attrs' || "only_full" `inAttrs` attrs'   where     attrs' = flatteningAttrs attrs +-- | Is this one of the flattening attributes that select which versions are+-- generated? These are the ones an enclosing @only_full@ overrides.+isVersioningAttr :: Attr -> Bool+isVersioningAttr (AttrComp f [x]) =+  f `elem` ["flattening", "incremental_flattening"]+    && x `elem` ["no_outer", "no_intra", "only_intra", "only_outer", "only_full"]+isVersioningAttr _ = False+ intraBlockAlternative ::   Intrablock.IntrablockResult ->   FlattenM (SubExp, Body GPU)@@ -215,7 +223,7 @@       BasicOp $         CmpOp           (CmpSle Int64)-          (Intrablock.intraThreadBlockSize intra)+          (unCount (Intrablock.intraThreadBlockSize intra))           max_tblock_size   (intra_suff, _) <-     sufficientParallelism@@ -263,7 +271,7 @@     _       | parallel_fun_inside ->           full []-      | "sequential_inner" `inAttrs` attrs -> do+      | onlyExploitOuter attrs -> do           outer_body <- mkOuterBody           kernelAlternatives desc result_ts outer_body []     Nothing@@ -317,7 +325,7 @@ scanRedAlternatives desc result_ts attrs parallel_fun_inside allow_versioning ws mkFullBody mkOuterBody   | parallel_fun_inside =       fullAlternative-  | "sequential_inner" `inAttrs` attrs =+  | onlyExploitOuter attrs =       outerAlternative   | mayExploitOuter attrs && allow_versioning =       fullWithOuterAlternative@@ -375,16 +383,14 @@           0       where         attrs = stmAuxAttrs $ stmAux stm-        sequential_inner = "sequential_inner" `inAttrs` attrs          zeroIfTooSmall (Constant (IntValue x))           | intToInt64 x < 32 = 0         zeroIfTooSmall _ = 1 -        mapLike w lam' =-          if sequential_inner-            then 0-            else max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam'))+        mapLike w lam'+          | onlyExploitOuter attrs = 0+          | otherwise = max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam'))  -- | A lambda is worth sequentialising if it contains enough nested parallelism -- of an interesting kind, or if distributing it would fragment sequential@@ -404,7 +410,7 @@           0 :: Int       | Op (Screma _ _ form@(ScremaForm lam' _ _ _)) <- stmExp stm,         isJust $ isMapSOAC form =-          if sequential_inner+          if onlyExploitOuter attrs             then 0             else bodyInterest (depth + 1) (lambdaBody lam')       | Loop _ _ body <- stmExp stm =@@ -432,7 +438,6 @@           0       where         attrs = stmAuxAttrs $ stmAux stm-        sequential_inner = "sequential_inner" `inAttrs` attrs  bodyHasParallelism :: FunHasParallelism -> Body SOACS -> Bool bodyHasParallelism funHasParallelism =@@ -521,15 +526,17 @@           <$> preprocessStms scope stms       pure $ mkBody stms' $ varsRes $ patNames pat --- | Add the flattening attributes of the enclosing context to a statement. A--- statement that carries flattening attributes of its own is left alone, as--- those are more specific.+-- | Add the flattening attributes of the enclosing context to a statement, and+-- to the statements nested in its sequential control flow (see Note [Flattening+-- attribute propagation]). propagateAttrs :: Attrs -> Stm SOACS -> Stm SOACS propagateAttrs attrs stm   | attrs' == mempty = stm-  | flatteningAttrs (stmAuxAttrs (stmAux stm)) == mempty =-      stm {stmAux = (stmAux stm) {stmAuxAttrs = attrs' <> stmAuxAttrs (stmAux stm)}}-  | otherwise = stm+  | otherwise =+      stm+        { stmAux = (stmAux stm) {stmAuxAttrs = stm_attrs},+          stmExp = propagateInExp stm_attrs $ stmExp stm+        }   where     -- 'flatteningAttrs' strips the enclosing 'flattening', which has to be put     -- back for the attributes to be recognised on the statement.@@ -537,17 +544,44 @@       mconcat . mapAttrs (oneAttr . AttrComp "flattening" . pure) $         flatteningAttrs attrs +    own = stmAuxAttrs $ stmAux stm++    stm_attrs+      | "only_full" `inAttrs` flatteningAttrs attrs =+          attrs' <> mconcat (mapAttrs keepNonVersioning own)+      | flatteningAttrs own == mempty =+          attrs' <> own+      | otherwise =+          own++    keepNonVersioning attr+      | isVersioningAttr attr = mempty+      | otherwise = oneAttr attr++-- | Propagate attributes into the bodies of sequential control flow, but not+-- into SOAC lambdas.+propagateInExp :: Attrs -> Exp SOACS -> Exp SOACS+propagateInExp attrs (Loop merge form body) =+  Loop merge form $ propagateInBody attrs body+propagateInExp attrs (Match ses cases defbody dec) =+  Match ses (map (fmap $ propagateInBody attrs) cases) (propagateInBody attrs defbody) dec+propagateInExp attrs (WithAcc inputs lam) =+  WithAcc inputs lam {lambdaBody = propagateInBody attrs $ lambdaBody lam}+propagateInExp _ e = e++propagateInBody :: Attrs -> Body SOACS -> Body SOACS+propagateInBody attrs body =+  body {bodyStms = fmap (propagateAttrs attrs) (bodyStms body)}+ -- | Impose outside flattening attributes on a statement. Only SOACs are--- affected, and only those that carry no flattening attributes of their own, as--- those are more specific.+-- affected, following the rules of 'propagateAttrs'. imposeAttrs :: Attrs -> Stm SOACS -> Stm SOACS imposeAttrs attrs stm   | Op {} <- stmExp stm = propagateAttrs attrs stm   | otherwise = stm --- | Propagate incremental flattening attributes to the statements of--- a map lambda body. Statements that carry their own incremental--- flattening attributes are left alone.+-- | Propagate flattening attributes to the statements of a map lambda body,+-- following the rules of 'propagateAttrs'. propagateVersioningAttrs :: Attrs -> Lambda SOACS -> Lambda SOACS propagateVersioningAttrs attrs lam   | flatteningAttrs attrs == mempty = lam@@ -558,3 +592,15 @@               { bodyStms = fmap (propagateAttrs attrs) (bodyStms (lambdaBody lam))               }         }++-- Note [Flattening attribute propagation]+--+-- A flattening attribute on a SOAC applies to every SOAC nested inside it,+-- including inside loops, branches and with_acc. A nested statement keeps its+-- own flattening attributes, except that an enclosing @only_full@ replaces its+-- versioning attributes ('isVersioningAttr').+--+-- Attributes are copied onto a SOAC's body only once it has been preprocessed+-- ('propagateVersioningAttrs', 'factorScremaForParallelism'), as preprocessing+-- may rewrite statements and lose them. Nested SOAC lambdas are reached when+-- their own SOAC is flattened.
src/Futhark/Pass/Flatten/Intrablock.hs view
@@ -10,8 +10,10 @@  import Control.Monad import Control.Monad.RWS-import Control.Monad.Writer+import Control.Monad.State+import Data.Bitraversable import Data.Map qualified as M+import Data.Maybe import Data.Set qualified as S import Futhark.Analysis.PrimExp.Convert import Futhark.IR.GPU hiding (HistOp)@@ -21,7 +23,6 @@ import Futhark.Pass.Flatten.General import Futhark.Pass.Flatten.PreProcess (preprocessLambda) import Futhark.Tools-import Futhark.Transform.FirstOrderTransform qualified as FOT import Futhark.Transform.Rename import Futhark.Transform.ToGPU import Prelude hiding (log)@@ -34,13 +35,14 @@ data IntrablockResult = IntrablockResult   { intraMinPar :: SubExp,     intraAvailPar :: SubExp,-    intraThreadBlockSize :: SubExp,+    intraThreadBlockSize :: Count BlockSize SubExp,     intraPreludeStms :: Stms GPU,     intraKernelStms :: Stms GPU,     intraResultNames :: [VName]   }  type InBlockMapTransformer =+  SegLevel ->   Pat Type ->   SubExp ->   [VName] ->@@ -51,9 +53,10 @@ foldBinOp' _ [] = eSubExp $ intConst Int64 1 foldBinOp' bop (x : xs) = foldBinOp bop x xs --- | Extract the intra-block parallelism used by the body.-findParallelism :: KernelBody GPU -> [[SubExp]]-findParallelism = S.toList . execWriter . onKernelBody+-- | Extract the intra-block parallelism used by the body, paired with whether+-- it is virtualised.+findParallelism :: KernelBody GPU -> [([SubExp], SegVirt)]+findParallelism = S.toList . flip execState mempty . onKernelBody   where     onKernelBody = mapM_ onStm . bodyStms     onBody = mapM_ onStm . bodyStms@@ -64,37 +67,39 @@           walkOnOp = onOp         }     onOp (SegOp op) = do-      tell $ S.singleton $ segSpaceDims $ segSpace op+      modify $ S.insert (segSpaceDims $ segSpace op, segVirt (segLevel op))       onKernelBody $ segBody op     onOp _ = pure () -computeThreadBlockSize :: [[SubExp]] -> [[SubExp]] -> FlattenM (SubExp, SubExp)-computeThreadBlockSize wss_min wss_avail = do-  ws_min <--    mapM (letSubExp "one_intra_par_min" <=< foldBinOp' (Mul Int64 OverflowUndef)) $-      filter (not . null) wss_min-  ws_avail <--    mapM (letSubExp "one_intra_par_avail" <=< foldBinOp' (Mul Int64 OverflowUndef)) $-      filter (not . null) wss_avail+onlyNonVirt :: [(a, SegVirt)] -> [a]+onlyNonVirt = mapMaybe f+  where+    f (_, SegVirt) = Nothing+    f (ws, _) = Just ws -  -- The amount of parallelism available *in the worst case* is-  -- equal to the smallest parallel loop, or *at least* 1.+computeThreadBlockSize ::+  [([SubExp], SegVirt)] ->+  FlattenM (SubExp, Count BlockSize SubExp)+computeThreadBlockSize wss = do+  let prod = letSubExp "one_intra_par" <=< foldBinOp' (Mul Int64 OverflowUndef)+  ws <- mapM (bitraverse prod pure) $ filter (not . null . fst) wss++  -- The amount of parallelism available *in the worst case* is equal to the+  -- smallest parallel loop.   intra_avail_par <--    letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) ws_avail+    letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) (map fst ws) -  tblock_size <- newVName "computed_tblock_size"-  -- The group size is either the maximum of the minimum parallelism-  -- exploited, or the desired parallelism (bounded by the max group-  -- size) in case there is no minimum.-  letBindNames [tblock_size]-    =<< if null ws_min-      then-        eBinOp-          (SMin Int64)-          (eSubExp =<< letSubExp "max_tblock_size" (Op $ SizeOp $ GetSizeMax SizeThreadBlock))-          (eSubExp intra_avail_par)-      else foldBinOp' (SMax Int64) ws_min-  pure (intra_avail_par, Var tblock_size)+  -- If we have purely virtualised SegOps, then the thread block size can be set+  -- freely, but otherwise it must fit all the non-virtualised SegOps. Note that+  -- in the virtualised case, we do not cap the block size by the actual+  -- exploitable parallelism. This is to ensure that the block size is+  -- statically known when the kernel is compiled, which can result in much+  -- better code generation.+  tblock_size <-+    case onlyNonVirt ws of+      [] -> getSize "tblock_size" SizeThreadBlock+      ws' -> letSubExp "tblock_size" =<< foldBinOp' (SMax Int64) (intConst Int64 1 : ws')+  pure (intra_avail_par, Count tblock_size)  -- | Check whether this result is actually something that is acceptable to use: -- the parallel dimensions must all be bound outside the kernel, as nonuniform@@ -155,13 +160,18 @@   let new_segments = segments <> pure w       wss = findParallelism kbody -  noNonuniformPar (freeIn wss) $ do+  noNonuniformPar (freeIn $ map fst wss) $ do     ((intra_avail_par, tblock_size, kspace, num_tblocks), prelude_stms) <-       collectStms $ do-        num_tblocks <--          letSubExp "intra_num_tblocks"-            =<< foldBinOp' (Mul Int64 OverflowUndef) new_segments-        (intra_avail_par, tblock_size) <- computeThreadBlockSize wss wss+        (intra_avail_par, tblock_size) <-+          computeThreadBlockSize wss+        -- CalcNumBlocks needs a "thread count" to cap the actual number of+        -- blocks. Tell a white lie here that is good enough, hopefully.+        num_threads <-+          letSubExp "num_threads" <=< toExp $+            product $+              pe64 (unCount tblock_size) : map pe64 new_segments+        num_tblocks <- getGridSize "intrablock" num_threads tblock_size         gtids <- mapM (const $ newVName "gtid") new_segments         kspace <- mkSegSpace $ zip gtids new_segments         pure (intra_avail_par, tblock_size, kspace, num_tblocks)@@ -176,8 +186,8 @@      let kbody' = kbody {bodyStms = read_input_stms <> bodyStms kbody}         rts = map (length new_segments `stripArray`) result_ts-        grid = KernelGrid (Count num_tblocks) (Count tblock_size)-        lvl = SegBlock SegNoVirt (Just grid)+        grid = KernelGrid num_tblocks tblock_size+        lvl = SegBlock SegVirt (Just grid)         kstm = Let nested_pat aux $ Op $ SegOp $ SegMap lvl kspace rts kbody'      pure $@@ -250,10 +260,14 @@   lam' <- intrablockLambda map_in_block lam   pure (shape, arrs, Just (lam', nes)) +bindPatToRes :: Pat Type -> [VName] -> FlattenM ()+bindPatToRes pat res = forM_ (zip (patNames pat) res) $ \(pe_v, v) ->+  letBindNames [pe_v] $ BasicOp $ SubExp $ Var v+ intrablockStm :: InBlockMapTransformer -> Stm SOACS -> FlattenM () intrablockStm map_in_block stm@(Let pat aux e) = do   scope <- askScope-  let lvl = SegThreadInBlock SegNoVirt+  let lvl = SegThreadInBlock SegVirt    case e of     Loop merge form loopbody ->@@ -270,28 +284,19 @@       inputs' <- mapM (intrablockWithAccInput map_in_block) inputs       lam' <- intrablockLambda map_in_block lam       certifying (stmAuxCerts aux) . letBind pat $ WithAcc inputs' lam'-    Op soac-      | "sequential_outer" `inAttrs` stmAuxAttrs aux ->-          intrablockStms map_in_block . fmap (certify (stmAuxCerts aux))-            =<< runBuilder_ (FOT.transformSOAC pat soac)     Op (Screma w arrs form)       | Just lam <- isMapSOAC form ->-          map_in_block pat w arrs lam+          map_in_block lvl pat w arrs lam     Op (Screma w arrs form)       | Just (post_lam, scans, mapfun) <- isMaposcanomapSOAC form,-        -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileGroupOp+        -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileBlockScan         -- cannot handle multiple scan operators yet.         Scan scanfun nes <- singleScan scans -> do           let scanfun' = soacsLambdaToGPU scanfun               mapfun' = soacsLambdaToGPU mapfun               post_op = soacsLambdaToGPU post_lam-          scan_res <- certifying (stmAuxCerts aux) $ genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ())-          zipWithM_-            ( \pe v ->-                letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v-            )-            (patElems pat)-            scan_res+          bindPatToRes pat <=< certifying (stmAuxCerts aux) $+            genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ())     Op (Screma w arrs form)       | Just (reds, map_lam) <- isRedomapSOAC form -> do           let onRed red =@@ -302,15 +307,8 @@                  in Reduce comm (soacsLambdaToGPU red_lam) (redNeutral red)               reds_gpu = map onRed reds               map_lam' = soacsLambdaToGPU map_lam-          (red_res, stms) <- runBuilder (genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ()))-          certifying (stmAuxCerts aux) $ do-            addStms stms-            zipWithM_-              ( \pe v ->-                  letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v-              )-              (patElems pat)-              red_res+          bindPatToRes pat <=< certifying (stmAuxCerts aux) $+            genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ())     Op (Screma w arrs form) ->       -- This screma is too complicated for us to immediately do       -- anything, so split it up and try again.@@ -318,24 +316,12 @@         =<< runBuilderT (dissectScrema pat w form arrs) (scopeForSOACs scope)     Op (Hist w arrs ops bucket_fun) -> do       let bucket_fun' = soacsLambdaToGPU bucket_fun--      (hist_res, stms) <- runBuilder (genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ()))-      certifying (stmAuxCerts aux) $ do-        addStms stms-        zipWithM_-          ( \pe v ->-              letBindNames [patElemName pe] $-                BasicOp $-                  SubExp $-                    Var v-          )-          (patElems pat)-          hist_res+      bindPatToRes pat <=< certifying (stmAuxCerts aux) $+        genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ())     Op (Stream w arrs accs lam) -> do       types <- asksScope castScope-      ((), stream_stms) <--        runBuilderT (sequentialStreamWholeArray pat w accs lam arrs) types-      intrablockStms map_in_block stream_stms+      intrablockStms map_in_block+        =<< runBuilderT_ (sequentialStreamWholeArray pat w accs lam arrs) types     _ ->       addStm $ soacsStmToGPU stm 
src/Futhark/Pass/Flatten/PreProcess.hs view
@@ -52,7 +52,8 @@ rewriteSoacStm scope (Let pat aux (Op soac))   | "sequential_outer" `inAttrs` stmAuxAttrs aux =       Just $-        preprocessStms scope =<< runSimplifiedBuilder scope (FOT.transformSOAC pat soac)+        preprocessStms scope+          =<< runSimplifiedBuilder scope (certifying (stmAuxCerts aux) $ FOT.transformSOAC pat soac) rewriteSoacStm scope (Let pat aux (Op (Stream w arrs nes lam))) = Just $ do   stms <- runSimplifiedBuilder scope (auxing aux $ sequentialStreamWholeArray pat w nes lam arrs)   preprocessStms scope stms
src/Futhark/Pass/Flatten/SOAC.hs view
@@ -838,12 +838,13 @@  flattenMapForInBlock ::   FlattenOps ->+  SegLevel ->   Pat Type ->   SubExp ->   [VName] ->   Lambda SOACS ->   FlattenM ()-flattenMapForInBlock ops pat w arrs map_lam = do+flattenMapForInBlock ops lvl pat w arrs map_lam = do   scope <- askScope   lam <- preprocessLambda (castScope scope) map_lam   let arrs' = zipWith MapArray arrs $ map paramType (lambdaParams lam)@@ -851,7 +852,7 @@         distributeMapWith ops' scope pat [w] arrs' lam   transformDistributed ops' mempty [w] distributed   where-    ops' = atSegLevel inBlockSegLevel ops+    ops' = atSegLevel lvl ops  resultMapMode :: InnerMapMode -> DistInputs -> Type -> InnerMapMode resultMapMode SingleDim _ _ = SingleDim@@ -1116,7 +1117,7 @@ transformMap ops attrs [] _env _inps pat w arrs map_lam = do   -- Top-level map (no enclosing segments). Preprocess the body and then   -- propagate the attributes onto it, so they influence how the body is-  -- versioned (e.g. only_inner reaching a Screma produced by interchanging a+  -- versioned (e.g. only_full reaching a Screma produced by interchanging a   -- 'sequential_outer' loop). Order matters: preprocessing may rewrite a body   -- statement, so propagating first would lose the attributes on the rewritten   -- form. XXX: this is arguably a bug in preprocessing.