diff --git a/CHANGELOG.md b/CHANGELOG.md
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,51 @@
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.29.1]
+
+### Added
+
+* Better GPU code generation for reductions with very complicated operators.
+
+* Optimised internal book-keeping of GPU operations, resulting in lower static
+  memory usage for large programs.
+
+### Changed
+
+* The flattening attributes have changed somewhat. `#[sequential_inner]`
+  replaced by `#[flattening(only_outer)]`, and `#[flattening(only_inner)]` has
+  been renamed to `#[flattening(only_full)]`, which now also affects any nested
+  parallelism.
+
+### Fixed
+
+* Invalid memory expansion in some rare cases.
+
+* The ISPC backend would handle `f16` incorrectly in some cases.
+
+* Fixed a very rare out-of-bounds access in scans with array operands on GPU
+  backends.
+
+* Invalid handling of aliasing in flattening. (#2595)
+
+* Invalid short-circuiting could cause a compiler crash. (#2597)
+
+* Execution errors with array-operator scans with nonstandard tuning settings.
+
+* On GPU backends, a failing safety check in code fused with a scan could cause
+  barrier divergence or deadlocks.
+
+* On the `cuda` and `hip` backends, a failing safety check in a kernel with
+  barriers stopped only the failing thread rather than its whole thread block,
+  which could cause deadlocks.
+
+* On GPU backends, reductions and scans with operators that contain assertions
+  could cause barrier divergence.
+
+* On GPU and multicore backends, a failing operator in a `reduce_by_index` or
+  `hist` that is implemented with locks could leave a lock held, causing a
+  deadlock.
+
 ## [0.28.2]
 
 ### Added
diff --git a/docs/installation.rst b/docs/installation.rst
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -79,7 +79,7 @@
 
 Tarballs of binary releases can be `found online
 <https://futhark-lang.org/releases/>`_, but are available only for
-very few platforms (as of this writing, only GNU/Linux on x86_64).
+very few platforms (GNU/Linux on x86_64 and aarch64).
 See the enclosed ``README.md`` for installation instructions.
 
 Furthermore, every day a program automatically clones the Git
@@ -96,8 +96,11 @@
 **Linux (x86_64)**
   `futhark-nightly-linux-x86_64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-x86_64.tar.xz>`_
 
-**macOS (ARM64)**
-  `futhark-nightly-macos-arm64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-arm64.tar.xz>`_
+**Linux (Aarch64)**
+  `futhark-nightly-linux-aarch64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-aarch64.tar.xz>`_
+
+**macOS (AARCH64)**
+  `futhark-nightly-macos-aarch64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-aarch64.tar.xz>`_
 
 **Windows (x86_64)**
   `futhark-nightly-windows-x86_64.zip <https://futhark-lang.org/releases/futhark-nightly-windows-x86_64.zip>`_
diff --git a/docs/js-api.rst b/docs/js-api.rst
--- a/docs/js-api.rst
+++ b/docs/js-api.rst
@@ -3,69 +3,36 @@
 JavaScript API Reference
 ========================
 
-The :ref:`futhark-wasm(1)`, :ref:`futhark-wasm-multicore(1)`, and
-:ref:`futhark-webgpu(1)` compilers can generate JavaScript wrapper code
-when used with ``--library``.  This chapter describes the API exposed by
-those wrappers.
-
-The exact generated files and the top-level JavaScript interface differ
-somewhat between the WASM backend and the WebGPU backend, so the
-relevant differences are described below.
+The :ref:`futhark-wasm(1)` and :ref:`futhark-wasm-multicore(1)` compilers can
+generate JavaScript wrapper code when used with ``--library``. This chapter
+describes the API exposed by those wrappers.
 
-First a warning: **the JavaScript API is experimental**.  It may
-change incompatibly even in minor versions of the compiler.
+First a warning: **the JavaScript API is experimental**. It may change
+incompatibly even in minor versions of the compiler.
 
 A Futhark program ``futlib.fut`` compiled with ``--library`` produces
-generated C code as well as JavaScript-facing runtime files.  Some of
-these files are backend-specific.
-
-For the WASM backend, a build typically produces:
-
-* ``futlib.c``, ``futlib.h``: implementation and header C files
-  generated by the compiler, similar to ``futhark c``.
-* ``futlib.class.js``: an intermediate JavaScript build artifact.
-* ``futlib.wasm``: the compiled WebAssembly module, which must be
-  present at runtime.
-* ``futlib.mjs``: an ES module that exports the JavaScript wrapper API.
-
-For the WebGPU backend, a build typically produces:
-
-* ``futlib.c``, ``futlib.h``: implementation and header C files
-  generated by the compiler.
-* ``futlib.js``: the generated backend runtime module.
-* ``futlib.wrapper.js``: the JavaScript wrapper that exposes the
-  Futhark-facing API.
-* ``futlib.json``: manifest data used by the generated wrapper.
-* ``futlib.wasm``: a generated WebAssembly artifact used by the runtime.
-
-The exact file set is backend-dependent and may change over time.
+the following files:
 
-The generated JavaScript wrapper exposes a ``FutharkModule`` class,
-which is the preferred top-level interface for using a compiled Futhark
-program from JavaScript.  The module is initialised with the generated
-backend runtime module.
+* ``futlib.mjs``: an ES module that exports the JavaScript API.
+* ``futlib.wasm``: the compiled WebAssembly module, which is loaded by
+  ``futlib.mjs`` and must be in the same directory.
+* ``futlib.c`` and ``futlib.class.js``: intermediate files, which are
+  not needed at runtime.
 
-The initialisation pattern is the same for the WASM-style backend and
-the WebGPU backend. The only difference is how ``Module`` is made available.
-For the WASM-style backend, the generated ``.mjs`` file exports it as the
-default export, so it is imported together with ``FutharkModule``:
+The module exports a ``FutharkContext`` class, which is the top-level
+interface for using a compiled Futhark program from JavaScript:
 
 .. code-block:: javascript
 
-   import Module, { FutharkModule } from './futlib.mjs';
-
-   const module = await Module();
-
-   const fut = new FutharkModule();
-   await fut.init(module);
-
-For the WebGPU backend, the generated runtime module is usually loaded
-before the application code, for example:
+   import { FutharkContext } from './futlib.mjs';
 
-.. code-block:: html
+   const fut = new FutharkContext();
+   await fut.init();
 
-   <script src="futlib.js"></script>
-   <script src="main.js"></script>
+In a browser, the module must be loaded by a ``<script
+type="module">`` element (or imported by another module), and the
+files must be served over HTTP.  In Node.js, a CommonJS module can
+load it with ``await import('./futlib.mjs')``.
 
 General concerns
 ----------------
@@ -75,102 +42,110 @@
 You are responsible for eventually freeing all objects produced by the
 API, using the appropriate methods.
 
+Methods that may run Futhark code are asynchronous, and return a
+promise.  The asynchronous operations on a ``FutharkContext`` run one at
+a time, in the order they were started, and while one is running, no
+synchronous method may be called on the ``FutharkContext`` or its
+values.  Errors, including errors that occur while running Futhark
+code, are reported by throwing an ``Error`` (or rejecting the promise
+with one).  The arguments of every method are checked, and an invalid
+argument is reported with a ``TypeError`` or ``RangeError``.
+
 Top-level wrapper objects
 -------------------------
 
-The preferred top-level JavaScript interface is ``FutharkModule``.  A
-``FutharkModule`` object represents an instance of a compiled Futhark
-program. It owns the underlying Futhark context and is used to construct
-arrays, call entry points, and free resources.
+The top-level JavaScript interface is ``FutharkContext``.  A
+``FutharkContext`` object represents an instance of a compiled Futhark
+program, and corresponds to a context in the C API.  It is used to
+construct arrays, call entry points, and free resources.
 
-.. js:class:: FutharkModule()
+.. js:class:: FutharkContext()
 
    A bookkeeping class representing an instance of a compiled Futhark
    program.
 
-.. js:function:: FutharkModule.init(module)
-
-   Asynchronously initialise the ``FutharkModule`` object.
-
-   For the WASM-style backend, the ``module`` argument is optional. If
-   omitted, the generated wrapper loads the WebAssembly module itself.
-
-   For the WebGPU backend, ``module`` must be the generated backend
-   runtime module.
-
-.. js:function:: FutharkModule.free()
+.. js:function:: FutharkContext.init(options)
 
-   Frees the Futhark context and configuration associated with the
-   module. It is an error to use a ``FutharkArray`` or ``FutharkOpaque``
-   after the ``FutharkModule`` on which it was created has been freed.
+   Asynchronously initialise the ``FutharkContext`` object.  This must
+   be done exactly once, before the object is used for anything else.
+   Each ``FutharkContext`` loads its own instance of the WebAssembly
+   module, so values cannot be passed between different
+   ``FutharkContext`` objects.
 
-.. js:attribute:: FutharkModule.entry
+   The ``options`` argument is optional.  If given, it must be an object
+   with any of the following fields, which correspond to the
+   configuration functions of the C API (see :ref:`c-api`):
 
-   Object containing the generated entry point functions. For example,
-   an entry point named ``main`` is available as ``fut.entry.main``.
+   * ``debugging``, ``logging``, ``profiling``: booleans that enable
+     debugging, logging, or profiling.  Logging information is printed
+     with ``console.error``.
 
-.. js:attribute:: FutharkModule.types
+   * ``tuning_params``: an object mapping the names of tuning parameters
+     to their values.
 
-   Object containing generated type information and array constructor
-   objects for the array types used by the program.
+   * ``num_threads``: the number of threads to use.  Only supported by
+     :ref:`futhark-wasm-multicore(1)`.
 
-The backends also provides utility methods for synchronisation,
-cache management, and profiling.
+   With :ref:`futhark-wasm-multicore(1)`, ``init()`` fails unless
+   ``SharedArrayBuffer`` is available, which in a browser requires the
+   page to be *cross-origin isolated*.
 
-.. js:function:: FutharkModule.context_sync()
+.. js:function:: FutharkContext.free()
 
-   Wait for pending backend work associated with the context to finish.
+   Frees the underlying Futhark context and configuration.  It is an
+   error to use a ``FutharkArray`` or ``FutharkOpaque`` after the
+   ``FutharkContext`` on which it was created has been freed.
 
-.. js:function:: FutharkModule.clear_caches()
+.. js:attribute:: FutharkContext.entry
 
-   Clear backend caches associated with the context.
+   Object containing the entry point functions. For example, an entry
+   point named ``main`` is available as ``fut.entry.main``.
 
-.. js:function:: FutharkModule.report()
+.. js:attribute:: FutharkContext.types
 
-   Return a profiling report string.
+   Object mapping the name of each array type used by the program, such
+   as ``[]i32``, to its constructor object (see `Array construction`_).
 
-.. js:function:: FutharkModule.pause_profiling()
+The ``FutharkContext`` also provides utility methods for
+synchronisation, cache management, and profiling.  They are all
+asynchronous.
 
-   Pause profiling.
+.. js:function:: FutharkContext.context_sync()
 
-.. js:function:: FutharkModule.unpause_profiling()
+   Wait for pending work associated with the context to finish.
 
-   Resume profiling.
+.. js:function:: FutharkContext.clear_caches()
 
-WASM compatibility interface
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+   Clear caches associated with the context.
 
-The WASM-style backend also keep the older ``FutharkContext`` interface
-for backwards compatibility.
+.. js:function:: FutharkContext.report()
 
-.. js:function:: newFutharkContext()
+   Return a profiling report string.
 
-   Asynchronously create a new ``FutharkContext`` object.
+.. js:function:: FutharkContext.pause_profiling()
 
-.. js:class:: FutharkContext()
+   Pause profiling.
 
-   Backwards-compatible wrapper class for WASM-style backend.
-   ``FutharkContext`` supports the same main interface as
-   ``FutharkModule``, including ``entry``, ``types``, array constructors,
-   and ``free``.
+.. js:function:: FutharkContext.unpause_profiling()
 
-New code should prefer ``FutharkModule``.
+   Resume profiling.
 
 Values
 ------
 
 Numeric types ``u8``, ``u16``, ``u32``, ``i8``, ``i16``, ``i32``, ``f16``,
 ``f32``, and ``f64`` are mapped to JavaScript's standard number type.
-64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``.  ``bool``
-is mapped to JavaScript's ``boolean`` type.  Arrays are represented by
-``FutharkArray`` objects.  Complex types (records, nested tuples, etc.)
-are represented by ``FutharkOpaque`` objects.
+64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``, although a
+number is also accepted as an argument.  An integer argument must be in
+the range of its type.  ``bool`` is mapped to JavaScript's ``boolean``
+type.  Arrays are represented by ``FutharkArray`` objects.  Complex
+types (records, nested tuples, etc.) are represented by
+``FutharkOpaque`` objects.
 
 FutharkArray
 ------------
 
-Arrays are represented by ``FutharkArray`` objects.  The common array API is
-shared by the WASM backend and the WebGPU backend.
+Arrays are represented by ``FutharkArray`` objects.
 
 .. js:function:: FutharkArray.shape()
 
@@ -178,61 +153,69 @@
 
 .. js:function:: FutharkArray.values()
 
-   Asynchronously returns a flat typed array containing the array data.
-
-.. js:function:: FutharkArray.free()
-
-   Frees the memory used by the ``FutharkArray``.
-
-The WASM backend also provides a few older convenience methods.
-
-.. js:function:: FutharkArray.toArray()
-
-   Returns a nested JavaScript array.
+   Asynchronously returns a flat typed array, in row-major order,
+   containing a copy of the array elements.  It remains valid after the
+   ``FutharkArray`` is freed.  The typed array corresponds to the
+   element type, such as ``Int32Array`` for ``i32``, except that:
 
-.. js:function:: FutharkArray.toTypedArray()
+   * For ``f16``, it is a ``Float16Array`` if supported by the
+     JavaScript implementation, and otherwise a ``Float32Array``.
 
-   Returns a flat typed array of the underlying data.
+   * For ``bool``, it is a ``Uint8Array`` whose elements are 0 or 1.
 
-.. js:function:: FutharkArray.shape()
+.. js:function:: FutharkArray.free()
 
-   Returns the shape of the ``FutharkArray`` as an array of ``BigInt`` values.
+   Frees the memory used by the ``FutharkArray``.
 
 Array construction
 ~~~~~~~~~~~~~~~~~~
 
-Each array type used by an entry point is exposed through a generated
-constructor object.  For example, for the type ``[]i32``:
+Each array type used by an entry point, including as the field of a
+record, is exposed through a generated constructor object.  For
+example, for the type ``[]i32``:
 
 .. js:function:: fut.i32_1d.from_data(data, dim0)
 
    Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from
-   a JavaScript ``Array`` or the corresponding typed array, with the
-   given shape.
-
-For the WASM backend, the generated constructor object also
-provides a convenience method for nested JavaScript arrays.
-
-.. js:function:: fut.i32_1d.from_jsarray(jsarray)
+   a JavaScript ``Array`` or typed array in row-major order, with the
+   given shape.  The dimensions can be numbers or ``BigInt``\ s, and
+   their product must be the length of ``data``.  The elements must be
+   valid entry point arguments of the element type, except that for
+   ``bool``, they may also be 0 or 1 (as in the result of
+   :js:func:`FutharkArray.values`).
 
-   Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from
-   a JavaScript array.  For higher-dimensional arrays, the JavaScript
-   array is expected to be nested according to the array rank.
+The constructor object for an array type with ``n`` dimensions has a
+``from_data`` method that takes ``n`` dimensions.
 
 FutharkOpaque
 -------------
 
 Complex types (records, nested tuples, etc) are represented by
-``FutharkOpaque``.  It has no use outside of being accepted and
-returned by entry point functions. For this reason the method only has
-one function for freeing the memory when ``FutharkOpaque`` is no
-longer used.
+``FutharkOpaque``.  They can be passed to and returned by entry point
+functions, and the fields of records (including tuples) can be read
+by projection, as described below.
 
 .. js:function:: FutharkOpaque.free()
 
-   Frees  memory used by FutharkOpaque. Should be called when Futhark
-   Opaque is no longer used.
+   Frees the memory used by the ``FutharkOpaque``.
 
+Records
+~~~~~~~
+
+Records, including tuples, are ``FutharkOpaque`` objects whose fields
+can be *projected*.  For each field of a record type, the top-level
+wrapper object has a method named after the corresponding projection
+function in the C API (see :ref:`records`), without the ``futhark_``
+prefix.  For example, the field ``x`` of a record with C type ``struct
+futhark_opaque_foo`` is projected as follows:
+
+.. js:function:: fut.project_opaque_foo_x(obj)
+
+   Asynchronously returns the value of field ``x`` of the record
+   ``obj``.  An array or opaque field is returned as a new
+   ``FutharkArray`` or ``FutharkOpaque``, which must be freed
+   independently of ``obj``.
+
 Entry Points
 ------------
 
@@ -246,8 +229,13 @@
 
 .. js:function:: fut.entry.<entry_point_name>(in1, ..., inN)
 
-   The entry point function takes the JavaScript representations of the
-   Futhark entry point arguments and returns the result.
+   Asynchronously call the entry point with the JavaScript
+   representations of its arguments, and return the result.
 
-   If the entry point has multiple outputs, the return value is an array
-   containing the outputs in order.
+   An entry point that returns a tuple returns a single
+   ``FutharkOpaque``, whose elements can be obtained by projection (see
+   `Records`_).
+
+   An argument that the entry point consumes (see
+   :ref:`in-place-updates`) can afterwards only be freed, and must not
+   also be passed as another argument.
diff --git a/docs/language-reference.rst b/docs/language-reference.rst
--- a/docs/language-reference.rst
+++ b/docs/language-reference.rst
@@ -717,11 +717,13 @@
 Slicing of multiple dimensions can be done by separating with commas,
 and may be intermixed freely with indexing.
 
-If ``s`` is elided it defaults to ``1``.  If ``i`` or ``j`` is elided, their
-value depends on the sign of ``s``.  If ``s`` is positive, ``i`` become ``0``
-and ``j`` become the length of the array.  If ``s`` is negative, ``i`` becomes
-the length of the array minus one, and ``j`` becomes minus one.  This means that
-``a[::-1]`` is the reverse of the array ``a``.
+If ``s`` is elided it defaults to ``1``. If ``i`` or ``j`` is elided, their
+value depends on the sign of ``s``. If ``s`` is positive, ``i`` becomes ``0`` and
+``j`` becomes the length of the array. If ``s`` is negative, ``i`` becomes the
+length of the array minus one, and ``j`` becomes minus one. This means that
+``a[::-1]`` is the reverse of the array ``a``, and that ``a[:]`` retrieves the
+entire dimension - this is only useful when slicing multiple dimensions. Note
+that ``a[::]`` is not syntactically valid.
 
 In the general case, the size of the array produced by a slice is
 unknown (see `Size types`_).  In a few cases, the size is known
@@ -1767,12 +1769,13 @@
 attribute, written as ``#[attr]``. This may affect how it is treated by the
 compiler or other tools. In no case will attributes affect or change the
 *semantics* of a program, but it may affect how well it compiles and runs (or in
-some cases, whether it compiles or runs at all). Unknown attributes are silently
-ignored. Most have no effect in the interpreter. An attribute can be either an
-*atom*, written as an identifier or number, or *compound*, consisting of an
-identifier and a comma-separated sequence of attributes. The latter is used for
-grouping and encoding of more complex information. An identifier prefixed with a
-``$`` must be the name of a term variable in scope.
+some cases, whether it compiles or runs at all). Unknown attributes are usually
+ignored silently, but may in some cases cause a warning. Most have no effect in
+the interpreter. An attribute can be either an *atom*, written as an identifier
+or number, or *compound*, consisting of an identifier and a comma-separated
+sequence of attributes. The latter is used for grouping and encoding of more
+complex information. An identifier prefixed with a ``$`` must be the name of a
+term variable in scope.
 
 Expression attributes
 ~~~~~~~~~~~~~~~~~~~~~
@@ -1838,29 +1841,35 @@
 ``flattening(no_outer)``
 ........................
 
-When using incremental flattening, do not generate the "only outer
-parallelism" version for the attributed SOACs.
+During incremental flattening, do not generate the version that exploits only
+outermost parallelism.
 
 ``flattening(no_intra)``
 ........................
 
-When using incremental flattening, do not generate the "intra-block
-parallelism" version for the attributed SOACs.
+During incremental flattening, do not generate the version that maps outermost
+parallelism onto GPU thread blocks.
 
+``flattening(only_outer)``
+..........................
+
+During incremental flattening, only generate the version that exploits outermost
+parallelism and sequentialises inner parallelism.
+
 ``flattening(only_intra)``
 ..........................
 
-When using incremental flattening, *only* generate the "intra-block
-parallelism" version of the attributed SOACs.  **Beware**: the
-resulting program will fail to run if the inner parallelism does not
-fit on the device.
+During incremental flattening, *only* generate the version that maps outermost
+parallelism onto GPU thread blocks.
 
-``flattening(only_inner)``
+**Beware**: the resulting program will fail to run if the inner parallelism does
+not fit on the device.
+
+``flattening(only_full)``
 ..........................
 
-When using incremental flattening, do not generate multiple versions
-for this SOAC, but do exploit inner parallelism (which may give rise
-to multiple versions at deeper levels).
+During incremental flattening, generate only the fully flattened version. Any
+versioning attributes inside the SOAC are ignored.
 
 ``noinline``
 ............
@@ -1886,11 +1895,6 @@
 
 Turn the outer parallelism in the attributed SOAC sequential, but
 preserve any inner parallelism.
-
-``sequential_inner``
-....................
-
-Exploit only outer parallelism in the attributed SOAC.
 
 ``unroll``
 ..........
diff --git a/docs/man/futhark-wasm-multicore.rst b/docs/man/futhark-wasm-multicore.rst
--- a/docs/man/futhark-wasm-multicore.rst
+++ b/docs/man/futhark-wasm-multicore.rst
@@ -16,10 +16,8 @@
 
 ``futhark wasm-multicore`` translates a Futhark program to
 multi-threaded WebAssembly code by first generating C as ``futhark
-c``, and then using Emscripten (``emcc``).  This produces a ``.js``
-file that allows the compiled code to be invoked from JavaScript.
-Executables implement the Futhark server protocol and can be run with
-Node.js.
+c``, and then using Emscripten (``emcc``).  The files produced are
+the same as for :ref:`futhark-wasm(1)`.
 
 OPTIONS
 =======
@@ -35,7 +33,12 @@
 ==================
 
 Generated executables accept the same options as those generated by
-:ref:`futhark-wasm(1)`.
+:ref:`futhark-wasm(1)`.  The following additional options are
+accepted.
+
+--num-threads=INT
+
+  Use this many threads.
 
 SEE ALSO
 ========
diff --git a/docs/man/futhark-wasm.rst b/docs/man/futhark-wasm.rst
--- a/docs/man/futhark-wasm.rst
+++ b/docs/man/futhark-wasm.rst
@@ -16,9 +16,12 @@
 
 ``futhark wasm`` translates a Futhark program to sequential
 WebAssembly code by first generating C as ``futhark c``, and then
-using Emscripten (``emcc``).  This produces a ``.js`` file that allows
-the compiled code to be invoked from JavaScript.  Executables
-implement the Futhark server protocol and can be run with Node.js.
+using Emscripten (``emcc``).  For a program ``prog.fut``, this
+produces an ES module ``prog.mjs``, along with the WebAssembly module
+``prog.wasm``.  With ``--library``, the ES module exports the API
+described in :ref:`js-api`.  Otherwise, ``prog`` is an executable
+that runs the ES module with Node.js, and implements the Futhark
+server protocol.
 
 OPTIONS
 =======
@@ -31,7 +34,7 @@
 ``CFLAGS``
 
   Space-separated list of options passed to ``emcc``.  Defaults
-  to ``-O3 -std=c99`` if unset.
+  to ``-O3 -msimd128`` if unset.
 
 ``EMCFLAGS``
 
@@ -57,6 +60,27 @@
   Print various low-overhead logging information to stderr while
   running.
 
+-P, --profile
+
+  Enable the collection of profiling information.
+
+--param=ASSIGNMENT
+
+  Set a tuning parameter to the given value.  ``ASSIGNMENT`` must be of
+  the form ``NAME=INT``.
+
+--print-params
+
+  Print all tuning parameters that can be set with ``--param`` or
+  ``--tuning``.
+
+--tuning=FILE
+
+  Read size=value assignments from the given file.
+
+--cache-file=FILE
+
+  Accepted for compatibility with other backends, but has no effect.
 
 SEE ALSO
 ========
diff --git a/futhark.cabal b/futhark.cabal
--- a/futhark.cabal
+++ b/futhark.cabal
@@ -1,6 +1,6 @@
 cabal-version: 3.0
 name:           futhark
-version:        0.28.2
+version:        0.29.1
 synopsis:       An optimising compiler for a functional, array-oriented language.
 
 description:    Futhark is a small programming language designed to be compiled to
@@ -70,7 +70,7 @@
     rts/style.css
     rts/javascript/server.js
     rts/javascript/values.js
-    rts/javascript/wrapperclasses.js
+    rts/javascript/wrappers.js
     rts/opencl/copy.cl
     rts/opencl/prelude.cl
     rts/opencl/transpose.cl
diff --git a/rts/c/backends/cuda.h b/rts/c/backends/cuda.h
--- a/rts/c/backends/cuda.h
+++ b/rts/c/backends/cuda.h
@@ -215,6 +215,8 @@
   CUdeviceptr global_failure_args;
   // True if a potentially failing kernel has been enqueued.
   int32_t failure_is_an_option;
+  CUdeviceptr bookkeeping_counters;
+  CUdeviceptr bookkeeping_locks;
   int total_runs;
   long int total_runtime;
   int64_t peak_mem_usage_device;
@@ -724,6 +726,16 @@
   return 0;
 }
 
+static int zero_bookkeeping(struct futhark_context* ctx) {
+  CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_counters, 0,
+                                         sizeof(int32_t)*bookkeeping_counters_size,
+                                         ctx->stream));
+  CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_locks, 0,
+                                         sizeof(int32_t)*bookkeeping_locks_size,
+                                         ctx->stream));
+  return 0;
+}
+
 int futhark_context_sync(struct futhark_context* ctx) {
   CUDA_SUCCEED_OR_RETURN(cuCtxPushCurrent(ctx->cu_ctx));
   CUDA_SUCCEED_OR_RETURN(cuCtxSynchronize());
@@ -745,6 +757,11 @@
                                           &no_failure,
                                           sizeof(int32_t)));
 
+      // The failing kernel may have left the bookkeeping buffers nonzero.
+      if (zero_bookkeeping(ctx) != 0) {
+        return 1;
+      }
+
       int64_t args[max_failure_args+1];
       CUDA_SUCCEED_OR_RETURN(
                              cuMemcpyDtoH(&args,
@@ -856,6 +873,12 @@
   // The +1 is to avoid zero-byte allocations.
   CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1)));
 
+  CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));
+  CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));
+  if (zero_bookkeeping(ctx) != 0) {
+    return 1;
+  }
+
   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {
     return 1;
   }
@@ -868,7 +891,11 @@
     free_builtin_kernels(ctx, ctx->kernels);
     cuMemFree(ctx->global_failure);
     cuMemFree(ctx->global_failure_args);
-    CUDA_SUCCEED_FATAL(gpu_free_all(ctx));
+    cuMemFree(ctx->bookkeeping_counters);
+    cuMemFree(ctx->bookkeeping_locks);
+    if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {
+      futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);
+    }
     CUDA_SUCCEED_FATAL(cuStreamDestroy(ctx->stream));
     CUDA_SUCCEED_FATAL(cuModuleUnload(ctx->module));
     if (ctx->cfg->use_primary_context) {
diff --git a/rts/c/backends/hip.h b/rts/c/backends/hip.h
--- a/rts/c/backends/hip.h
+++ b/rts/c/backends/hip.h
@@ -182,6 +182,8 @@
   void* global_failure_args;
   // True if a potentially failing kernel has been enqueued.
   int32_t failure_is_an_option;
+  void* bookkeeping_counters;
+  void* bookkeeping_locks;
   int total_runs;
   long int total_runtime;
   int64_t peak_mem_usage_device;
@@ -578,6 +580,16 @@
   return 0;
 }
 
+static int zero_bookkeeping(struct futhark_context* ctx) {
+  HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_counters, 0,
+                                       sizeof(int32_t)*bookkeeping_counters_size,
+                                       ctx->stream));
+  HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_locks, 0,
+                                       sizeof(int32_t)*bookkeeping_locks_size,
+                                       ctx->stream));
+  return 0;
+}
+
 int futhark_context_sync(struct futhark_context* ctx) {
   HIP_SUCCEED_OR_RETURN(hipStreamSynchronize(ctx->stream));
   if (ctx->failure_is_an_option) {
@@ -596,6 +608,11 @@
                                           &no_failure,
                                           sizeof(int32_t)));
 
+      // The failing kernel may have left the bookkeeping buffers nonzero.
+      if (zero_bookkeeping(ctx) != 0) {
+        return 1;
+      }
+
       int64_t args[max_failure_args+1];
       HIP_SUCCEED_OR_RETURN(hipMemcpyDtoH(&args,
                                           ctx->global_failure_args,
@@ -685,6 +702,12 @@
   // The +1 is to avoid zero-byte allocations.
   HIP_SUCCEED_FATAL(hipMalloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1)));
 
+  HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));
+  HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));
+  if (zero_bookkeeping(ctx) != 0) {
+    return 1;
+  }
+
   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {
     return 1;
   }
@@ -697,7 +720,11 @@
     free_builtin_kernels(ctx, ctx->kernels);
     hipFree(ctx->global_failure);
     hipFree(ctx->global_failure_args);
-    HIP_SUCCEED_FATAL(gpu_free_all(ctx));
+    hipFree(ctx->bookkeeping_counters);
+    hipFree(ctx->bookkeeping_locks);
+    if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {
+      futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);
+    }
     HIP_SUCCEED_FATAL(hipStreamDestroy(ctx->stream));
     HIP_SUCCEED_FATAL(hipModuleUnload(ctx->module));
   }
diff --git a/rts/c/backends/opencl.h b/rts/c/backends/opencl.h
--- a/rts/c/backends/opencl.h
+++ b/rts/c/backends/opencl.h
@@ -449,6 +449,8 @@
   cl_mem global_failure_args;
   // True if a potentially failing kernel has been enqueued.
   cl_int failure_is_an_option;
+  cl_mem bookkeeping_counters;
+  cl_mem bookkeeping_locks;
   int total_runs;
   long int total_runtime;
   int64_t peak_mem_usage_device;
@@ -672,6 +674,19 @@
   return 0;
 }
 
+static int zero_bookkeeping(struct futhark_context* ctx) {
+  cl_int zero = 0;
+  OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_counters,
+                                               &zero, sizeof(zero),
+                                               0, sizeof(int32_t)*bookkeeping_counters_size,
+                                               0, NULL, NULL));
+  OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_locks,
+                                               &zero, sizeof(zero),
+                                               0, sizeof(int32_t)*bookkeeping_locks_size,
+                                               0, NULL, NULL));
+  return 0;
+}
+
 int futhark_context_sync(struct futhark_context* ctx) {
   // Check for any delayed error.
   cl_int failure_idx = -1;
@@ -696,6 +711,11 @@
                                                   0, sizeof(cl_int), &no_failure,
                                                   0, NULL, NULL));
 
+    // The failing kernel may have left the bookkeeping buffers nonzero.
+    if (zero_bookkeeping(ctx) != 0) {
+      return 1;
+    }
+
     int64_t args[max_failure_args+1];
     OPENCL_SUCCEED_OR_RETURN(
                              clEnqueueReadBuffer(ctx->queue,
@@ -1154,6 +1174,18 @@
                    sizeof(int64_t)*(max_failure_args+1), NULL, &error);
   OPENCL_SUCCEED_OR_RETURN(error);
 
+  ctx->bookkeeping_counters =
+    clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,
+                   sizeof(int32_t)*bookkeeping_counters_size, NULL, &error);
+  OPENCL_SUCCEED_OR_RETURN(error);
+  ctx->bookkeeping_locks =
+    clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,
+                   sizeof(int32_t)*bookkeeping_locks_size, NULL, &error);
+  OPENCL_SUCCEED_OR_RETURN(error);
+  if (zero_bookkeeping(ctx) != 0) {
+    return 1;
+  }
+
   if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) {
     return 1;
   }
@@ -1168,6 +1200,8 @@
     free_builtin_kernels(ctx, ctx->kernels);
     OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure));
     OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure_args));
+    OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_counters));
+    OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_locks));
     (void)gpu_free_all(ctx);
     (void)clReleaseProgram(ctx->clprogram);
     (void)clReleaseCommandQueue(ctx->queue);
diff --git a/rts/c/gpu.h b/rts/c/gpu.h
--- a/rts/c/gpu.h
+++ b/rts/c/gpu.h
@@ -172,6 +172,7 @@
 
 struct builtin_kernels* init_builtin_kernels(struct futhark_context* ctx) {
   struct builtin_kernels *kernels = malloc(sizeof(struct builtin_kernels));
+
   gpu_create_kernel(ctx, &kernels->map_transpose_1b, "map_transpose_1b");
   gpu_create_kernel(ctx, &kernels->map_transpose_1b_large, "map_transpose_1b_large");
   gpu_create_kernel(ctx, &kernels->map_transpose_1b_low_height, "map_transpose_1b_low_height");
@@ -375,23 +376,27 @@
 
   void* args[15];
   size_t args_sizes[15] = {
-    sizeof(gpu_mem), sizeof(int64_t),
-    sizeof(gpu_mem), sizeof(int64_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t),
-    sizeof(int32_t)
+    sizeof(int64_t), // dst_offset
+    sizeof(int64_t), // src_offset
+    sizeof(int32_t), // num_arrays
+    sizeof(int32_t), // x_elems
+    sizeof(int32_t), // y_elems
+    sizeof(int32_t), // dst_array_stride
+    sizeof(int32_t), // dst_row_stride
+    sizeof(int32_t), // src_array_stride
+    sizeof(int32_t), // src_row_stride
+    sizeof(int32_t), // mulx
+    sizeof(int32_t), // muly
+    sizeof(int32_t), // repeat_1
+    sizeof(int32_t), // repeat_2
+    sizeof(gpu_mem), // dst
+    sizeof(gpu_mem)  // src
   };
 
-  args[0] = &dst;
-  args[1] = &dst_offset;
-  args[2] = &src;
-  args[3] = &src_offset;
+  args[0] = &dst_offset;
+  args[1] = &src_offset;
+  args[13] = &dst;
+  args[14] = &src;
 
   if (dst_extent <= 2147483647L && src_extent <= 2147483647L) {
     if (m <= TR_BLOCK_DIM/2 && n <= TR_BLOCK_DIM/2) {
@@ -435,12 +440,12 @@
       block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD;
       block[2] = 1;
     }
-    args[4] = &k32;
-    args[5] = &m32;
-    args[6] = &n32;
-    for (int i = 0; i < 4; i++) { args[7+i] = &strides32[i]; }
-    args[11] = &mulx32;
-    args[12] = &muly32;
+    args[2] = &k32;
+    args[3] = &m32;
+    args[4] = &n32;
+    for (int i = 0; i < 4; i++) { args[5+i] = &strides32[i]; }
+    args[9] = &mulx32;
+    args[10] = &muly32;
   } else {
     if (ctx->logging) { fprintf(ctx->log, "Using large kernel\n"); }
     kernel = kernel_large;
@@ -450,13 +455,13 @@
     block[0] = TR_TILE_DIM;
     block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD;
     block[2] = 1;
-    args[4] = &k;
-    args[5] = &m;
-    args[6] = &n;
-    for (int i = 0; i < 4; i++) { args[7+i] = &strides[i]; }
-    args[11] = &mulx;
-    args[12] = &muly;
-    for (int i = 4; i < 13; i++) { args_sizes[i] = sizeof(int64_t); }
+    args[2] = &k;
+    args[3] = &m;
+    args[4] = &n;
+    for (int i = 0; i < 4; i++) { args[5+i] = &strides[i]; }
+    args[9] = &mulx;
+    args[10] = &muly;
+    for (int i = 2; i < 11; i++) { args_sizes[i] = sizeof(int64_t); }
   }
 
   // Cap the number of thead blocks we launch and figure out how many
@@ -465,10 +470,10 @@
   int32_t repeat_2 = grid[2] / MAX_TR_THREAD_BLOCKS;
   grid[1] = repeat_1 > 0 ? MAX_TR_THREAD_BLOCKS : grid[1];
   grid[2] = repeat_2 > 0 ? MAX_TR_THREAD_BLOCKS : grid[2];
-  args[13] = &repeat_1;
-  args[14] = &repeat_2;
-  args_sizes[13] = sizeof(repeat_1);
-  args_sizes[14] = sizeof(repeat_2);
+  args[11] = &repeat_1;
+  args[12] = &repeat_2;
+  args_sizes[11] = sizeof(repeat_1);
+  args_sizes[12] = sizeof(repeat_2);
 
   if (ctx->logging) {
     fprintf(ctx->log, "\n");
@@ -515,19 +520,20 @@
   void* args[6+(8*3)];
   size_t args_sizes[6+(8*3)];
 
-  args[0] = &dst;
-  args_sizes[0] = sizeof(gpu_mem);
-  args[1] = &dst_offset;
-  args_sizes[1] = sizeof(dst_offset);
-  args[2] = &src;
-  args_sizes[2] = sizeof(gpu_mem);
-  args[3] = &src_offset;
-  args_sizes[3] = sizeof(src_offset);
-  args[4] = &n;
-  args_sizes[4] = sizeof(n);
-  args[5] = &r;
-  args_sizes[5] = sizeof(r);
+  args[28] = &dst;
+  args_sizes[28] = sizeof(gpu_mem);
+  args[29] = &src;
+  args_sizes[29] = sizeof(gpu_mem);
 
+  args[0] = &dst_offset;
+  args_sizes[0] = sizeof(dst_offset);
+  args[1] = &src_offset;
+  args_sizes[1] = sizeof(src_offset);
+  args[2] = &n;
+  args_sizes[2] = sizeof(n);
+  args[3] = &r;
+  args_sizes[3] = sizeof(r);
+
   int64_t zero = 0;
 
   // To improve the odds of coalesced memory accesses, we pass the dimensions in
@@ -536,7 +542,10 @@
   int perm[8];
   for (int i = 0; i < r; i++) {
     int j = i;
-    while (j > 0 && llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])) {
+    while (j > 0 &&
+           (llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])
+            || (llabs(src_strides[perm[j-1]]) == llabs(src_strides[i])
+                && llabs(dst_strides[perm[j-1]]) > llabs(dst_strides[i])))) {
       perm[j] = perm[j-1];
       j--;
     }
@@ -544,17 +553,17 @@
   }
 
   for (int i = 0; i < 8; i++) {
-    args_sizes[6+i*3] = sizeof(int64_t);
-    args_sizes[6+i*3+1] = sizeof(int64_t);
-    args_sizes[6+i*3+2] = sizeof(int64_t);
+    args_sizes[4+i*3] = sizeof(int64_t);
+    args_sizes[4+i*3+1] = sizeof(int64_t);
+    args_sizes[4+i*3+2] = sizeof(int64_t);
     if (i < r) {
-      args[6+i*3] = &shape[perm[i]];
-      args[6+i*3+1] = &dst_strides[perm[i]];
-      args[6+i*3+2] = &src_strides[perm[i]];
+      args[4+i*3] = &shape[perm[i]];
+      args[4+i*3+1] = &dst_strides[perm[i]];
+      args[4+i*3+2] = &src_strides[perm[i]];
     } else {
-      args[6+i*3] = &zero;
-      args[6+i*3+1] = &zero;
-      args[6+i*3+2] = &zero;
+      args[4+i*3] = &zero;
+      args[4+i*3+1] = &zero;
+      args[4+i*3+2] = &zero;
     }
   }
   const size_t w = 256; // XXX: hardcoded thread block size.
diff --git a/rts/cuda/prelude.cu b/rts/cuda/prelude.cu
--- a/rts/cuda/prelude.cu
+++ b/rts/cuda/prelude.cu
@@ -128,7 +128,9 @@
 #define INFINITY (1.0/0.0)
 #endif
 
-extern volatile __shared__ unsigned char shared_mem[];
+// shared_mem itself must be 8-byte aligned no matter what, in case we access it
+// with atomics.
+extern volatile __shared__ __attribute__((aligned(8))) unsigned char shared_mem[];
 
 #define SHARED_MEM_PARAM
 #define FUTHARK_KERNEL extern "C" __global__ __launch_bounds__(MAX_THREADS_PER_BLOCK)
diff --git a/rts/javascript/server.js b/rts/javascript/server.js
--- a/rts/javascript/server.js
+++ b/rts/javascript/server.js
@@ -1,8 +1,8 @@
 // Start of server.js
 
-// The server is implemented directly on top of the C API exported by the
-// WebAssembly module, much like the C server.  The manifest describes the
-// types and entry points, and names the C functions that operate on them.
+// The server is implemented in terms of the C API exported by the WebAssembly
+// module. The manifest describes the types and entry points, and names the C
+// functions that operate on them.
 //
 // A variable is a pair of a type name and a value.  A value of primitive type
 // is a JavaScript number, or a BigInt for 64-bit integers.  Booleans are 0 or
@@ -38,8 +38,13 @@
     this.wasm = ctx.wasm;
     this.manifest = manifest;
     this._vars = {};
+    this._output = [];
   }
 
+  _print(line) {
+    this._output.push(line);
+  }
+
   _get_arg(args, i) {
     if (i < args.length) {
       return args[i];
@@ -99,18 +104,33 @@
 
   // Calling into the C API.
 
-  _c(fname, ...args) {
-    return this.wasm['_' + fname](this.ctx.ctx, ...args);
+  async _c(fname, ...args) {
+    return await this.wasm.ccall(fname, 'number',
+      Array(1 + args.length).fill('number'),
+      [this.ctx.ctx, ...args], {async: true});
   }
 
+  // Read a NUL-terminated string allocated by C, and free it.
+  _c_string(p) {
+    var heap = this.wasm.HEAPU8;
+    // Browsers' TextDecoder rejects shared memory.
+    var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));
+    this.wasm._free(p);
+    return str;
+  }
+
+  _get_error() {
+    return this._c_string(this.wasm._futhark_context_get_error(this.ctx.ctx));
+  }
+
   _check(err) {
     if (err != 0) {
-      throw this.ctx.get_error();
+      throw this._get_error();
     }
   }
 
-  _sync() {
-    this._check(this.wasm._futhark_context_sync(this.ctx.ctx));
+  async _sync() {
+    this._check(await this._c('futhark_context_sync'));
   }
 
   _malloc(n) {
@@ -141,10 +161,10 @@
 
   // Call 'f' with a pointer to space for one value of the given type, and
   // return the value that it stores there.
-  _with_out(tname, f) {
+  async _with_out(tname, f) {
     var p = this._malloc(this._sizeof(tname));
     try {
-      f(p);
+      await f(p);
       return this._load(tname, p);
     } finally {
       this.wasm._free(p);
@@ -152,9 +172,19 @@
   }
 
   _bytes(p, n) {
-    return Buffer.from(this.wasm.HEAPU8.slice(p, p + n));
+    return this.wasm.HEAPU8.slice(p, p + n);
   }
 
+  async _read_file(fname) {
+    const fs = await import('node:fs');
+    return new Uint8Array(fs.readFileSync(fname));
+  }
+
+  async _write_file(fname, bytes) {
+    const fs = await import('node:fs');
+    fs.writeFileSync(fname, bytes);
+  }
+
   // Types.
 
   _type(tname) {
@@ -210,16 +240,16 @@
     return this._type(tname).sum;
   }
 
-  _free_value(tname, value) {
+  async _free_value(tname, value) {
     var t = this._type(tname);
     if (t !== null) {
-      this._check(this._c(t.ops.free, value));
+      this._check(await this._c(t.ops.free, value));
     }
   }
 
-  _shape(a, arr) {
+  async _shape(a, arr) {
     var fshape = a.ops ? a.ops.shape : a.shape;
-    var p = this._c(fshape, arr);
+    var p = await this._c(fshape, arr);
     var shape = [];
     for (var i = 0; i < a.rank; i++) {
       shape.push(this._load('i64', p + i * 8));
@@ -237,7 +267,7 @@
 
   // Values.
 
-  _restore_value(reader, tname) {
+  async _restore_value(reader, tname) {
     var t = this._type(tname);
     if (t === null) {
       return read_value(tname, reader);
@@ -247,11 +277,11 @@
       var p = this._malloc(bytes.length);
       try {
         this.wasm.HEAPU8.set(bytes, p);
-        var arr = this._c(t.ops.new, p, ...shape);
+        var arr = await this._c(t.ops.new, p, ...shape);
         if (arr == 0) {
-          throw this.ctx.get_error();
+          throw this._get_error();
         }
-        this._sync();
+        await this._sync();
         return arr;
       } finally {
         this.wasm._free(p);
@@ -263,22 +293,22 @@
       var p = this._malloc(buff.length);
       try {
         this.wasm.HEAPU8.set(buff, p);
-        var obj = this._c(t.ops.restore, p);
+        var obj = await this._c(t.ops.restore, p);
         if (obj == 0) {
-          throw this.ctx.get_error();
+          throw this._get_error();
         }
       } finally {
         this.wasm._free(p);
       }
-      var n = this._with_out('size', (np) => {
-        this._check(this._c(t.ops.store, obj, 0, np));
+      var n = await this._with_out('size', async (np) => {
+        this._check(await this._c(t.ops.store, obj, 0, np));
       });
       reader.buff = buff.slice(n);
       return obj;
     }
   }
 
-  _store_value(tname, value) {
+  async _store_value(tname, value) {
     var t = this._type(tname);
     if (t === null) {
       var p = this._malloc(this._sizeof(tname));
@@ -289,12 +319,12 @@
         this.wasm._free(p);
       }
     } else if (t.kind == "array") {
-      var shape = this._shape(this._array_type(tname), value);
+      var shape = await this._shape(this._array_type(tname), value);
       var n = shape.reduce((x, y) => x * y, 1n) * BigInt(this._sizeof(t.elemtype));
       var p = this._malloc(Number(n));
       try {
-        this._check(this._c(t.ops.values, value, p));
-        this._sync();
+        this._check(await this._c(t.ops.values, value, p));
+        await this._sync();
         return binary_value(t.elemtype, shape, this._bytes(p, Number(n)));
       } finally {
         this.wasm._free(p);
@@ -304,8 +334,8 @@
       var np = pp + serverPointer.size;
       try {
         this._poke('ptr', pp, 0);
-        this._check(this._c(t.ops.store, value, pp, np));
-        this._sync();
+        this._check(await this._c(t.ops.store, value, pp, np));
+        await this._sync();
         var p = this._load('ptr', pp);
         var bytes = this._bytes(p, this._load('size', np));
         this.wasm._free(p);
@@ -321,23 +351,35 @@
   _cmd_inputs(args) {
     var entry = this._get_entry_point(this._get_arg(args, 0));
     for (var input of entry.inputs) {
-      console.log((input.consumed ? "*" : "") + input.type);
+      this._print((input.consumed ? "*" : "") + input.type);
     }
   }
 
   _cmd_output(args) {
     var entry = this._get_entry_point(this._get_arg(args, 0));
-    console.log((entry.output.fresh ? "*" : "") + entry.output.type);
+    this._print((entry.output.fresh ? "*" : "") + entry.output.type);
   }
 
-  _cmd_dummy(args) {
-    // pass
+  async _cmd_clear(args) {
+    this._check(await this._c('futhark_context_clear_caches'));
   }
 
-  _cmd_free(args) {
+  async _cmd_pause_profiling(args) {
+    await this._c('futhark_context_pause_profiling');
+  }
+
+  async _cmd_unpause_profiling(args) {
+    await this._c('futhark_context_unpause_profiling');
+  }
+
+  async _cmd_report(args) {
+    this._print(this._c_string(await this._c('futhark_context_report')));
+  }
+
+  async _cmd_free(args) {
     for (var vname of args) {
       var v = this._get_var(vname);
-      this._free_value(v.type, v.value);
+      await this._free_value(v.type, v.value);
       delete this._vars[vname];
     }
   }
@@ -353,27 +395,27 @@
 
   _cmd_types(args) {
     for (var t in serverPrimtypes) {
-      console.log(t);
+      this._print(t);
     }
     for (var t in this.manifest.types) {
-      console.log(t);
+      this._print(t);
     }
   }
 
   _cmd_entry_points(args) {
     for (var e in this.manifest.entry_points) {
-      console.log(e);
+      this._print(e);
     }
   }
 
   _cmd_attributes(args) {
     var entry = this._get_entry_point(this._get_arg(args, 0));
     for (var attr of entry.attributes) {
-      console.log(attr);
+      this._print(attr);
     }
   }
 
-  _cmd_call(args) {
+  async _cmd_call(args) {
     var entry = this._get_entry_point(this._get_arg(args, 0));
     var out_vname = this._get_arg(args, 1);
     var in_vnames = args.slice(2);
@@ -384,39 +426,39 @@
     this._check_new_var(out_vname);
 
     var bef = performance.now() * 1000;
-    var out = this._with_out(entry.output.type, (p) => {
-      this._check(this._c(entry.cfun, p, ...ins));
-      this._sync();
+    var out = await this._with_out(entry.output.type, async (p) => {
+      this._check(await this._c(entry.cfun, p, ...ins));
+      await this._sync();
     });
     var aft = performance.now() * 1000;
-    console.log("runtime: " + Math.round(aft - bef));
+    this._print("runtime: " + Math.round(aft - bef));
     this._set_var(out_vname, entry.output.type, out);
   }
 
-  _cmd_store(args) {
+  async _cmd_store(args) {
     var fname = this._get_arg(args, 0);
     var bufs = [];
     for (var vname of args.slice(1)) {
       var v = this._get_var(vname);
-      bufs.push(this._store_value(v.type, v.value));
+      bufs.push(await this._store_value(v.type, v.value));
     }
-    require("fs").writeFileSync(fname, Buffer.concat(bufs));
+    await this._write_file(fname, concat_bytes(bufs));
   }
 
-  _cmd_restore(args) {
+  async _cmd_restore(args) {
     var fname = this._get_arg(args, 0);
     if (args.length % 2 == 0) {
       throw "Invalid argument count";
     }
 
-    var reader = new Reader(fname);
+    var reader = new Reader(await this._read_file(fname));
     for (var i = 1; i < args.length; i += 2) {
       var vname = args[i];
       var tname = args[i + 1];
       this._check_new_var(vname);
       this._type(tname);
       try {
-        this._set_var(vname, tname, this._restore_value(reader, tname));
+        this._set_var(vname, tname, await this._restore_value(reader, tname));
       } catch (err) {
         throw "Failed to restore variable " + vname + ".\nPossibly malformed data in " + fname + ".\n" + err.toString();
       }
@@ -428,29 +470,29 @@
   }
 
   _cmd_kind(args) {
-    console.log(this._kind(this._get_arg(args, 0)));
+    this._print(this._kind(this._get_arg(args, 0)));
   }
 
   _cmd_type(args) {
-    console.log(this._get_var(this._get_arg(args, 0)).type);
+    this._print(this._get_var(this._get_arg(args, 0)).type);
   }
 
   _cmd_rank(args) {
-    console.log(this._array_type(this._get_arg(args, 0)).rank);
+    this._print(this._array_type(this._get_arg(args, 0)).rank);
   }
 
   _cmd_elemtype(args) {
-    console.log(this._array_type(this._get_arg(args, 0)).elemtype);
+    this._print(this._array_type(this._get_arg(args, 0)).elemtype);
   }
 
-  _cmd_shape(args) {
+  async _cmd_shape(args) {
     var v = this._get_var(this._get_arg(args, 0));
-    for (var d of this._shape(this._array_type(v.type), v.value)) {
-      console.log(d.toString());
+    for (var d of await this._shape(this._array_type(v.type), v.value)) {
+      this._print(d.toString());
     }
   }
 
-  _cmd_new_array(args) {
+  async _cmd_new_array(args) {
     var dst = this._get_arg(args, 0);
     var tname = this._get_arg(args, 1);
     this._check_new_var(dst);
@@ -469,49 +511,49 @@
       vs.forEach((v, i) => this._poke(a.elemtype, p + i * size, v));
       var arr;
       if (a.ops) {
-        arr = this._c(a.ops.new, p, ...dims);
+        arr = await this._c(a.ops.new, p, ...dims);
         if (arr == 0) {
-          throw this.ctx.get_error();
+          throw this._get_error();
         }
       } else {
-        arr = this._with_out(tname, (out) => {
-          this._check(this._c(a.new, out, p, ...dims));
+        arr = await this._with_out(tname, async (out) => {
+          this._check(await this._c(a.new, out, p, ...dims));
         });
       }
-      this._sync();
+      await this._sync();
       this._set_var(dst, tname, arr);
     } finally {
       this.wasm._free(p);
     }
   }
 
-  _cmd_set(args) {
+  async _cmd_set(args) {
     var arr = this._get_var(this._get_arg(args, 0));
     var a = this._array_type(arr.type);
     var val = this._get_typed_var(this._get_arg(args, 1), a.elemtype);
     var is = this._parse_ints(args.slice(2), a.rank);
-    this._check_bounds(this._shape(a, arr.value), is);
+    this._check_bounds(await this._shape(a, arr.value), is);
     var fset = a.ops ? a.ops.set : a.set;
-    this._check(this._c(fset, arr.value, val, ...is));
-    this._sync();
+    this._check(await this._c(fset, arr.value, val, ...is));
+    await this._sync();
   }
 
-  _cmd_index(args) {
+  async _cmd_index(args) {
     var dst = this._get_arg(args, 0);
     var arr = this._get_var(this._get_arg(args, 1));
     this._check_new_var(dst);
     var a = this._array_type(arr.type);
     var is = this._parse_ints(args.slice(2), a.rank);
-    this._check_bounds(this._shape(a, arr.value), is);
+    this._check_bounds(await this._shape(a, arr.value), is);
     var findex = a.ops ? a.ops.index : a.index;
-    var v = this._with_out(a.elemtype, (p) => {
-      this._check(this._c(findex, p, arr.value, ...is));
-      this._sync();
+    var v = await this._with_out(a.elemtype, async (p) => {
+      this._check(await this._c(findex, p, arr.value, ...is));
+      await this._sync();
     });
     this._set_var(dst, a.elemtype, v);
   }
 
-  _cmd_zip(args) {
+  async _cmd_zip(args) {
     var dst = this._get_arg(args, 0);
     var tname = this._get_arg(args, 1);
     this._check_new_var(dst);
@@ -524,14 +566,14 @@
       throw a.fields.length + " arrays expected but " + vnames.length + " values provided.";
     }
     var vs = vnames.map((v, i) => this._get_typed_var(v, a.fields[i].type));
-    var arr = this._with_out(tname, (p) => {
-      this._check(this._c(a.zip, p, ...vs));
-      this._sync();
+    var arr = await this._with_out(tname, async (p) => {
+      this._check(await this._c(a.zip, p, ...vs));
+      await this._sync();
     });
     this._set_var(dst, tname, arr);
   }
 
-  _cmd_unzip(args) {
+  async _cmd_unzip(args) {
     var arr = this._get_var(this._get_arg(args, 0));
     var a = this._array_type(arr.type);
     if (!a.fields) {
@@ -542,22 +584,23 @@
       throw a.fields.length + " arrays expected but " + dsts.length + " values provided.";
     }
     dsts.forEach((dst) => this._check_new_var(dst));
-    a.fields.forEach((f, i) => {
-      var v = this._with_out(f.type, (p) => {
-        this._check(this._c(f.project, p, arr.value));
+    for (var i = 0; i < a.fields.length; i++) {
+      var f = a.fields[i];
+      var v = await this._with_out(f.type, async (p) => {
+        this._check(await this._c(f.project, p, arr.value));
       });
       this._set_var(dsts[i], f.type, v);
-    });
-    this._sync();
+    }
+    await this._sync();
   }
 
   _cmd_fields(args) {
     for (var f of this._record_type(this._get_arg(args, 0)).fields) {
-      console.log(f.name + " " + f.type);
+      this._print(f.name + " " + f.type);
     }
   }
 
-  _cmd_new(args) {
+  async _cmd_new(args) {
     var dst = this._get_arg(args, 0);
     var tname = this._get_arg(args, 1);
     this._check_new_var(dst);
@@ -567,13 +610,13 @@
       throw r.fields.length + " fields expected but " + vnames.length + " values provided.";
     }
     var vs = vnames.map((v, i) => this._get_typed_var(v, r.fields[i].type));
-    var obj = this._with_out(tname, (p) => {
-      this._check(this._c(r.new, p, ...vs));
+    var obj = await this._with_out(tname, async (p) => {
+      this._check(await this._c(r.new, p, ...vs));
     });
     this._set_var(dst, tname, obj);
   }
 
-  _cmd_project(args) {
+  async _cmd_project(args) {
     var dst = this._get_arg(args, 0);
     var from = this._get_var(this._get_arg(args, 1));
     var field = this._get_arg(args, 2);
@@ -582,31 +625,31 @@
     if (f === undefined) {
       throw "No such field: " + field;
     }
-    var v = this._with_out(f.type, (p) => {
-      this._check(this._c(f.project, p, from.value));
+    var v = await this._with_out(f.type, async (p) => {
+      this._check(await this._c(f.project, p, from.value));
     });
     this._set_var(dst, f.type, v);
   }
 
   _cmd_variants(args) {
     for (var v of this._sum_type(this._get_arg(args, 0)).variants) {
-      console.log(v.name);
+      this._print(v.name);
       for (var t of v.payload) {
-        console.log("- " + t);
+        this._print("- " + t);
       }
     }
   }
 
-  _variant_of(v) {
+  async _variant_of(v) {
     var s = this._sum_type(v.type);
-    return s.variants[this._c(s.variant, v.value)];
+    return s.variants[await this._c(s.variant, v.value)];
   }
 
-  _cmd_variant(args) {
-    console.log(this._variant_of(this._get_var(this._get_arg(args, 0))).name);
+  async _cmd_variant(args) {
+    this._print((await this._variant_of(this._get_var(this._get_arg(args, 0)))).name);
   }
 
-  _cmd_construct(args) {
+  async _cmd_construct(args) {
     var dst = this._get_arg(args, 0);
     var tname = this._get_arg(args, 1);
     var vname = this._get_arg(args, 2);
@@ -620,15 +663,15 @@
       throw variant.payload.length + " values expected but " + vnames.length + " provided.";
     }
     var vs = vnames.map((v, i) => this._get_typed_var(v, variant.payload[i]));
-    var obj = this._with_out(tname, (p) => {
-      this._check(this._c(variant.construct, p, ...vs));
+    var obj = await this._with_out(tname, async (p) => {
+      this._check(await this._c(variant.construct, p, ...vs));
     });
     this._set_var(dst, tname, obj);
   }
 
-  _cmd_destruct(args) {
+  async _cmd_destruct(args) {
     var v = this._get_var(this._get_arg(args, 0));
-    var variant = this._variant_of(v);
+    var variant = await this._variant_of(v);
     var dsts = args.slice(1);
     if (dsts.length != variant.payload.length) {
       throw variant.payload.length + " variables expected but " + dsts.length + " provided.";
@@ -636,75 +679,82 @@
     dsts.forEach((dst) => this._check_new_var(dst));
     var ps = variant.payload.map((t) => this._malloc(this._sizeof(t)));
     try {
-      this._check(this._c(variant.destruct, ...ps, v.value));
+      this._check(await this._c(variant.destruct, ...ps, v.value));
       variant.payload.forEach((t, i) => this._set_var(dsts[i], t, this._load(t, ps[i])));
     } finally {
       ps.forEach((p) => this.wasm._free(p));
     }
   }
 
-  _process_line(line) {
+  async _process_command(cmd, args) {
+    switch (cmd) {
+    case 'inputs': return this._cmd_inputs(args);
+    case 'output': return this._cmd_output(args);
+    case 'call': return this._cmd_call(args);
+    case 'restore': return this._cmd_restore(args);
+    case 'store': return this._cmd_store(args);
+    case 'free': return this._cmd_free(args);
+    case 'clear': return this._cmd_clear(args);
+    case 'pause_profiling': return this._cmd_pause_profiling(args);
+    case 'unpause_profiling': return this._cmd_unpause_profiling(args);
+    case 'report': return this._cmd_report(args);
+    case 'rename': return this._cmd_rename(args);
+    case 'types': return this._cmd_types(args);
+    case 'entry_points': return this._cmd_entry_points(args);
+    case 'attributes': return this._cmd_attributes(args);
+    case 'kind': return this._cmd_kind(args);
+    case 'type': return this._cmd_type(args);
+    case 'rank': return this._cmd_rank(args);
+    case 'elemtype': return this._cmd_elemtype(args);
+    case 'shape': return this._cmd_shape(args);
+    case 'new_array': return this._cmd_new_array(args);
+    case 'set': return this._cmd_set(args);
+    case 'index': return this._cmd_index(args);
+    case 'zip': return this._cmd_zip(args);
+    case 'unzip': return this._cmd_unzip(args);
+    case 'fields': return this._cmd_fields(args);
+    case 'new': return this._cmd_new(args);
+    case 'project': return this._cmd_project(args);
+    case 'variants': return this._cmd_variants(args);
+    case 'construct': return this._cmd_construct(args);
+    case 'destruct': return this._cmd_destruct(args);
+    case 'variant': return this._cmd_variant(args);
+    default: throw "Unknown command: " + cmd;
+    }
+  }
+
+  async _process_line(line) {
     var words = split_words(line);
     if (words.length == 0) {
       throw "Empty line";
-    } else {
-      var cmd = words[0];
-      var args = words.slice(1);
-      switch (cmd) {
-      case 'inputs': this._cmd_inputs(args); break;
-      case 'output': this._cmd_output(args); break;
-      case 'call': this._cmd_call(args); break;
-      case 'restore': this._cmd_restore(args); break;
-      case 'store': this._cmd_store(args); break;
-      case 'free': this._cmd_free(args); break;
-      case 'clear': this._cmd_dummy(args); break;
-      case 'pause_profiling': this._cmd_dummy(args); break;
-      case 'unpause_profiling': this._cmd_dummy(args); break;
-      case 'report': this._cmd_dummy(args); break;
-      case 'rename': this._cmd_rename(args); break;
-      case 'types': this._cmd_types(args); break;
-      case 'entry_points': this._cmd_entry_points(args); break;
-      case 'attributes': this._cmd_attributes(args); break;
-      case 'kind': this._cmd_kind(args); break;
-      case 'type': this._cmd_type(args); break;
-      case 'rank': this._cmd_rank(args); break;
-      case 'elemtype': this._cmd_elemtype(args); break;
-      case 'shape': this._cmd_shape(args); break;
-      case 'new_array': this._cmd_new_array(args); break;
-      case 'set': this._cmd_set(args); break;
-      case 'index': this._cmd_index(args); break;
-      case 'zip': this._cmd_zip(args); break;
-      case 'unzip': this._cmd_unzip(args); break;
-      case 'fields': this._cmd_fields(args); break;
-      case 'new': this._cmd_new(args); break;
-      case 'project': this._cmd_project(args); break;
-      case 'variants': this._cmd_variants(args); break;
-      case 'construct': this._cmd_construct(args); break;
-      case 'destruct': this._cmd_destruct(args); break;
-      case 'variant': this._cmd_variant(args); break;
-      default: throw "Unknown command: " + cmd;
-      }
     }
+    await this._process_command(words[0], words.slice(1));
   }
 
-  run() {
+  async run() {
     console.log('%%% OK'); // TODO figure out if flushing is neccesary for JS
-    const readline = require('readline');
+    const readline = await import('node:readline');
     const rl = readline.createInterface(process.stdin);
-    rl.on('line', (line) => {
+    // Commands are asynchronous, so we must finish one before reading the
+    // next.
+    for await (const line of rl) {
       if (line == "") {
-        rl.close();
-        return;
+        break;
       }
+      this._output = [];
       try {
-        this._process_line(line);
+        await this._process_line(line);
+        this._output.forEach((l) => console.log(l));
         console.log('%%% OK');
       } catch (err) {
         console.log('%%% FAILURE');
+        this._output.forEach((l) => console.log(l));
         console.log(err);
         console.log('%%% OK');
       }
-    }).on('close', () => { process.exit(0); });
+    }
+    this.ctx.free();
+    process.exit(0);
   }
 }
 
@@ -726,6 +776,142 @@
     throw "Unterminated quote";
   }
   return words;
+}
+
+// Run a server as the main program of a Node.js process.
+async function futhark_server_main() {
+  const fs = await import('node:fs');
+  const progname = process.argv[1];
+  const fail = (msg) => {
+    console.error(progname + ": " + msg);
+    process.exit(1);
+  };
+  const parse_int = (what, x) => {
+    if (!/^[0-9]+$/.test(x)) {
+      fail("Invalid " + what + ": " + x);
+    }
+    return Number(x);
+  };
+
+  const options = { tuning_params: {} };
+  var print_params = false;
+  // Long name, short name, argument, description, and action.
+  const table = [
+    ["debugging", "D", null,
+     "Perform possibly expensive internal correctness checks and verbose logging.",
+     () => { options.debugging = true; }],
+    ["log", "L", null,
+     "Print various low-overhead logging information while running.",
+     () => { options.logging = true; }],
+    ["profile", "P", null,
+     "Enable the collection of profiling information.",
+     () => { options.profiling = true; }],
+    ["help", "h", null,
+     "Print help information and exit.",
+     () => {
+       console.log("Usage: " + progname + " [OPTIONS]...\nOptions:\n");
+       for (const [long, short, arg, desc] of table) {
+         console.log("  " + (short ? "-" + short + "/" : "") + "--" + long +
+                     (arg ? " " + arg : "") + "\n    " + desc);
+       }
+       console.log("\nFor more information, consult the Futhark User's Guide or the man pages.");
+       process.exit(0);
+     }],
+    ["print-params", null, null,
+     "Print all tuning parameters that can be set with --param or --tuning.",
+     () => { print_params = true; }],
+    ["param", null, "ASSIGNMENT",
+     "Set a tuning parameter to the given value.",
+     (x) => {
+       const eq = x.indexOf("=");
+       if (eq < 0) {
+         fail("Invalid argument for --param option: " + x);
+       }
+       options.tuning_params[x.slice(0, eq)] = parse_int("tuning parameter value", x.slice(eq + 1));
+     }],
+    ["tuning", null, "FILE",
+     "Read size=value assignments from the given file.",
+     (fname) => {
+       var lines;
+       try {
+         lines = fs.readFileSync(fname, "utf8").split("\n");
+       } catch (e) {
+         fail("When loading tuning file '" + fname + "': Cannot open file: " + e.message);
+       }
+       lines.forEach((line, i) => {
+         if (/^[ \t]*($|--)/.test(line)) {
+           return;
+         }
+         const m = /^([^=]*)=(-?[0-9]+)$/.exec(line);
+         if (m === null) {
+           fail("When loading tuning file '" + fname + "': Invalid line " + (i + 1) +
+                " (must be of form 'name=int').");
+         }
+         options.tuning_params[m[1]] = Number(m[2]);
+       });
+     }],
+    // There is nothing to cache.
+    ["cache-file", null, "FILE",
+     "Store program cache here (has no effect for this backend).",
+     () => {}],
+  ];
+  if (futhark_manifest.backend == "wasm_multicore") {
+    table.push(["num-threads", null, "INT",
+                "Set number of threads used for execution.",
+                (x) => { options.num_threads = parse_int("number of threads", x); }]);
+  }
+
+  const args = process.argv.slice(2);
+  for (var i = 0; i < args.length; i++) {
+    const a = args[i];
+    if (a.startsWith("--")) {
+      const eq = a.indexOf("=");
+      const name = eq < 0 ? a.slice(2) : a.slice(2, eq);
+      const opt = table.find((o) => o[0] == name);
+      if (opt === undefined) {
+        fail("unrecognized option '" + a + "'");
+      }
+      var arg;
+      if (opt[2] !== null) {
+        arg = eq < 0 ? args[++i] : a.slice(eq + 1);
+        if (arg === undefined) {
+          fail("option '--" + name + "' requires an argument");
+        }
+      } else if (eq >= 0) {
+        fail("option '--" + name + "' doesn't allow an argument");
+      }
+      opt[4](arg);
+    } else if (a.startsWith("-") && a.length > 1) {
+      for (const c of a.slice(1)) {
+        const opt = table.find((o) => o[1] == c);
+        if (opt === undefined) {
+          fail("invalid option -- '" + c + "'");
+        }
+        opt[4]();
+      }
+    } else {
+      fail("unexpected argument: " + a);
+    }
+  }
+
+  if (print_params) {
+    const m = await loadWASM();
+    const heap = () => m.HEAPU8;
+    const str = (p) => new TextDecoder().decode(heap().slice(p, heap().indexOf(0, p)));
+    for (var j = 0; j < m._futhark_get_tuning_param_count(); j++) {
+      console.log(str(m._futhark_get_tuning_param_name(j)) + " (" +
+                  str(m._futhark_get_tuning_param_class(j)) + ")");
+    }
+    process.exit(0);
+  }
+
+  const context = new FutharkContext();
+  try {
+    await context.init(options);
+  } catch (e) {
+    fail(e.message);
+  }
+  await new Server(context, futhark_manifest).run();
 }
 
 // End of server.js
diff --git a/rts/javascript/values.js b/rts/javascript/values.js
--- a/rts/javascript/values.js
+++ b/rts/javascript/values.js
@@ -1,4 +1,8 @@
 // Start of values.js
+//
+// Reading and writing values in the binary data format.  Values are read from
+// and written to byte buffers (Uint8Arrays).
+
 var futharkPrimtypes =
   new Set([
     'i8',
@@ -29,19 +33,6 @@
                   'bool' : Uint8Array
                 };
 
-function binToStringArray(buff, array) {
-  for (var i = 0; i < array.length; i++) {
-    array[i] = buff[i];
-  }
-}
-
-function fileToBuff(fname) {
-  var readline = require('readline');
-  var fs = require('fs');
-  var buff =  fs.readFileSync(fname);
-  return buff;
-}
-
 var typToSize = {
   "bool" : 1,
   "  u8" : 1,
@@ -61,6 +52,16 @@
   return new Uint8Array(ta.buffer, ta.byteOffset, ta.byteLength);
 }
 
+function concat_bytes(arrs) {
+  var out = new Uint8Array(arrs.reduce((n, a) => n + a.length, 0));
+  var off = 0;
+  for (var a of arrs) {
+    out.set(a, off);
+    off += a.length;
+  }
+  return out;
+}
+
 // The binary encoding of a value with the given primitive type and shape (a
 // list of BigInts), where 'bytes' is the payload.
 function binary_value(typ, shape, bytes) {
@@ -73,65 +74,59 @@
     header[3+i] = ftype.charCodeAt(i);
   }
   var dims = new BigInt64Array(shape);
-  return Buffer.concat([header, toU8(dims), bytes]);
+  return concat_bytes([header, toU8(dims), bytes]);
 }
 
+function is_space(b) {
+  return b == 32 || (b >= 9 && b <= 13);
+}
+
 class Reader {
-  constructor(f) {
-    this.f = f;
-    this.buff = fileToBuff(f);
+  constructor(buff) {
+    this.buff = buff;
   }
 
-  read_bin_array(num_dim, typ) {
-    var u8_array = new Uint8Array(num_dim * 8);
-    binToStringArray(this.buff.slice(0, num_dim * 8), u8_array);
-    var shape = new BigInt64Array(u8_array.buffer);
-    var length = shape[0];
-    for (var i = 1; i < shape.length; i++) {
-      length = length * shape[i];
+  // Remove the first n bytes from the buffer, returning a copy of them.
+  take(n) {
+    if (this.buff.length < n) {
+      throw "Unexpected end of input";
     }
-    length = Number(length);
-    var dbytes = typToSize[typ];
-    var u8_data = new Uint8Array(length * dbytes);
-    binToStringArray(this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length), u8_data);
-    var data  = new (typToType[typ])(u8_data.buffer);
-    var tmp_buff = this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length);
-    this.buff = this.buff.slice(num_dim * 8 + dbytes * length);
-   return [shape, data];
+    var bytes = this.buff.slice(0, n);
+    this.buff = this.buff.subarray(n);
+    return bytes;
   }
 
+  read_bin_array(num_dim, typ) {
+    var shape = new BigInt64Array(this.take(num_dim * 8).buffer);
+    var length = Number(shape.reduce((x, y) => x * y, 1n));
+    var data = new (typToType[typ])(this.take(length * typToSize[typ]).buffer);
+    return [shape, data];
+  }
+
   read_bin_scalar(typ) {
-    var size = typToSize[typ];
-    var u8_array = new Uint8Array(size);
-    binToStringArray(this.buff, u8_array);
-    var array = new (typToType[typ])(u8_array.buffer);
-    this.buff = this.buff.slice(u8_array.length); // Update buff to be unread part of the string
-    return array[0];
+    return new (typToType[typ])(this.take(typToSize[typ]).buffer)[0];
   }
 
   skip_spaces() {
-    while (this.buff.length > 0 && this.buff.slice(0, 1).toString().trim() == "") {
-      this.buff = this.buff.slice(1);
+    while (this.buff.length > 0 && is_space(this.buff[0])) {
+      this.buff = this.buff.subarray(1);
     }
   }
 
   read_binary(typename, dim) {
-    // Skip leading white space
-    while (this.buff.slice(0, 1).toString().trim() == "") {
-      this.buff = this.buff.slice(1);
-    }
-    if (this.buff[0] != 'b'.charCodeAt(0)) {
+    this.skip_spaces();
+    var header = this.take(7);
+    if (header[0] != 'b'.charCodeAt(0)) {
       throw "Not in binary format"
     }
-    var version = this.buff[1];
+    var version = header[1];
     if (version != 2) {
       throw "Not version 2";
     }
-    var num_dim = this.buff[2];
-    var typ = this.buff.slice(3, 7);
-    this.buff = this.buff.slice(7);
+    var num_dim = header[2];
+    var typ = String.fromCharCode(...header.subarray(3, 7));
     var exp_typ = "[]".repeat(dim) + typename;
-    var given_typ = "[]".repeat(num_dim) + typ.toString().trim();
+    var given_typ = "[]".repeat(num_dim) + typ.trim();
     if (exp_typ !== given_typ) {
       throw ("Expected type : " + exp_typ + ", Actual type : " + given_typ);
     }
diff --git a/rts/javascript/wrapperclasses.js b/rts/javascript/wrapperclasses.js
deleted file mode 100644
--- a/rts/javascript/wrapperclasses.js
+++ /dev/null
@@ -1,91 +0,0 @@
-// Start of wrapperclasses.js
-
-class FutharkArray {
-  constructor(ctx, ptr, type_name, dim, heap, fshape, fvalues, ffree) {
-    this.ctx = ctx;
-    this.ptr = ptr;
-    this.type_name = type_name;
-    this.dim = dim;
-    this.heap = heap;
-    this.fshape = fshape;
-    this.fvalues = fvalues;
-    this.ffree = ffree;
-    this.valid = true;
-  }
-
-  validCheck() {
-    if (!this.valid) {
-      throw "Using freed memory"
-    }
-  }
-
-  futharkType() {
-    return this.type_name;
-  }
-
-  free() {
-    this.validCheck();
-    this.ffree(this.ctx.ctx, this.ptr);
-    this.valid = false;
-  }
-
-  shape() {
-    this.validCheck();
-    var s = this.fshape(this.ctx.ctx, this.ptr) >> 3;
-    return this.ctx.wasm.HEAP64.subarray(s, s + this.dim);
-  }
-
-  // get_shape() {
-  //   return BigInt64Array.from(this.shape());
-  // }
-
-  toTypedArray(dims = this.shape()) {
-    this.validCheck();
-    console.assert(dims.length === this.dim, "dim=%s,dims=%s", this.dim, dims.toString());
-    var length = Number(dims.reduce((a, b) => a * b));
-    var v = this.fvalues(this.ctx.ctx, this.ptr) / this.heap.BYTES_PER_ELEMENT;
-    return this.heap.subarray(v, v + length);
-  }
-
-  toArray() {
-    this.validCheck();
-    var dims = this.shape();
-    var ta = this.toTypedArray(dims);
-    return (function nest(offs, ds) {
-      var d0 = Number(ds[0]);
-      if (ds.length === 1) {
-        return Array.from(ta.subarray(offs, offs + d0));
-      } else {
-        var d1 = Number(ds[1]);
-        return Array.from(Array(d0), (x,i) => nest(offs + i * d1, ds.slice(1)));
-      }
-    })(0, dims);
-  }
-
-  async values() {
-    return this.toTypedArray();
-  }
-}
-
-class FutharkOpaque {
-  constructor(ctx, ptr, ffree) {
-    this.ctx = ctx;
-    this.ptr = ptr;
-    this.ffree = ffree;
-    this.valid = true;
-  }
-
-  validCheck() {
-    if (!this.valid) {
-      throw "Using freed memory"
-    }
-  }
-
-  free() {
-    this.validCheck();
-    this.ffree(this.ctx.ctx, this.ptr);
-    this.valid = false;
-  }
-}
-
-// End of wrapperclasses.js
diff --git a/rts/javascript/wrappers.js b/rts/javascript/wrappers.js
new file mode 100644
--- /dev/null
+++ b/rts/javascript/wrappers.js
@@ -0,0 +1,567 @@
+// Start of wrappers.js
+//
+// The JavaScript API (docs/js-api.rst), driven by 'futhark_manifest'.
+//
+// Asynchronous operations run one at a time, as Asyncify (used by WebGPU)
+// allows no other calls into the module while one is suspended.
+
+// f16 values and their bit patterns, rounding to nearest even.
+function futhark_f16_bits(x) {
+  if (Number.isNaN(x)) {
+    return 0x7e00;
+  }
+  var sign = (x < 0 || Object.is(x, -0)) ? 0x8000 : 0;
+  x = Math.abs(x);
+  if (x < 2 ** -14) {
+    // Subnormal or zero.
+    return sign | futhark_round_even(x * 2 ** 24);
+  }
+  var e = Math.floor(Math.log2(x));
+  // Math.log2 may be inexact.
+  if (2 ** e > x) {
+    e--;
+  } else if (2 ** (e + 1) <= x) {
+    e++;
+  }
+  var m = futhark_round_even((x / 2 ** e - 1) * 1024);
+  if (m == 1024) {
+    m = 0;
+    e++;
+  }
+  if (e > 15) {
+    return sign | 0x7c00;
+  }
+  return sign | ((e + 15) << 10) | m;
+}
+
+function futhark_round_even(x) {
+  var r = Math.floor(x);
+  var d = x - r;
+  return (d > 0.5 || (d == 0.5 && r % 2 == 1)) ? r + 1 : r;
+}
+
+function futhark_f16_value(b) {
+  var sign = (b & 0x8000) ? -1 : 1;
+  var e = (b >> 10) & 0x1f;
+  var m = b & 0x3ff;
+  if (e == 0) {
+    return sign * m * 2 ** -24;
+  } else if (e == 31) {
+    return m == 0 ? sign * Infinity : NaN;
+  } else {
+    return sign * (1 + m / 1024) * 2 ** (e - 15);
+  }
+}
+
+function futhark_check_int(x, bits, signed, what) {
+  var ok;
+  if (typeof x == "bigint") {
+    ok = (signed ? BigInt.asIntN(bits, x) : BigInt.asUintN(bits, x)) == x;
+  } else if (typeof x == "number") {
+    var lo = signed ? -(2 ** (bits - 1)) : 0;
+    var hi = signed ? 2 ** (bits - 1) : 2 ** bits;
+    ok = Number.isInteger(x) && x >= lo && x < hi;
+  } else {
+    throw new TypeError(what + " must be a number or a BigInt");
+  }
+  if (!ok) {
+    throw new RangeError(what + " is not a " + (signed ? "i" : "u") + bits + " value: " + x);
+  }
+}
+
+// 'array' is used for values(), and 'c_array' for the C representation.
+function futhark_int_prim(bits, signed, array) {
+  var big = bits == 64;
+  return {
+    size: bits / 8,
+    array: array,
+    c_array: array,
+    to_c: (x, what) => {
+      futhark_check_int(x, bits, signed, what);
+      return big ? BigInt(x) : Number(x);
+    },
+    get: (dv, p) => big
+      ? (signed ? dv.getBigInt64(p, true) : dv.getBigUint64(p, true))
+      : dv["get" + (signed ? "Int" : "Uint") + bits](p, true),
+    to_values: (bytes) => new array(bytes.buffer),
+  };
+}
+
+function futhark_float_prim(bits, array) {
+  return {
+    size: bits / 8,
+    array: array,
+    c_array: array,
+    to_c: (x, what) => {
+      if (typeof x != "number") {
+        throw new TypeError(what + " must be a number");
+      }
+      return x;
+    },
+    get: (dv, p) => dv["getFloat" + bits](p, true),
+    to_values: (bytes) => new array(bytes.buffer),
+  };
+}
+
+// Float32Array represents all f16 values exactly.
+var futhark_f16_array = globalThis.Float16Array ?? Float32Array;
+
+var futhark_prims = {
+  i8: futhark_int_prim(8, true, Int8Array),
+  i16: futhark_int_prim(16, true, Int16Array),
+  i32: futhark_int_prim(32, true, Int32Array),
+  i64: futhark_int_prim(64, true, BigInt64Array),
+  u8: futhark_int_prim(8, false, Uint8Array),
+  u16: futhark_int_prim(16, false, Uint16Array),
+  u32: futhark_int_prim(32, false, Uint32Array),
+  u64: futhark_int_prim(64, false, BigUint64Array),
+  f16: {
+    size: 2,
+    array: futhark_f16_array,
+    c_array: Uint16Array,
+    to_c: (x, what) => futhark_f16_bits(futhark_prims.f32.to_c(x, what)),
+    get: (dv, p) => futhark_f16_value(dv.getUint16(p, true)),
+    to_values: (bytes) =>
+      futhark_f16_array.from(new Uint16Array(bytes.buffer), futhark_f16_value),
+  },
+  f32: futhark_float_prim(32, Float32Array),
+  f64: futhark_float_prim(64, Float64Array),
+  bool: {
+    size: 1,
+    array: Uint8Array,
+    c_array: Uint8Array,
+    to_c: (x, what) => {
+      if (typeof x != "boolean") {
+        throw new TypeError(what + " must be a boolean");
+      }
+      return x ? 1 : 0;
+    },
+    get: (dv, p) => dv.getUint8(p) != 0,
+    to_values: (bytes) => bytes,
+  },
+};
+
+// Booleans may also be 0 or 1, as returned by values().
+function futhark_from_data(type, prim, data) {
+  if (data instanceof prim.c_array && prim.c_array === prim.array && type != "bool") {
+    return data;
+  }
+  var out = new prim.c_array(data.length);
+  for (var i = 0; i < data.length; i++) {
+    var x = data[i];
+    if (type == "bool" && (x === 0 || x === 1)) {
+      x = x === 1;
+    }
+    out[i] = prim.to_c(x, "Element " + i + " of the array data");
+  }
+  return out;
+}
+
+function futhark_check_value(obj, cls, ctx, type, what) {
+  if (!(obj instanceof cls)) {
+    throw new TypeError(what + " must be a " + cls.name);
+  }
+  if (obj.ctx !== ctx) {
+    throw new Error(what + " belongs to a different FutharkContext");
+  }
+  if (obj.type !== type) {
+    throw new TypeError(what + " has type " + obj.type + ", but must have type " + type);
+  }
+  obj._check_usable();
+}
+
+class FutharkValue {
+  constructor(ctx, type, ptr) {
+    this.ctx = ctx;
+    this.type = type;
+    this.ptr = ptr;
+    this.consumed = false;
+  }
+
+  _check_usable() {
+    this._check_not_freed();
+    if (this.consumed) {
+      throw new Error("Using a value that has been consumed by an entry point");
+    }
+  }
+
+  _check_not_freed() {
+    if (this.ptr === undefined) {
+      throw new Error("Using a value that has been freed");
+    }
+    this.ctx._check_usable();
+  }
+
+  free() {
+    this._check_not_freed();
+    this.ctx._check_idle();
+    this.ctx.wasm["_" + this.ctx.manifest.types[this.type].ops.free](this.ctx.ctx, this.ptr);
+    this.ptr = undefined;
+  }
+}
+
+class FutharkArray extends FutharkValue {
+  shape() {
+    this._check_usable();
+    this.ctx._check_idle();
+    return this._shape();
+  }
+
+  _shape() {
+    var info = this.ctx.manifest.types[this.type];
+    var p = this.ctx.wasm["_" + info.ops.shape](this.ctx.ctx, this.ptr);
+    return new BigInt64Array(this.ctx.wasm.HEAPU8.slice(p, p + 8 * info.rank).buffer);
+  }
+
+  values() {
+    var ctx = this.ctx;
+    return ctx._exclusive(async () => {
+      this._check_usable();
+      var info = ctx.manifest.types[this.type];
+      var prim = futhark_prims[info.elemtype];
+      var bytes = Number(this._shape().reduce((a, b) => a * b, 1n)) * prim.size;
+      var p = ctx._malloc(bytes);
+      try {
+        ctx._check(await ctx._call(info.ops.values, ctx.ctx, this.ptr, p));
+        await ctx._sync();
+        return prim.to_values(ctx.wasm.HEAPU8.slice(p, p + bytes));
+      } finally {
+        ctx.wasm._free(p);
+      }
+    });
+  }
+}
+
+class FutharkOpaque extends FutharkValue {
+}
+
+class FutharkContext {
+  constructor() {
+    this.manifest = futhark_manifest;
+    this.wasm = undefined;
+    this.cfg = undefined;
+    this.ctx = undefined;
+    this.initialised = false;
+    this.freed = false;
+    this.queue = Promise.resolve();
+    this.running = false;
+    this.entry = Object.create(null);
+    this.types = Object.create(null);
+  }
+
+  async init(options = {}) {
+    if (this.initialised) {
+      throw new Error("FutharkContext.init() may only be called once");
+    }
+    this.initialised = true;
+    const flags = ["debugging", "logging", "profiling"];
+    for (const option of Object.keys(options)) {
+      if (option != "num_threads" && option != "tuning_params" && !flags.includes(option)) {
+        throw new Error("Unknown FutharkContext.init() option: " + option);
+      }
+    }
+    for (const flag of flags) {
+      if (options[flag] !== undefined && typeof options[flag] != "boolean") {
+        throw new TypeError("The " + flag + " option must be a boolean");
+      }
+    }
+    if (options.num_threads !== undefined) {
+      futhark_check_int(options.num_threads, 32, true, "The num_threads option");
+    }
+    const tuning_params = Object.entries(options.tuning_params ?? {});
+    for (const [name, value] of tuning_params) {
+      futhark_check_int(value, 32, false, "Tuning parameter " + name);
+    }
+
+    if (this.manifest.backend == "wasm_multicore" && globalThis.SharedArrayBuffer === undefined) {
+      throw new Error("The wasm-multicore backend requires SharedArrayBuffer, which browsers " +
+                      "only provide to pages that are cross-origin isolated");
+    }
+    try {
+      this.wasm = await loadWASM();
+    } catch (e) {
+      throw e instanceof Error ? e : new Error("Failed to load the WebAssembly module: " + e);
+    }
+    if (options.num_threads !== undefined &&
+        this.wasm._futhark_context_config_set_num_threads === undefined) {
+      throw new Error("The num_threads option is not supported by this backend");
+    }
+
+    this.cfg = this.wasm._futhark_context_config_new();
+    for (const flag of flags) {
+      if (options[flag]) {
+        this.wasm["_futhark_context_config_set_" + flag](this.cfg, 1);
+      }
+    }
+    if (options.num_threads !== undefined) {
+      this.wasm._futhark_context_config_set_num_threads(this.cfg, options.num_threads);
+    }
+    for (const [name, value] of tuning_params) {
+      if (this.wasm.ccall("futhark_context_config_set_tuning_param", "number",
+                          ["number", "string", "number"], [this.cfg, name, value]) != 0) {
+        this.free();
+        throw new Error("Unknown tuning parameter: " + name);
+      }
+    }
+    var ctx = await this._call("futhark_context_new", this.cfg);
+    var err = this._get_error(ctx);
+    if (err !== null) {
+      this.wasm._futhark_context_free(ctx);
+      this.free();
+      throw new Error(err);
+    }
+    this.ctx = ctx;
+
+    for (const [name, info] of Object.entries(this.manifest.types)) {
+      if (info.kind == "array") {
+        const constructor = {
+          from_data: (data, ...shape) => this._new_array(name, data, shape),
+        };
+        this.types[name] = constructor;
+        this[info.elemtype + "_" + info.rank + "d"] = constructor;
+      } else if (info.record !== undefined) {
+        for (const field of info.record.fields) {
+          this[field.project.replace(/^futhark_/, "")] =
+            (obj) => this._exclusive(() => this._project(name, field, obj));
+        }
+      }
+    }
+    for (const [name, info] of Object.entries(this.manifest.entry_points)) {
+      this.entry[name] = (...args) => this._exclusive(() => this._call_entry(name, info, args));
+    }
+  }
+
+  free() {
+    this._check_idle();
+    if (this.ctx !== undefined) {
+      this.wasm._futhark_context_free(this.ctx);
+      this.ctx = undefined;
+    }
+    if (this.cfg !== undefined) {
+      this.wasm._futhark_context_config_free(this.cfg);
+      this.cfg = undefined;
+    }
+    this.freed = true;
+  }
+
+  context_sync() {
+    return this._exclusive(async () => {
+      this._check_usable();
+      await this._sync();
+    });
+  }
+
+  clear_caches() {
+    return this._exclusive(async () => {
+      this._check_usable();
+      this._check(await this._call("futhark_context_clear_caches", this.ctx));
+    });
+  }
+
+  report() {
+    return this._exclusive(async () => {
+      this._check_usable();
+      return this._c_string(await this._call("futhark_context_report", this.ctx));
+    });
+  }
+
+  pause_profiling() {
+    return this._exclusive(async () => {
+      this._check_usable();
+      await this._call("futhark_context_pause_profiling", this.ctx);
+    });
+  }
+
+  unpause_profiling() {
+    return this._exclusive(async () => {
+      this._check_usable();
+      await this._call("futhark_context_unpause_profiling", this.ctx);
+    });
+  }
+
+  // Everything below is internal.
+
+  _check_usable() {
+    if (this.freed) {
+      throw new Error("The FutharkContext has been freed");
+    }
+    if (this.ctx === undefined) {
+      throw new Error("The FutharkContext has not been initialised");
+    }
+  }
+
+  // Run 'f' after all previously started asynchronous operations.
+  _exclusive(f) {
+    var run = this.queue.then(async () => {
+      this.running = true;
+      try {
+        return await f();
+      } finally {
+        this.running = false;
+      }
+    });
+    this.queue = run.catch(() => {});
+    return run;
+  }
+
+  _check_idle() {
+    if (this.running) {
+      throw new Error("Cannot do this while an asynchronous operation on the FutharkContext is running");
+    }
+  }
+
+  async _call(cfun, ...args) {
+    return await this.wasm.ccall(cfun, "number", args.map(() => "number"), args, {async: true});
+  }
+
+  _malloc(n) {
+    // malloc(0) may return NULL.
+    var p = this.wasm._malloc(Math.max(n, 1));
+    if (p == 0) {
+      throw new RangeError("Failed to allocate " + n + " bytes");
+    }
+    return p;
+  }
+
+  _c_string(p) {
+    var heap = this.wasm.HEAPU8;
+    // Browsers' TextDecoder rejects shared memory.
+    var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));
+    this.wasm._free(p);
+    return str;
+  }
+
+  _get_error(ctx = this.ctx) {
+    var p = this.wasm._futhark_context_get_error(ctx);
+    return p == 0 ? null : this._c_string(p);
+  }
+
+  _error() {
+    return new Error(this._get_error() ?? "Unknown error");
+  }
+
+  _check(ret) {
+    if (ret != 0) {
+      throw this._error();
+    }
+  }
+
+  async _sync() {
+    this._check(await this._call("futhark_context_sync", this.ctx));
+  }
+
+  _to_c(type, x, what) {
+    if (type in futhark_prims) {
+      return futhark_prims[type].to_c(x, what);
+    }
+    var cls = this.manifest.types[type].kind == "array" ? FutharkArray : FutharkOpaque;
+    futhark_check_value(x, cls, this, type, what);
+    return x.ptr;
+  }
+
+  _from_c(type, p) {
+    var dv = new DataView(this.wasm.HEAPU8.buffer);
+    if (type in futhark_prims) {
+      return futhark_prims[type].get(dv, p);
+    }
+    var ptr = dv.getUint32(p, true);
+    if (this.manifest.types[type].kind == "array") {
+      return new FutharkArray(this, type, ptr);
+    } else {
+      return new FutharkOpaque(this, type, ptr);
+    }
+  }
+
+  _free_c(type, p) {
+    if (!(type in futhark_prims)) {
+      var ptr = new DataView(this.wasm.HEAPU8.buffer).getUint32(p, true);
+      this.wasm["_" + this.manifest.types[type].ops.free](this.ctx, ptr);
+    }
+  }
+
+  // For C functions with an output parameter after the context.
+  async _call_with_output(cfun, type, args, after_call) {
+    var out = this._malloc(8);
+    try {
+      var ret = await this._call(cfun, this.ctx, out, ...args);
+      after_call();
+      this._check(ret);
+      try {
+        await this._sync();
+      } catch (e) {
+        this._free_c(type, out);
+        throw e;
+      }
+      return this._from_c(type, out);
+    } finally {
+      this.wasm._free(out);
+    }
+  }
+
+  _new_array(type, data, shape) {
+    this._check_usable();
+    this._check_idle();
+    var info = this.manifest.types[type];
+    var prim = futhark_prims[info.elemtype];
+    if (shape.length != info.rank) {
+      throw new TypeError("Expected " + info.rank + " dimensions, but got " + shape.length);
+    }
+    shape = shape.map((d, i) => {
+      futhark_check_int(d, 64, true, "dimension " + i);
+      if (d < 0) {
+        throw new RangeError("dimension " + i + " is negative: " + d);
+      }
+      return BigInt(d);
+    });
+    if (!(data instanceof Array || (ArrayBuffer.isView(data) && !(data instanceof DataView)))) {
+      throw new TypeError("Array data must be an Array or a typed array");
+    }
+    var n = shape.reduce((a, b) => a * b, 1n);
+    if (BigInt(data.length) != n) {
+      throw new RangeError("Array data has " + data.length + " elements, but the shape requires " + n);
+    }
+    var elems = futhark_from_data(info.elemtype, prim, data);
+    var bytes = new Uint8Array(elems.buffer, elems.byteOffset, elems.byteLength);
+    var p = this._malloc(bytes.length);
+    try {
+      this.wasm.HEAPU8.set(bytes, p);
+      var ptr = this.wasm["_" + info.ops.new](this.ctx, p, ...shape);
+      if (ptr == 0) {
+        throw this._error();
+      }
+      return new FutharkArray(this, type, ptr);
+    } finally {
+      this.wasm._free(p);
+    }
+  }
+
+  async _project(type, field, obj) {
+    this._check_usable();
+    futhark_check_value(obj, FutharkOpaque, this, type, "The record");
+    return await this._call_with_output(field.project, field.type, [obj.ptr], () => {});
+  }
+
+  async _call_entry(name, info, args) {
+    this._check_usable();
+    if (args.length != info.inputs.length) {
+      throw new TypeError("Entry point " + name + " expects " + info.inputs.length +
+                          " arguments, but got " + args.length);
+    }
+    var c_args = info.inputs.map((input, i) =>
+      this._to_c(input.type, args[i], "Argument " + i + " of entry point " + name));
+    info.inputs.forEach((input, i) => {
+      if (input.consumed && args.some((x, j) => j != i && x === args[i])) {
+        throw new Error("Argument " + i + " of entry point " + name +
+                        " is consumed, so it must not also be passed as another argument");
+      }
+    });
+    var mark_consumed = () => info.inputs.forEach((input, i) => {
+      if (input.consumed) {
+        args[i].consumed = true;
+      }
+    });
+    return await this._call_with_output(info.cfun, info.output.type, c_args, mark_consumed);
+  }
+}
+
+// End of wrappers.js
diff --git a/rts/opencl/copy.cl b/rts/opencl/copy.cl
--- a/rts/opencl/copy.cl
+++ b/rts/opencl/copy.cl
@@ -2,9 +2,7 @@
 
 #define GEN_COPY_KERNEL(NAME, ELEM_TYPE) \
 FUTHARK_KERNEL void lmad_copy_##NAME(SHARED_MEM_PARAM                   \
-                               __global ELEM_TYPE *dst_mem,             \
                                int64_t dst_offset,                      \
-                               __global ELEM_TYPE *src_mem,             \
                                int64_t src_offset,                      \
                                int64_t n,                               \
                                int r,                                   \
@@ -15,7 +13,9 @@
                                int64_t shape4, int64_t dst_stride4, int64_t src_stride4, \
                                int64_t shape5, int64_t dst_stride5, int64_t src_stride5, \
                                int64_t shape6, int64_t dst_stride6, int64_t src_stride6, \
-                               int64_t shape7, int64_t dst_stride7, int64_t src_stride7) { \
+                               int64_t shape7, int64_t dst_stride7, int64_t src_stride7, \
+                               __global ELEM_TYPE *dst_mem,             \
+                               __global ELEM_TYPE *src_mem) { \
   int64_t gtid = get_global_id(0);                                      \
   int64_t remainder = gtid;                                             \
                                                                         \
diff --git a/rts/opencl/transpose.cl b/rts/opencl/transpose.cl
--- a/rts/opencl/transpose.cl
+++ b/rts/opencl/transpose.cl
@@ -3,9 +3,7 @@
 #define GEN_TRANSPOSE_KERNELS(NAME, ELEM_TYPE)                          \
 FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\
 void map_transpose_##NAME(SHARED_MEM_PARAM                              \
-                          __global ELEM_TYPE *dst_mem,                  \
                           int64_t dst_offset,                           \
-                          __global ELEM_TYPE *src_mem,                  \
                           int64_t src_offset,                           \
                           int32_t num_arrays,                           \
                           int32_t x_elems,                              \
@@ -17,7 +15,9 @@
                           int32_t mulx,                                 \
                           int32_t muly,                                 \
                           int32_t repeat_1,                             \
-                          int32_t repeat_2) {                           \
+                          int32_t repeat_2,                             \
+                          __global ELEM_TYPE *dst_mem,                  \
+                          __global ELEM_TYPE *src_mem) {                \
   (void)mulx; (void)muly;                                               \
   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \
   int tblock_id_0 = get_tblock_id(0);                                   \
@@ -29,6 +29,8 @@
     int global_id_2 = get_global_id(2);                                 \
     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \
       if (tblock_id_2 >= num_arrays) { break; }                         \
+      /* The previous iteration may still be reading shared memory. */  \
+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \
       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \
       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \
       int32_t x_index = global_id_0;                                    \
@@ -66,21 +68,21 @@
                                                                         \
 FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1)                     \
 void map_transpose_##NAME##_low_height(SHARED_MEM_PARAM                 \
-                                                __global ELEM_TYPE *dst_mem, \
-                                                int64_t dst_offset,     \
-                                                __global ELEM_TYPE *src_mem, \
-                                                int64_t src_offset,     \
-                                                int32_t num_arrays,     \
-                                                int32_t x_elems,        \
-                                                int32_t y_elems,        \
-                                                int32_t dst_array_stride, \
-                                                int32_t dst_row_stride, \
-                                                int32_t src_array_stride, \
-                                                int32_t src_row_stride, \
-                                                int32_t mulx,           \
-                                                int32_t muly,           \
-                                                int32_t repeat_1,       \
-                                                int32_t repeat_2) {     \
+                                       int64_t dst_offset,              \
+                                       int64_t src_offset,              \
+                                       int32_t num_arrays,              \
+                                       int32_t x_elems,                 \
+                                       int32_t y_elems,                 \
+                                       int32_t dst_array_stride,        \
+                                       int32_t dst_row_stride,          \
+                                       int32_t src_array_stride,        \
+                                       int32_t src_row_stride,          \
+                                       int32_t mulx,                    \
+                                       int32_t muly,                    \
+                                       int32_t repeat_1,                \
+                                       int32_t repeat_2,                \
+                                       __global ELEM_TYPE *dst_mem,     \
+                                       __global ELEM_TYPE *src_mem) {   \
   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \
   int tblock_id_0 = get_tblock_id(0);                                   \
   int global_id_0 = get_global_id(0);                                   \
@@ -91,6 +93,8 @@
     int global_id_2 = get_global_id(2);                                 \
     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \
       if (tblock_id_2 >= num_arrays) { break; }                         \
+      /* The previous iteration may still be reading shared memory. */  \
+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \
       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \
       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \
       int32_t x_index =                                                 \
@@ -124,9 +128,7 @@
                                                                         \
 FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1)                     \
 void map_transpose_##NAME##_low_width(SHARED_MEM_PARAM                  \
-                                      __global ELEM_TYPE *dst_mem,      \
                                       int64_t dst_offset,               \
-                                      __global ELEM_TYPE *src_mem,      \
                                       int64_t src_offset,               \
                                       int32_t num_arrays,               \
                                       int32_t x_elems,                  \
@@ -138,7 +140,9 @@
                                       int32_t mulx,                     \
                                       int32_t muly,                     \
                                       int32_t repeat_1,                 \
-                                      int32_t repeat_2) {               \
+                                      int32_t repeat_2,                 \
+                                      __global ELEM_TYPE *dst_mem,      \
+                                      __global ELEM_TYPE *src_mem) {    \
   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \
   int tblock_id_0 = get_tblock_id(0);                                   \
   int global_id_0 = get_global_id(0);                                   \
@@ -149,6 +153,8 @@
     int global_id_2 = get_global_id(2);                                 \
     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \
       if (tblock_id_2 >= num_arrays) { break; }                         \
+      /* The previous iteration may still be reading shared memory. */  \
+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \
       int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \
       int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \
       int32_t x_index = tblock_id_0 * TR_BLOCK_DIM + get_local_id(0)/muly; \
@@ -178,10 +184,8 @@
 }                                                                       \
                                                                         \
 FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*TR_BLOCK_DIM, 1, 1)                   \
-void map_transpose_##NAME##_small(SHARED_MEM_PARAM                       \
-                                  __global ELEM_TYPE *dst_mem,          \
+void map_transpose_##NAME##_small(SHARED_MEM_PARAM                      \
                                   int64_t dst_offset,                   \
-                                  __global ELEM_TYPE *src_mem,          \
                                   int64_t src_offset,                   \
                                   int32_t num_arrays,                   \
                                   int32_t x_elems,                      \
@@ -193,7 +197,9 @@
                                   int32_t mulx,                         \
                                   int32_t muly,                         \
                                   int32_t repeat_1,                     \
-                                  int32_t repeat_2) {                   \
+                                  int32_t repeat_2,                     \
+                                  __global ELEM_TYPE *dst_mem,          \
+                                  __global ELEM_TYPE *src_mem) {        \
   (void)mulx; (void)muly;                                               \
   __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \
   int tblock_id_0 = get_tblock_id(0);                                   \
@@ -224,9 +230,7 @@
                                                                         \
 FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\
 void map_transpose_##NAME##_large(SHARED_MEM_PARAM                      \
-                                  __global ELEM_TYPE *dst_mem,          \
                                   int64_t dst_offset,                   \
-                                  __global ELEM_TYPE *src_mem,          \
                                   int64_t src_offset,                   \
                                   int64_t num_arrays,                   \
                                   int64_t x_elems,                      \
@@ -238,9 +242,11 @@
                                   int64_t mulx,                         \
                                   int64_t muly,                         \
                                   int32_t repeat_1,                     \
-                                  int32_t repeat_2) {                   \
+                                  int32_t repeat_2,                     \
+                                  __global ELEM_TYPE *dst_mem,          \
+                                  __global ELEM_TYPE *src_mem) {        \
   (void)mulx; (void)muly;                                               \
-  __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;             \
+  __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem;            \
   int tblock_id_0 = get_tblock_id(0);                                   \
   int global_id_0 = get_global_id(0);                                   \
   int tblock_id_1 = get_tblock_id(1);                                   \
@@ -250,6 +256,8 @@
     int global_id_2 = get_global_id(2);                                 \
     for (int i2 = 0; i2 <= repeat_2; i2++) {                            \
       if (tblock_id_2 >= num_arrays) { break; }                         \
+      /* The previous iteration may still be reading shared memory. */  \
+      if (i1 > 0 || i2 > 0) { barrier_local(); }                        \
       int64_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \
       int64_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \
       int64_t x_index = global_id_0;                                    \
diff --git a/rts/python/opencl.py b/rts/python/opencl.py
--- a/rts/python/opencl.py
+++ b/rts/python/opencl.py
@@ -183,6 +183,14 @@
     )
     self.failure_is_an_option = np.int32(0)
 
+    self.bookkeeping_counters = self.pool.allocate(
+        np.int32().itemsize * self.bookkeeping_counters_size
+    )
+    self.bookkeeping_locks = self.pool.allocate(
+        np.int32().itemsize * self.bookkeeping_locks_size
+    )
+    zero_bookkeeping(self)
+
     if "default_group_size" in user_sizes:
         default_group_size = user_sizes["default_group_size"]
         del user_sizes["default_group_size"]
@@ -389,6 +397,23 @@
     self.pool.free_held()
 
 
+def zero_bookkeeping(self):
+    cl.enqueue_fill_buffer(
+        self.queue,
+        self.bookkeeping_counters,
+        np.int32(0),
+        0,
+        np.int32().itemsize * self.bookkeeping_counters_size,
+    )
+    cl.enqueue_fill_buffer(
+        self.queue,
+        self.bookkeeping_locks,
+        np.int32(0),
+        0,
+        np.int32().itemsize * self.bookkeeping_locks_size,
+    )
+
+
 def sync(self):
     failure = np.empty(1, dtype=np.int32)
     cl.enqueue_copy(self.queue, failure, self.global_failure, is_blocking=True)
@@ -403,6 +428,9 @@
             np.int32().itemsize,
         )
 
+        # The failing kernel may have left the bookkeeping buffers nonzero.
+        zero_bookkeeping(self)
+
         # Read failure args.
         failure_args = np.empty(
             self.global_failure_args_max + 1, dtype=np.int64
@@ -441,9 +469,7 @@
 
     kernel.set_args(
         cl.LocalMemory(TR_TILE_DIM * (TR_TILE_DIM + 1) * elem_size),
-        dst,
         dst_offset,
-        src,
         src_offset,
         k32,
         m32,
@@ -453,6 +479,8 @@
         muly32,
         np.int32(0),
         np.int32(0),
+        dst,
+        src,
     )
     cl.enqueue_nd_range_kernel(self.queue, kernel, dims, group_dims)
 
@@ -490,13 +518,13 @@
     kernel = self.copy_kernels[elem_size]
     kernel.set_args(
         cl.LocalMemory(1),
-        dst,
         dst_offset,
-        src,
         src_offset,
         n,
         np.int32(r),
         *layout_args,
+        dst,
+        src,
     )
     w = 256
     dims = ((n + w - 1) // w * w,)
diff --git a/rts/python/scalar.py b/rts/python/scalar.py
--- a/rts/python/scalar.py
+++ b/rts/python/scalar.py
@@ -409,6 +409,13 @@
         return ssignum(x)
 
 
+def sitofp_T_f16(x):
+    return np.float16(x)
+
+
+sitofp_i8_f16 = sitofp_i16_f16 = sitofp_i32_f16 = sitofp_i64_f16 = sitofp_T_f16
+
+
 def sitofp_T_f32(x):
     return np.float32(x)
 
@@ -423,6 +430,13 @@
 sitofp_i8_f64 = sitofp_i16_f64 = sitofp_i32_f64 = sitofp_i64_f64 = sitofp_T_f64
 
 
+def uitofp_T_f16(x):
+    return np.float36(unsigned(x))
+
+
+uitofp_i8_f16 = uitofp_i16_f16 = uitofp_i32_f16 = uitofp_i64_f16 = uitofp_T_f16
+
+
 def uitofp_T_f32(x):
     return np.float32(unsigned(x))
 
@@ -481,7 +495,7 @@
     if np.isnan(x) or x < 0 or x >= (1 << 8):
         return np.int8(0)
     else:
-        return np.int8(np.trunc(x))
+        return np.uint8(np.trunc(x)).astype(np.int8)
 
 
 fptoui_f16_i8 = fptoui_f32_i8 = fptoui_f64_i8 = fptoui_T_i8
@@ -491,7 +505,7 @@
     if np.isnan(x) or x < 0 or x >= (1 << 16):
         return np.int16(0)
     else:
-        return np.int16(np.trunc(x))
+        return np.uint16(np.trunc(x)).astype(np.int16)
 
 
 fptoui_f16_i16 = fptoui_f32_i16 = fptoui_f64_i16 = fptoui_T_i16
@@ -501,7 +515,7 @@
     if np.isnan(x) or x < 0 or x >= (1 << 32):
         return np.int32(0)
     else:
-        return np.int32(np.trunc(x))
+        return np.uint32(np.trunc(x)).astype(np.int32)
 
 
 fptoui_f16_i32 = fptoui_f32_i32 = fptoui_f64_i32 = fptoui_T_i32
@@ -511,7 +525,7 @@
     if np.isnan(x) or x < 0 or x >= (1 << 64):
         return np.int64(0)
     else:
-        return np.int64(np.trunc(x))
+        return np.uint64(np.trunc(x)).astype(np.int64)
 
 
 fptoui_f16_i64 = fptoui_f32_i64 = fptoui_f64_i64 = fptoui_T_i64
diff --git a/src/Futhark/Actions.hs b/src/Futhark/Actions.hs
--- a/src/Futhark/Actions.hs
+++ b/src/Futhark/Actions.hs
@@ -33,12 +33,14 @@
 import Control.Monad
 import Control.Monad.IO.Class
 import Control.Monad.State (get)
+import Data.Aeson qualified as JSON
 import Data.Bifunctor
 import Data.ByteString.Lazy.Char8 qualified as BS
 import Data.List (intercalate)
 import Data.Map qualified as M
 import Data.Maybe (fromMaybe)
 import Data.Text qualified as T
+import Data.Text.Encoding qualified as T
 import Data.Text.IO qualified as T
 import Futhark.Analysis.AccessPattern
 import Futhark.Analysis.Alias
@@ -49,6 +51,8 @@
 import Futhark.Analysis.Metrics
 import Futhark.CodeGen.Backends.CCUDA qualified as CCUDA
 import Futhark.CodeGen.Backends.COpenCL qualified as COpenCL
+import Futhark.CodeGen.Backends.GenericC qualified as GC
+import Futhark.CodeGen.Backends.GenericWASM qualified as GenericWASM
 import Futhark.CodeGen.Backends.HIP qualified as HIP
 import Futhark.CodeGen.Backends.MulticoreC qualified as MulticoreC
 import Futhark.CodeGen.Backends.MulticoreISPC qualified as MulticoreISPC
@@ -532,30 +536,46 @@
 cmdEMCFLAGS :: [String] -> [String]
 cmdEMCFLAGS def = maybe def words $ lookup "EMCFLAGS" unixEnvironment
 
-runEMCC :: String -> String -> FilePath -> [String] -> [String] -> [String] -> Bool -> FutharkM ()
-runEMCC cpath outpath classpath cflags_def ldflags expfuns lib = do
+-- | The typed views of the WASM heap, which the generated JavaScript
+-- uses to move array data in and out of the module.
+heapExports :: [String]
+heapExports =
+  [ "'HEAP8'",
+    "'HEAP16'",
+    "'HEAP32'",
+    "'HEAP64'",
+    "'HEAPU8'",
+    "'HEAPU16'",
+    "'HEAPU32'",
+    "'HEAPU64'",
+    "'HEAPF32'",
+    "'HEAPF64'"
+  ]
+
+-- | Options passed to @emcc@ by both WASM backends.
+wasmFlags :: FilePath -> [String] -> [String]
+wasmFlags classpath expfuns =
+  [ "--extern-post-js",
+    classpath,
+    -- Called by the generated JavaScript.
+    "-sEXPORT_NAME=loadWASM",
+    "-sWASM_BIGINT",
+    -- Up to 2GiB; more would need unsigned pointers in JavaScript.
+    "-sALLOW_MEMORY_GROWTH",
+    "-sEXPORTED_RUNTIME_METHODS=[" ++ intercalate "," ("'ccall'" : heapExports) ++ "]",
+    "-sEXPORTED_FUNCTIONS=[" ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns) ++ "]"
+  ]
+
+runEMCC :: String -> String -> [String] -> [String] -> [String] -> FutharkM ()
+runEMCC cpath outpath cflags_def ldflags extra_flags = do
   ret <-
     liftIO $
       runProgramWithExitCode
         "emcc"
         ( [cpath, "-o", outpath]
-            ++ ["-lnodefs.js"]
-            ++ ["-s", "--extern-post-js", classpath]
-            ++ ( if lib
-                   then ["-s", "EXPORT_NAME=loadWASM"]
-                   else []
-               )
-            ++ ["-s", "WASM_BIGINT"]
-            ++ [ "-s",
-                 "EXPORTED_RUNTIME_METHODS=['HEAP8','HEAP16','HEAP32','HEAP64','HEAPU8','HEAPU16','HEAPU32','HEAPF32','HEAPF64']"
-               ]
+            ++ extra_flags
             ++ cmdCFLAGS cflags_def
             ++ cmdEMCFLAGS [""]
-            ++ [ "-s",
-                 "EXPORTED_FUNCTIONS=["
-                   ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns)
-                   ++ "]"
-               ]
             -- The default LDFLAGS are always added.
             ++ ldflags
         )
@@ -572,73 +592,63 @@
     Right (ExitSuccess, _, _) ->
       pure ()
 
+-- | Compile to the ES module @outpath.mjs@ with emcc.  Unless compiling a
+-- library, @outpath@ is an executable that runs it as a server.
+compileWASM ::
+  CompilerMode ->
+  FilePath ->
+  [String] ->
+  (GC.CParts, T.Text, [String]) ->
+  FutharkM ()
+compileWASM mode outpath ldflags (cprog, jsprog, exps) = do
+  let (_, impl, _) = GC.asLibrary cprog
+  liftIO $ T.writeFile cpath impl
+  liftIO $ T.writeFile classpath $ jsprog <> extra
+  runEMCC cpath mjspath ["-O3", "-msimd128"] ldflags (wasmFlags classpath exps)
+  when (mode /= ToLibrary) . liftIO $ do
+    T.writeFile outpath $ wasmLauncher $ takeFileName mjspath
+    perms <- getPermissions outpath
+    setPermissions outpath $ setOwnerExecutable True perms
+  where
+    extra = case mode of
+      ToLibrary -> GenericWASM.libraryExports
+      _ -> GenericWASM.runServer
+    cpath = outpath `addExtension` "c"
+    mjspath = outpath `addExtension` "mjs"
+    classpath = outpath `addExtension` "class.js"
+
+-- | A Node.js script that runs the given ES module.  Unlike the module
+-- itself, it works whatever module type @package.json@ specifies.
+wasmLauncher :: FilePath -> T.Text
+wasmLauncher mjsname =
+  T.unlines
+    [ "#!/usr/bin/env node",
+      "import(\"./\" + encodeURIComponent(" <> jsString <> "));"
+    ]
+  where
+    jsString = T.decodeUtf8 $ BS.toStrict $ JSON.encode mjsname
+
 -- | The @futhark wasm@ action.
 compileCtoWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action SeqMem
 compileCtoWASMAction fcfg mode outpath =
   Action
-    { actionName = "Compile to sequential C",
-      actionDescription = "Compile to sequential C",
-      actionProcedure = helper
+    { actionName = "Compile to WebAssembly",
+      actionDescription = "Compile to sequential WebAssembly",
+      actionProcedure =
+        compileWASM mode outpath ["-lm"]
+          <=< handleWarnings fcfg . SequentialWASM.compileProg versionString
     }
-  where
-    helper prog = do
-      (cprog, jsprog, exps) <-
-        handleWarnings fcfg $ SequentialWASM.compileProg versionString prog
-      case mode of
-        ToLibrary -> do
-          writeLibs cprog jsprog
-          liftIO $ T.appendFile classpath SequentialWASM.libraryExports
-          runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm"] exps True
-        _ -> do
-          -- Non-server executables are not supported.
-          writeLibs cprog jsprog
-          liftIO $ T.appendFile classpath $ SequentialWASM.runServer $ SequentialWASM.cManifest cprog
-          runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm"] exps False
-    writeLibs cprog jsprog = do
-      let (h, imp, _) = SequentialC.asLibrary cprog
-      liftIO $ T.writeFile hpath h
-      liftIO $ T.writeFile cpath imp
-      liftIO $ T.writeFile classpath jsprog
 
-    cpath = outpath `addExtension` "c"
-    hpath = outpath `addExtension` "h"
-    mjspath = outpath `addExtension` "mjs"
-    classpath = outpath `addExtension` ".class.js"
-
 -- | The @futhark wasm-multicore@ action.
 compileMulticoreToWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action MCMem
 compileMulticoreToWASMAction fcfg mode outpath =
   Action
-    { actionName = "Compile to sequential C",
-      actionDescription = "Compile to sequential C",
-      actionProcedure = helper
+    { actionName = "Compile to WebAssembly",
+      actionDescription = "Compile to multicore WebAssembly",
+      actionProcedure =
+        compileWASM mode outpath ["-lm", "-pthread"]
+          <=< handleWarnings fcfg . MulticoreWASM.compileProg versionString
     }
-  where
-    helper prog = do
-      (cprog, jsprog, exps) <-
-        handleWarnings fcfg $ MulticoreWASM.compileProg versionString prog
-
-      case mode of
-        ToLibrary -> do
-          writeLibs cprog jsprog
-          liftIO $ T.appendFile classpath MulticoreWASM.libraryExports
-          runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps True
-        _ -> do
-          -- Non-server executables are not supported.
-          writeLibs cprog jsprog
-          liftIO $ T.appendFile classpath $ MulticoreWASM.runServer $ MulticoreWASM.cManifest cprog
-          runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps False
-
-    writeLibs cprog jsprog = do
-      let (h, imp, _) = MulticoreC.asLibrary cprog
-      liftIO $ T.writeFile hpath h
-      liftIO $ T.writeFile cpath imp
-      liftIO $ T.writeFile classpath jsprog
-
-    cpath = outpath `addExtension` "c"
-    hpath = outpath `addExtension` "h"
-    mjspath = outpath `addExtension` "mjs"
-    classpath = outpath `addExtension` ".class.js"
 
 runWith ::
   Name ->
diff --git a/src/Futhark/Analysis/Interference.hs b/src/Futhark/Analysis/Interference.hs
--- a/src/Futhark/Analysis/Interference.hs
+++ b/src/Futhark/Analysis/Interference.hs
@@ -117,6 +117,10 @@
           defbody : map caseBody cases
     Loop merge _ body ->
       analyseLoopParams merge <$> analyseBody lumap inuse_outside body
+    -- The memory of the accumulated arrays is that of the pattern, and is thus
+    -- in use throughout the lambda.
+    WithAcc _ lam ->
+      analyseLambda lumap inuse_outside lam
     Op (Inner (SegOp segop)) -> do
       analyseSegOp lumap inuse_outside segop
     _ ->
diff --git a/src/Futhark/CodeGen/Backends/GPU.hs b/src/Futhark/CodeGen/Backends/GPU.hs
--- a/src/Futhark/CodeGen/Backends/GPU.hs
+++ b/src/Futhark/CodeGen/Backends/GPU.hs
@@ -159,6 +159,16 @@
   GC.stm [C.cstm|$id:v = $exp:e;|]
 callKernel (LaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size) =
   genLaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size
+callKernel (GetBookkeeping v b) =
+  GC.stm
+    [C.cstm|$id:v = (struct memblock_device) {.references = NULL,
+                                              .mem = $exp:buf,
+                                              .size = $int:(4 * bookkeepingSize b),
+                                              .desc = $string:desc};|]
+  where
+    (buf, desc) = case b of
+      BookkeepingCounters -> ([C.cexp|ctx->bookkeeping_counters|], "bookkeeping_counters")
+      BookkeepingLocks -> ([C.cexp|ctx->bookkeeping_locks|], "bookkeeping_locks")
 
 copygpu2gpu :: GC.DoCopy op s
 copygpu2gpu _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do
@@ -472,6 +482,8 @@
   mapM_
     GC.earlyDecl
     [C.cunit|static const int max_failure_args = $int:max_failure_args;
+             static const typename int64_t bookkeeping_counters_size = $int:(bookkeepingSize BookkeepingCounters);
+             static const typename int64_t bookkeeping_locks_size = $int:(bookkeepingSize BookkeepingLocks);
              static const int f64_required = $exp:f64_required;
              static const char *gpu_program[] = {$inits:program_fragments};
              $esc:(T.unpack gpuPrototypesH)
diff --git a/src/Futhark/CodeGen/Backends/GenericC.hs b/src/Futhark/CodeGen/Backends/GenericC.hs
--- a/src/Futhark/CodeGen/Backends/GenericC.hs
+++ b/src/Futhark/CodeGen/Backends/GenericC.hs
@@ -50,16 +50,18 @@
 defCall dests fname args = do
   let out_args = [[C.cexp|&$id:d|] | d <- dests]
       args' = [C.cexp|ctx|] : out_args ++ args
-  item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; goto cleanup; }|]
+  failure <- failureJump
+  item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; $stm:failure }|]
 
 defError :: ErrorCompiler op s
 defError msg stacktrace = do
   (formatstr, formatargs) <- errorMsgString msg
   let formatstr' = "Error: " <> formatstr <> "\n\nBacktrace:\n%s"
+  failure <- failureJump
   items
     [C.citems|set_error(ctx, msgprintf($string:formatstr', $args:formatargs, $string:stacktrace));
               err = FUTHARK_PROGRAM_ERROR;
-              goto cleanup;|]
+              $stm:failure|]
 
 lmadcopyCPU :: DoCopy op s
 lmadcopyCPU _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do
@@ -657,6 +659,7 @@
       compileCode init_consts
     decl_mem <- declAllocatedMem
     free_mem <- freeAllocatedMem
+    error_label <- errorLabel
     libDecl
       [C.cedecl|static int init_constants($ty:ctx_ty *ctx) {
         (void)ctx;
@@ -666,6 +669,7 @@
         $items:init_consts'
         $items:free_mem
         $items:undefs
+        $items:error_label
         cleanup:
         return err;
       }|]
@@ -700,7 +704,7 @@
 
     freeConst ScalarParam {} = pure ()
     freeConst (MemParam name space) =
-      unRefMem [C.cexp|ctx->constants->$id:name|] space [C.cstm|err = 1;|]
+      unRefMem [C.cexp|ctx->constants->$id:name|] space $ pure [C.cstm|err = 1;|]
 
     getConst (ScalarParam name bt) = do
       let ctp = primTypeToCType bt
diff --git a/src/Futhark/CodeGen/Backends/GenericC/Code.hs b/src/Futhark/CodeGen/Backends/GenericC/Code.hs
--- a/src/Futhark/CodeGen/Backends/GenericC/Code.hs
+++ b/src/Futhark/CodeGen/Backends/GenericC/Code.hs
@@ -286,6 +286,25 @@
   stm [C.cstm|if (!$exp:e') { $items:err }|]
   where
     stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs
+compileCode (OnFailure body handler) = do
+  -- Failures in the body jump to its own error label, after which the handler
+  -- runs and the failure continues to the enclosing error label.
+  (body', error_label) <- withoutPendingFailures $ do
+    body' <- collect $ compileCode body
+    (body',) <$> errorLabel
+  if null error_label
+    then items [C.citems|{ $items:body' }|]
+    else do
+      handler' <- collect $ compileCode handler
+      failure <- failureJump
+      no_failure <- newVName "no_failure"
+      items
+        [C.citems|{ $items:body' }
+                  goto $id:no_failure;
+                  $items:error_label
+                  { $items:handler' }
+                  $stm:failure
+                  $id:no_failure: ;|]
 compileCode (Allocate _ _ ScalarSpace {}) =
   -- Handled by the declaration of the memory block, which is
   -- translated to an actual array.
@@ -294,19 +313,20 @@
   size <- compileExp e
   cached <- cacheMem name
   case cached of
-    Just cur_size ->
+    Just cur_size -> do
+      failure <- failureJump
       stm
         [C.cstm|if ($exp:cur_size < $exp:size) {
                  err = lexical_realloc(ctx, &$exp:name, &$exp:cur_size, $exp:size);
                  if (err != FUTHARK_SUCCESS) {
-                   goto cleanup;
+                   $stm:failure
                  }
                 }|]
     _ ->
-      allocMem name size space [C.cstm|{err = 1; goto cleanup;}|]
+      allocMem name size space memoryFailure
 compileCode (Free name space) = do
   cached <- isJust <$> cacheMem name
-  unless cached $ unRefMem name space [C.cstm|{err = 1; goto cleanup;}|]
+  unless cached $ unRefMem name space memoryFailure
 compileCode (For i bound body) = do
   let i' = C.toIdent i
       t = primTypeToCType $ primExpType bound
@@ -403,7 +423,7 @@
   src' <- compileExp src
   stm [C.cstm|$id:dest = $exp:src';|]
 compileCode (SetMem dest src space) =
-  setMem dest src space [C.cstm|{err = 1; goto cleanup;}|]
+  setMem dest src space
 compileCode (Call [dest] fname args)
   | isBuiltInFunction fname = do
       args' <- mapM compileArg args
diff --git a/src/Futhark/CodeGen/Backends/GenericC/Fun.hs b/src/Futhark/CodeGen/Backends/GenericC/Fun.hs
--- a/src/Futhark/CodeGen/Backends/GenericC/Fun.hs
+++ b/src/Futhark/CodeGen/Backends/GenericC/Fun.hs
@@ -32,7 +32,7 @@
     setRetVal' p (MemParam name space) =
       -- It is required that the memory block is already initialised
       -- (although it may be NULL).
-      setMem [C.cexp|*$exp:p|] name space [C.cstm|{err = 1; goto cleanup;}|]
+      setMem [C.cexp|*$exp:p|] name space
     setRetVal' p (ScalarParam name _) =
       stm [C.cstm|*$exp:p = $id:name;|]
 
@@ -64,6 +64,7 @@
     body' <- collect $ compileFunBody out_ptrs outputs body
     decl_mem <- declAllocatedMem
     free_mem <- freeAllocatedMem
+    error_label <- errorLabel
     let futhark_function =
           C.DeclSpec [] [C.EscTypeQual "FUTHARK_FUN_ATTR" mempty] (C.Tint Nothing mempty) mempty
 
@@ -76,6 +77,7 @@
                $items:decl_mem
                $items:get_constants
                $items:body'
+               $items:error_label
               cleanup:
                {
                $stms:free_cached
diff --git a/src/Futhark/CodeGen/Backends/GenericC/Monad.hs b/src/Futhark/CodeGen/Backends/GenericC/Monad.hs
--- a/src/Futhark/CodeGen/Backends/GenericC/Monad.hs
+++ b/src/Futhark/CodeGen/Backends/GenericC/Monad.hs
@@ -29,6 +29,10 @@
     generateProgramStruct,
     runCompilerM,
     inNewFunction,
+    failureJump,
+    memoryFailure,
+    errorLabel,
+    withoutPendingFailures,
     cachingMemory,
     volQuals,
     rawMem,
@@ -116,7 +120,10 @@
     compCtxFields :: DL.DList (C.Id, C.Type, Maybe C.Exp, Maybe (C.Stm, C.Stm)),
     compClearItems :: DL.DList C.BlockItem,
     compDeclaredMem :: [(VName, Space)],
-    compItems :: DL.DList C.BlockItem
+    compItems :: DL.DList C.BlockItem,
+    -- | The label that the failures since the previous 'errorLabel' jump to, if
+    -- there have been any.
+    compErrorLabel :: Maybe VName
   }
 
 newCompilerState :: VNameSource -> s -> CompilerState s
@@ -131,7 +138,8 @@
       compCtxFields = mempty,
       compClearItems = mempty,
       compDeclaredMem = mempty,
-      compItems = mempty
+      compItems = mempty,
+      compErrorLabel = Nothing
     }
 
 -- | In which part of the header file we put the declaration.  This is
@@ -252,7 +260,7 @@
 freeAllocatedMem :: CompilerM op s [C.BlockItem]
 freeAllocatedMem = collect $ mapM_ free =<< gets compDeclaredMem
   where
-    free (name, space) = unRefMem name space [C.cstm|err = 1;|]
+    free (name, space) = unRefMem name space $ pure [C.cstm|err = 1;|]
 
 declAllocatedMem :: CompilerM op s [C.BlockItem]
 declAllocatedMem = collect $ mapM_ f =<< gets compDeclaredMem
@@ -383,12 +391,50 @@
 inNewFunction m = do
   old_mem <- gets compDeclaredMem
   modify $ \s -> s {compDeclaredMem = mempty}
-  x <- local noCached m
+  x <- local noCached $ withoutPendingFailures m
   modify $ \s -> s {compDeclaredMem = old_mem}
   pure x
   where
     noCached env = env {envCachedMem = mempty}
 
+-- | A statement that jumps to the next error label; see 'errorLabel'. Don't do
+-- this without setting @err@ or similar first.
+failureJump :: CompilerM op s C.Stm
+failureJump = do
+  label <- maybe (newVName "failure") pure =<< gets compErrorLabel
+  modify $ \s -> s {compErrorLabel = Just label}
+  pure [C.cstm|goto $id:label;|]
+
+-- | What do do after failing a memory operation: set @err@ and jump as by
+-- 'failureJump'.
+memoryFailure :: CompilerM op s C.Stm
+memoryFailure = do
+  failure <- failureJump
+  pure [C.cstm|{err = 1; $stm:failure}|]
+
+-- | The label that the failures since the previous error label jump to, if
+-- there have been any. A failure thus jumps to the first error label that
+-- follows it in the program text. Every function that may fail must end with an
+-- error label just before its @cleanup@ label.
+errorLabel :: CompilerM op s [C.BlockItem]
+errorLabel = do
+  label <- gets compErrorLabel
+  modify $ \s -> s {compErrorLabel = Nothing}
+  pure $ case label of
+    Nothing -> []
+    Just label' -> [C.citems|$id:label': ;|]
+
+-- | Run an action as if no failures had happened before it, and afterwards
+-- restore the failures that had. The action must place the error labels for its
+-- own failures.
+withoutPendingFailures :: CompilerM op s a -> CompilerM op s a
+withoutPendingFailures m = do
+  old_label <- gets compErrorLabel
+  modify $ \s -> s {compErrorLabel = Nothing}
+  x <- m
+  modify $ \s -> s {compErrorLabel = old_label}
+  pure x
+
 -- | Insert a block item in the generated code at this point.
 item :: C.BlockItem -> CompilerM op s ()
 item x = modify $ \s -> s {compItems = DL.snoc (compItems s) x}
@@ -589,12 +635,14 @@
       when refcount $
         stm [C.cstm|$exp:mem.references = NULL;|]
 
-setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> C.Stm -> CompilerM op s ()
-setMem dest src space on_failure = do
+-- | Set @dest@ to @src@.  A failure is handled by 'memoryFailure'.
+setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> CompilerM op s ()
+setMem dest src space = do
   refcount <- fatMemory space
   let src_s = T.unpack $ expText $ C.toExp src noLoc
   if refcount
-    then
+    then do
+      on_failure <- memoryFailure
       stm
         [C.cstm|if ($id:(fatMemSet space)(ctx, &$exp:dest, &$exp:src,
                                                $string:src_s) != 0) {
@@ -613,15 +661,17 @@
                   }|]
       _ -> stm [C.cstm|$exp:dest = $exp:src;|]
 
-unRefMem :: (C.ToExp a) => a -> Space -> C.Stm -> CompilerM op s ()
+-- | The failure code is only generated if it is needed.
+unRefMem :: (C.ToExp a) => a -> Space -> CompilerM op s C.Stm -> CompilerM op s ()
 unRefMem mem space on_failure = do
   refcount <- fatMemory space
   cached <- isJust <$> cacheMem mem
   let mem_s = T.unpack $ expText $ C.toExp mem noLoc
-  when (refcount && not cached) $
+  when (refcount && not cached) $ do
+    on_failure' <- on_failure
     stm
       [C.cstm|if ($id:(fatMemUnRef space)(ctx, &$exp:mem, $string:mem_s) != 0) {
-                  $stm:on_failure
+                  $stm:on_failure'
                 }|]
 
 allocMem ::
@@ -629,22 +679,23 @@
   a ->
   b ->
   Space ->
-  C.Stm ->
+  CompilerM op s C.Stm ->
   CompilerM op s ()
 allocMem mem size space on_failure = do
   refcount <- fatMemory space
   let mem_s = T.unpack $ expText $ C.toExp mem noLoc
+  on_failure' <- on_failure
   if refcount
     then
       stm
         [C.cstm|if ($id:(fatMemAlloc space)(ctx, &$exp:mem, $exp:size,
                                                  $string:mem_s)) {
-                       $stm:on_failure
+                       $stm:on_failure'
                      }|]
     else do
       freeRawMem mem size space mem_s
       alloc <- allocRawMem mem size space [C.cexp|desc|]
-      stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure }|]
+      stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure' }|]
 
 copyMemoryDefaultSpace ::
   C.Exp ->
diff --git a/src/Futhark/CodeGen/Backends/GenericC/Types.hs b/src/Futhark/CodeGen/Backends/GenericC/Types.hs
--- a/src/Futhark/CodeGen/Backends/GenericC/Types.hs
+++ b/src/Futhark/CodeGen/Backends/GenericC/Types.hs
@@ -56,7 +56,7 @@
     [C.cexp|$exp:arr->mem|]
     [C.cexp|$exp:arr_size * $int:(primByteSize pt::Int)|]
     space
-    [C.cstm|err = 1;|]
+    (pure [C.cstm|err = 1;|])
 
 arrayLibraryFunctions ::
   Publicness ->
@@ -114,7 +114,7 @@
       let dim_s = "dim" ++ show i
        in stm [C.cstm|arr->shape[$int:i] = $id:dim_s;|]
 
-  free_body <- collect $ unRefMem [C.cexp|arr->mem|] space [C.cstm|ret = 1;|]
+  free_body <- collect $ unRefMem [C.cexp|arr->mem|] space $ pure [C.cstm|ret = 1;|]
 
   values_body <-
     collect $
diff --git a/src/Futhark/CodeGen/Backends/GenericPython.hs b/src/Futhark/CodeGen/Backends/GenericPython.hs
--- a/src/Futhark/CodeGen/Backends/GenericPython.hs
+++ b/src/Futhark/CodeGen/Backends/GenericPython.hs
@@ -1355,6 +1355,10 @@
   cond' <- compileExp $ Imp.untyped cond
   body' <- collect $ compileCode body
   stm $ While cond' body'
+compileCode (Imp.OnFailure body handler) = do
+  body' <- collect $ compileCode body
+  handler' <- collect $ compileCode handler
+  stm $ Try body' [Catch (Var "Exception") $ handler' ++ [Raise $ Var "e"]]
 compileCode (Imp.For i bound body) = do
   bound' <- compileExp bound
   let i' = compileName i
diff --git a/src/Futhark/CodeGen/Backends/GenericWASM.hs b/src/Futhark/CodeGen/Backends/GenericWASM.hs
--- a/src/Futhark/CodeGen/Backends/GenericWASM.hs
+++ b/src/Futhark/CodeGen/Backends/GenericWASM.hs
@@ -1,126 +1,22 @@
 {-# LANGUAGE QuasiQuotes #-}
 
+-- | Code generation shared by the WASM backends.  The JavaScript API is
+-- in @rts/javascript/wrappers.js@, driven by the manifest.
 module Futhark.CodeGen.Backends.GenericWASM
-  ( GC.CParts (..),
-    GC.asLibrary,
-    GC.asExecutable,
-    GC.asServer,
-    EntryPointType,
-    JSEntryPoint (..),
-    JSRecordField (..),
-    JSOpaqueType (..),
-    emccExportNames,
-    manifestFunctions,
+  ( emccExportNames,
     javascriptWrapper,
-    extToString,
-    opaqueToJS,
     runServer,
     libraryExports,
   )
 where
 
-import Data.List (intercalate)
 import Data.Map qualified as M
 import Data.Text qualified as T
-import Futhark.CodeGen.Backends.GenericC qualified as GC
-import Futhark.CodeGen.Backends.SimpleRep (isValidCName, opaqueName)
-import Futhark.CodeGen.ImpCode.Sequential qualified as Imp
 import Futhark.CodeGen.RTS.JavaScript
 import Futhark.Manifest
-import Futhark.Util (nubOrd, showText, zEncodeText)
-import Language.Futhark.Core (nameToText)
-import Language.Futhark.Primitive
+import Futhark.Util (nubOrd)
 import NeatInterpolation (text)
 
-extToString :: Imp.ExternalValue -> String
-extToString (Imp.TransparentValue (Imp.ArrayValue vn _ pt s dimSize)) =
-  concat (replicate (length dimSize) "[]") ++ extToString (Imp.TransparentValue (Imp.ScalarValue pt s vn))
-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float16) _ _)) = "f16"
-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float32) _ _)) = "f32"
-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float64) _ _)) = "f64"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Signed _)) = "i8"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Signed _)) = "i16"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Signed _)) = "i32"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Signed _)) = "i64"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Unsigned _)) = "u8"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Unsigned _)) = "u16"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Unsigned _)) = "u32"
-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Unsigned _)) = "u64"
-extToString (Imp.TransparentValue (Imp.ScalarValue Bool _ _)) = "bool"
-extToString (Imp.TransparentValue (Imp.ScalarValue Unit _ _)) = error "extToString: Unit"
-extToString (Imp.OpaqueValue oname _) = T.unpack $ opaqueName oname
-
-type EntryPointType = String
-
-data JSEntryPoint = JSEntryPoint
-  { name :: String,
-    parameters :: [EntryPointType],
-    ret :: EntryPointType
-  }
-
--- | A field in a JavaScript record opaque type.
-data JSRecordField = JSRecordField
-  { -- | Original field name (used in the server protocol).
-    jsrfName :: String,
-    -- | JavaScript type string (e.g. @"i32"@, @"[]f64"@, @"opaque_foo"@).
-    jsrfType :: EntryPointType,
-    -- | JavaScript method name on @FutharkContext@ (e.g. @"project_opaque_foo_x"@).
-    jsrfProjectFn :: String
-  }
-
--- | How an opaque type is represented for the JavaScript server.
-data JSOpaqueType
-  = -- | A record type with named, projectable fields.
-    JSOpaqueRecord [JSRecordField]
-  | -- | Any other opaque type (no project support).
-    JSOpaqueOther
-
--- | Convert the opaque-type table from an ImpCode program into the
--- JavaScript-oriented representation used by the code generator.
-opaqueToJS :: Imp.OpaqueTypes -> [(String, JSOpaqueType)]
-opaqueToJS (Imp.OpaqueTypes types) = map convertOne types
-  where
-    convertOne (desc, (Imp.OpaqueRecord fields, _)) =
-      (T.unpack (opaqueName desc), JSOpaqueRecord (map (convertField desc) fields))
-    convertOne (desc, _) =
-      (T.unpack (opaqueName desc), JSOpaqueOther)
-
-    convertField desc (fname, etype) =
-      JSRecordField
-        { jsrfName = T.unpack (nameToText fname),
-          jsrfType = entryTypeToJSString etype,
-          jsrfProjectFn =
-            "project_"
-              ++ T.unpack (opaqueName desc)
-              ++ "_"
-              ++ T.unpack f'
-        }
-      where
-        f'
-          | isValidCName (opaqueName desc <> "_" <> nameToText fname) =
-              nameToText fname
-          | otherwise = zEncodeText (nameToText fname)
-
--- | Convert an 'Imp.EntryPointType' to its JavaScript string representation.
-entryTypeToJSString :: Imp.EntryPointType -> String
-entryTypeToJSString (Imp.TypeOpaque desc) = T.unpack $ opaqueName desc
-entryTypeToJSString (Imp.TypeTransparent (Imp.ValueType sign (Imp.Rank rank) pt)) =
-  concat (replicate rank "[]") ++ primToJSString sign pt
-  where
-    primToJSString _ (FloatType Float16) = "f16"
-    primToJSString _ (FloatType Float32) = "f32"
-    primToJSString _ (FloatType Float64) = "f64"
-    primToJSString Imp.Signed (IntType Int8) = "i8"
-    primToJSString Imp.Signed (IntType Int16) = "i16"
-    primToJSString Imp.Signed (IntType Int32) = "i32"
-    primToJSString Imp.Signed (IntType Int64) = "i64"
-    primToJSString Imp.Unsigned (IntType Int8) = "u8"
-    primToJSString Imp.Unsigned (IntType Int16) = "u16"
-    primToJSString Imp.Unsigned (IntType Int32) = "u32"
-    primToJSString Imp.Unsigned (IntType Int64) = "u64"
-    primToJSString _ Bool = "bool"
-    primToJSString _ Unit = error "entryTypeToJSString: Unit"
-
 -- | The names of the C functions that must be exported from the WebAssembly
 -- module. This is every function named in the manifest, as the server calls
 -- these directly, as well as the context management functions.
@@ -129,6 +25,13 @@
   map (\f -> "'_" ++ T.unpack f ++ "'") (nubOrd $ manifestFunctions manifest)
     ++ [ "_futhark_context_config_new",
          "_futhark_context_config_free",
+         "_futhark_context_config_set_debugging",
+         "_futhark_context_config_set_logging",
+         "_futhark_context_config_set_profiling",
+         "_futhark_context_config_set_tuning_param",
+         "_futhark_get_tuning_param_count",
+         "_futhark_get_tuning_param_name",
+         "_futhark_get_tuning_param_class",
          "_futhark_context_new",
          "_futhark_context_free",
          "_futhark_context_get_error",
@@ -180,460 +83,29 @@
 
     variantFunctions v = [sumVariantConstruct v, sumVariantDestruct v]
 
-javascriptWrapper :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text
-javascriptWrapper entryPoints opaqueTypes =
+-- | The JavaScript code passed to @emcc@ with @--extern-post-js@.
+javascriptWrapper :: Manifest -> T.Text
+javascriptWrapper manifest =
   T.unlines
     [ serverJs,
       valuesJs,
-      wrapperclassesJs,
-      classFutharkContext entryPoints opaqueTypes
+      wrappersJs,
+      "const futhark_manifest = " <> manifestToJSON manifest <> ";"
     ]
 
--- Make FutharkModule the generated primary class, but keeps FutharkContext around for backwards compatibility.
-classFutharkContext :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text
-classFutharkContext entryPoints opaqueTypes =
-  T.unlines
-    [ "class FutharkModule {",
-      moduleConstructor,
-      moduleInitFromWasm entryPoints opaqueTypes,
-      moduleInit,
-      getFreeFun,
-      getEntryPointsFun,
-      getTypesFun,
-      getErrorFun,
-      getUtilityFuns,
-      T.unlines $ map toFutharkArray arrays,
-      T.unlines $ concatMap (generateProjectMethods . snd) opaqueTypes,
-      T.unlines $ map jsWrapEntryPoint entryPoints,
-      "}",
-      classFutharkContextCompat,
-      [text|
-      async function newFutharkContext(num_threads) {
-        var wasm = await loadWASM();
-        return new FutharkContext(wasm, num_threads);
-      }
-      |]
-    ]
-  where
-    -- Collect array types from entry points AND from record fields so the
-    -- array constructors are always available when returning projected values.
-    arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)
-    entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) entryPoints
-    recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]
-
-moduleConstructor :: T.Text
-moduleConstructor =
-  [text|
-  constructor() {
-    this.wasm = undefined;
-    this.cfg = undefined;
-    this.ctx = undefined;
-    this.entry_points = {};
-    this.types = {};
-    this.entry = {};
-  }
-  |]
-
-moduleInitFromWasm :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text
-moduleInitFromWasm jses opaqueTypes =
-  [text|
-  _init_from_wasm(wasm, num_threads) {
-    this.wasm = wasm;
-    this.cfg = this.wasm._futhark_context_config_new();
-    if (num_threads) this.wasm._futhark_context_config_set_num_threads(this.cfg, num_threads);
-    this.ctx = this.wasm._futhark_context_new(this.cfg);
-
-    this.entry_points = {
-      ${entries}
-    };
-    this.types = {
-      ${type_entries}
-    };
-    this.entry = {};
-    ${entry_aliases}
-    ${array_aliases}
-    ${array_type_aliases}
-  }
-  |]
-  where
-    entries = T.intercalate "," $ map dicEntry jses
-    type_entries = T.intercalate "," $ map dicTypeEntry opaqueTypes
-    entry_aliases = T.unlines $ map entryAlias jses
-    array_aliases = T.unlines $ map arrayAlias arrays
-    array_type_aliases = T.unlines $ map arrayTypeAlias arrays
-
-    arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)
-    entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) jses
-    recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]
-
-moduleInit :: T.Text
-moduleInit =
-  [text|
-  async init(wasm, num_threads) {
-    if (wasm === undefined) {
-      throw new Error("FutharkModule.init() requires the generated backend runtime module");
-    }
-
-    this._init_from_wasm(wasm, num_threads);
-  }
-  |]
-
-classFutharkContextCompat :: T.Text
-classFutharkContextCompat =
-  [text|
-  class FutharkContext extends FutharkModule {
-    constructor(wasm, num_threads) {
-      super();
-
-      if (wasm !== undefined) {
-        this._init_from_wasm(wasm, num_threads);
-      }
-    }
-  }
-  |]
-
-getFreeFun :: T.Text
-getFreeFun =
-  [text|
-  free() {
-    if (this.ctx !== undefined) {
-      this.wasm._futhark_context_free(this.ctx);
-      this.ctx = undefined;
-    }
-
-    if (this.cfg !== undefined) {
-      this.wasm._futhark_context_config_free(this.cfg);
-      this.cfg = undefined;
-    }
-  }
-  |]
-
-getEntryPointsFun :: T.Text
-getEntryPointsFun =
-  [text|
-  get_entry_points() {
-    return this.entry_points;
-  }
-  |]
-
-getTypesFun :: T.Text
-getTypesFun =
-  [text|
-  get_types() {
-    return this.types;
-  }
-  |]
-
-getErrorFun :: T.Text
-getErrorFun =
-  [text|
-  get_error() {
-    var ptr = this.wasm._futhark_context_get_error(this.ctx);
-    var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);
-    var str = String.fromCharCode(...this.wasm.HEAP8.subarray(ptr, ptr + len));
-    this.wasm._free(ptr);
-    return str;
-  }
-  |]
-
-getUtilityFuns :: T.Text
-getUtilityFuns =
-  [text|
-  async context_sync() {
-    return this.wasm._futhark_context_sync(this.ctx);
-  }
-
-  async clear_caches() {
-    return this.wasm._futhark_context_clear_caches(this.ctx);
-  }
-
-  async report() {
-    var ptr = this.wasm._futhark_context_report(this.ctx);
-    var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);
-    var bytes = this.wasm.HEAPU8.subarray(ptr, ptr + len);
-    var str = new TextDecoder().decode(bytes);
-    this.wasm._free(ptr);
-    return str;
-  }
-
-  async pause_profiling() {
-    return this.wasm._futhark_context_pause_profiling(this.ctx);
-  }
-
-  async unpause_profiling() {
-    return this.wasm._futhark_context_unpause_profiling(this.ctx);
-  }
-  |]
-
-entryAlias :: JSEntryPoint -> T.Text
-entryAlias jse =
-  [text|this.entry["${ename}"] = this.${fname}.bind(this);|]
-  where
-    fname = GC.escapeName $ T.pack $ name jse
-    ename = T.pack $ name jse
-
-arrayAlias :: String -> T.Text
-arrayAlias typ =
-  [text|
-  this.${signature} = {
-    from_data: (data, ${dims}) => this.new_${signature}(data, ${dims}),
-    from_jsarray: (data) => this.new_${signature}_from_jsarray(data)
-  };
-  |]
-  where
-    d = dim typ
-    ftype = baseType typ
-    signature = T.pack $ ftype ++ "_" ++ show d ++ "d"
-    dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]
-
-arrayTypeAlias :: String -> T.Text
-arrayTypeAlias typ =
-  [text|this.types["${typ_text}"] = this.${signature};|]
-  where
-    d = dim typ
-    ftype = baseType typ
-    signature = T.pack $ ftype ++ "_" ++ show d ++ "d"
-    typ_text = T.pack typ
-
-dicEntry :: JSEntryPoint -> T.Text
-dicEntry jse =
-  [text|
-       "${ename}" : ["${fname}", ${params}, ${rets}]
-  |]
-  where
-    fname = GC.escapeName $ T.pack $ name jse
-    ename = T.pack $ name jse
-    params = showText $ parameters jse
-    rets = showText $ ret jse
-
--- | Generate the @types@ dictionary entry for a single opaque type.
-dicTypeEntry :: (String, JSOpaqueType) -> T.Text
-dicTypeEntry (tname, JSOpaqueRecord fields) =
-  T.pack $
-    show tname
-      ++ ": [\"record\", ["
-      ++ intercalate ", " (map fieldEntry fields)
-      ++ "]]"
-  where
-    fieldEntry rf =
-      "["
-        ++ show (jsrfName rf)
-        ++ ", "
-        ++ show (jsrfType rf)
-        ++ ", \""
-        ++ jsrfProjectFn rf
-        ++ "\"]"
-dicTypeEntry (tname, JSOpaqueOther) =
-  T.pack $ show tname ++ ": [\"opaque\"]"
-
--- | Generate @project_*@ wrapper methods for a record opaque type.
-generateProjectMethods :: JSOpaqueType -> [T.Text]
-generateProjectMethods (JSOpaqueRecord fields) = map generateProjectMethod fields
-generateProjectMethods JSOpaqueOther = []
-
-generateProjectMethod :: JSRecordField -> T.Text
-generateProjectMethod rf =
-  T.pack $
-    unlines
-      [ "  " ++ jsrfProjectFn rf ++ "(obj) {",
-        "    var out = this.wasm._malloc(" ++ show (typeSize ftype) ++ ");",
-        "    this.wasm._futhark_" ++ jsrfProjectFn rf ++ "(this.ctx, out, obj.ptr);",
-        "    var result = " ++ readResult ++ ";",
-        "    this.wasm._free(out);",
-        "    return result;",
-        "  }"
-      ]
-  where
-    ftype = jsrfType rf
-    readout = typeHeap ftype ++ "[out >> " ++ show (typeShift ftype) ++ "]"
-    readResult
-      | isArray ftype =
-          "this.new_" ++ baseType ftype ++ "_" ++ show (dim ftype) ++ "d_from_ptr(" ++ readout ++ ")"
-      | isOpaque ftype =
-          "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ ftype ++ ")"
-      | ftype == "bool" = readout ++ "!==0"
-      | otherwise = readout
-
-jsWrapEntryPoint :: JSEntryPoint -> T.Text
-jsWrapEntryPoint jse =
-  [text|
-  ${func_name}(${inparams}) {
-    var out = [${outparams}].map(n => this.wasm._malloc(n));
-    var to_free = [];
-    var do_free = () => { out.forEach(this.wasm._free); to_free.forEach(f => f.free()); };
-    ${paramsToPtr}
-    if (this.wasm._futhark_entry_${func_name}(this.ctx, ...out, ${ins}) > 0) {
-      do_free();
-      throw this.get_error();
-    }
-    ${results}
-    do_free();
-    return ${res};
-  }
-  |]
-  where
-    func_name = GC.escapeName $ T.pack $ name jse
-
-    alp = [0 .. length (parameters jse) - 1]
-    inparams = T.pack $ intercalate ", " ["in" ++ show i | i <- alp]
-    ins = T.pack $ intercalate ", " [maybeDerefence ("in" ++ show i) $ parameters jse !! i | i <- alp]
-    paramsToPtr = T.pack $ unlines $ filter ("" /=) [arrayPointer ("in" ++ show i) $ parameters jse !! i | i <- alp]
-
-    outparams = showText $ typeSize $ ret jse
-    results = T.pack $ makeResult 0 $ ret jse
-    res = "result0"
-
-maybeDerefence :: String -> String -> String
-maybeDerefence arg typ =
-  if isScalar typ then arg else arg ++ ".ptr"
-
-arrayPointer :: String -> String -> String
-arrayPointer arg typ =
-  if isArray typ
-    then "  if (" ++ arg ++ " instanceof Array) { " ++ reassign ++ "; to_free.push(" ++ arg ++ "); }"
-    else ""
-  where
-    reassign = arg ++ " = this.new_" ++ signature ++ "_from_jsarray(" ++ arg ++ ")"
-    signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"
-
-makeResult :: Int -> String -> String
-makeResult i typ =
-  "  var result"
-    ++ show i
-    ++ " = "
-    ++ if isArray typ
-      then "this.new_" ++ signature ++ "_from_ptr(" ++ readout ++ ");"
-      else
-        if isOpaque typ
-          then "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ typ ++ ");"
-          else readout ++ if typ == "bool" then "!==0;" else ";"
-  where
-    res = "out[" ++ show i ++ "]"
-    readout = typeHeap typ ++ "[" ++ res ++ " >> " ++ show (typeShift typ) ++ "]"
-    signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"
-
-baseType :: String -> String
-baseType ('[' : ']' : end) = baseType end
-baseType typ = typ
-
-dim :: String -> Int
-dim ('[' : ']' : end) = dim end + 1
-dim _ = 0
-
-isArray :: String -> Bool
-isArray typ = take 2 typ == "[]"
-
-isOpaque :: String -> Bool
-isOpaque typ = take 6 typ == "opaque"
-
-isScalar :: String -> Bool
-isScalar typ = not (isArray typ || isOpaque typ)
-
-typeSize :: String -> Integer
-typeSize typ =
-  case typ of
-    "i8" -> 1
-    "i16" -> 2
-    "i32" -> 4
-    "i64" -> 8
-    "u8" -> 1
-    "u16" -> 2
-    "u32" -> 4
-    "u64" -> 8
-    "f16" -> 2
-    "f32" -> 4
-    "f64" -> 8
-    "bool" -> 1
-    _ -> 4
-
-typeShift :: String -> Integer
-typeShift typ =
-  case typ of
-    "i8" -> 0
-    "i16" -> 1
-    "i32" -> 2
-    "i64" -> 3
-    "u8" -> 0
-    "u16" -> 1
-    "u32" -> 2
-    "u64" -> 3
-    "f16" -> 1
-    "f32" -> 2
-    "f64" -> 3
-    "bool" -> 0
-    _ -> 2
-
-typeHeap :: String -> String
-typeHeap typ =
-  case typ of
-    "i8" -> "this.wasm.HEAP8"
-    "i16" -> "this.wasm.HEAP16"
-    "i32" -> "this.wasm.HEAP32"
-    "i64" -> "this.wasm.HEAP64"
-    "u8" -> "this.wasm.HEAPU8"
-    "u16" -> "this.wasm.HEAPU16"
-    "u32" -> "this.wasm.HEAPU32"
-    "u64" -> "(new BigUint64Array(this.wasm.HEAP64.buffer))"
-    "f16" -> "this.wasm.HEAPU16"
-    "f32" -> "this.wasm.HEAPF32"
-    "f64" -> "this.wasm.HEAPF64"
-    "bool" -> "this.wasm.HEAP8"
-    _ -> "this.wasm.HEAP32"
-
-toFutharkArray :: String -> T.Text
-toFutharkArray typ =
-  [text|
-  ${new}_from_jsarray(${arraynd_p}) {
-    return this.${new}(${arraynd_flat_p}, ${arraynd_dims_p});
-  }
-  ${new}(array, ${dims}) {
-    console.assert(array.length === ${dims_multiplied}, 'len=%s,dims=%s', array.length, [${dims}].toString());
-      var copy = this.wasm._malloc(array.length << ${shift});
-      ${heapType}.set(array, copy >> ${shift});
-      var ptr = ${fnew}(this.ctx, copy, ${bigint_dims});
-      this.wasm._free(copy);
-      return this.${new}_from_ptr(ptr);
-    }
-
-    ${new}_from_ptr(ptr) {
-      return new FutharkArray(this, ptr, ${args});
-    }
-    |]
-  where
-    d = dim typ
-    ftype = baseType typ
-    heap = typeHeap ftype
-    signature = ftype ++ "_" ++ show d ++ "d"
-    new = T.pack $ "new_" ++ signature
-    fnew = T.pack $ "this.wasm._futhark_new_" ++ signature
-    fshape = "this.wasm._futhark_shape_" ++ signature
-    fvalues = "this.wasm._futhark_values_raw_" ++ signature
-    ffree = "this.wasm._futhark_free_" ++ signature
-    arraynd = "array" ++ show d ++ "d"
-    shift = showText (typeShift ftype)
-    heapType = T.pack heap
-    arraynd_flat = if d > 1 then arraynd ++ ".flat()" else arraynd
-    arraynd_dims = intercalate ", " [arraynd ++ mult i "[0]" ++ ".length" | i <- [0 .. d - 1]]
-    dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]
-    dims_multiplied = T.pack $ intercalate "*" ["Number(d" ++ show i ++ ")" | i <- [0 .. d - 1]]
-    bigint_dims = T.pack $ intercalate ", " ["BigInt(d" ++ show i ++ ")" | i <- [0 .. d - 1]]
-    mult i s = concat $ replicate i s
-    (arraynd_p, arraynd_flat_p, arraynd_dims_p) = (T.pack arraynd, T.pack arraynd_flat, T.pack arraynd_dims)
-    args = T.pack $ intercalate ", " ["'" ++ ftype ++ "'", show d, heap, fshape, fvalues, ffree]
-
 -- | Javascript code that can be appended to the generated module to
--- run a Futhark server instance on startup.
-runServer :: Manifest -> T.Text
-runServer manifest =
+-- run a Futhark server instance when it is loaded.
+runServer :: T.Text
+runServer =
   [text|
-   Module.onRuntimeInitialized = () => {
-     var context = new FutharkContext(Module);
-     var server = new Server(context, ${manifest_json});
-     server.run();
+   // Worker threads (wasm-multicore) also load this module.
+   const is_main_thread = globalThis.process?.versions?.node
+     ? (await import("node:worker_threads")).isMainThread
+     : globalThis.WorkerGlobalScope === undefined;
+   if (is_main_thread) {
+     await futhark_server_main();
    }|]
-  where
-    manifest_json = manifestToJSON manifest
 
 -- | The names exported by the generated module.
 libraryExports :: T.Text
-libraryExports = "export {newFutharkContext, FutharkContext, FutharkModule, FutharkArray, FutharkOpaque};"
+libraryExports = "export {FutharkContext, FutharkArray, FutharkOpaque};"
diff --git a/src/Futhark/CodeGen/Backends/MulticoreC.hs b/src/Futhark/CodeGen/Backends/MulticoreC.hs
--- a/src/Futhark/CodeGen/Backends/MulticoreC.hs
+++ b/src/Futhark/CodeGen/Backends/MulticoreC.hs
@@ -110,8 +110,8 @@
 compileRetvalStructFields :: [VName] -> [(C.Type, C.Type, ValueType)] -> [C.FieldGroup]
 compileRetvalStructFields = zipWith field
   where
-    field name (ty, _, Prim _) =
-      [C.csdecl|$ty:ty *$id:(closureRetvalStructField name);|]
+    field name (_, storage_ty, Prim _) =
+      [C.csdecl|$ty:storage_ty *$id:(closureRetvalStructField name);|]
     field name (ty, _, ScalarMem) =
       [C.csdecl|$ty:ty $id:(closureRetvalStructField name);|]
     field name (_, _, _) =
@@ -134,32 +134,55 @@
     field name (_, _, ScalarMem) =
       [C.cstm|$id:struct.$id:(closureFreeStructField name)=$id:name;|]
 
+-- | Point the retval fields of a task struct at the variables that will receive
+-- the results, and return the statements that must be executed after the task
+-- has run. A scalar retval field points to the storage representation of its
+-- type (see 'compileRetvalStructFields'). When that differs from the
+-- representation of the variable itself, the field instead points to a
+-- temporary in storage representation, which is converted back after the task
+-- has run. This matters for ISPC, where the variable lives in ISPC code but is
+-- written by C code, and the two may not agree on the representation of
+-- 'Float16'.
 compileSetRetvalStructValues ::
   (C.ToIdent a) =>
   a ->
   [VName] ->
   [(C.Type, C.Type, ValueType)] ->
-  [C.Stm]
-compileSetRetvalStructValues struct vnames we = concat $ zipWith field vnames we
+  GC.CompilerM op s [C.Stm]
+compileSetRetvalStructValues struct vnames we = concat <$> zipWithM field vnames we
   where
-    field name (ct, _, Prim _) =
-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ct*)&$id:name);
-               $escstm:("#if defined(ISPC)")
-               $id:struct.$id:(closureRetvalStructField name)+= programIndex;
-               $escstm:("#endif")|]
-    field name (_, _, MemBlock) =
-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]
-    field name (_, _, RawMem) =
-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]
-    field name (_, _, ScalarMem) =
-      [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]
+    field name (ct, storage_ty, Prim pt)
+      | ct == storage_ty = do
+          point name storage_ty
+          pure []
+      | otherwise = do
+          tmp <- newVName $ baseName name <> "_storage"
+          GC.decl [C.cdecl|$ty:storage_ty $id:tmp = $exp:(toStorage pt (C.toExp name noLoc));|]
+          point tmp storage_ty
+          pure [[C.cstm|$id:name = $exp:(fromStorage pt (C.toExp tmp noLoc));|]]
+      where
+        point v ty =
+          GC.stms
+            [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ty*)&$id:v);
+                     $escstm:("#if defined(ISPC)")
+                     $id:struct.$id:(closureRetvalStructField name)+= programIndex;
+                     $escstm:("#endif")|]
+    field name (_, _, MemBlock) = do
+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]
+      pure []
+    field name (_, _, RawMem) = do
+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]
+      pure []
+    field name (_, _, ScalarMem) = do
+      GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]
+      pure []
 
 compileGetRetvalStructVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.InitGroup]
 compileGetRetvalStructVals struct = zipWith field
   where
-    field name (ty, _storage_ty, Prim _) =
+    field name (ty, _storage_ty, Prim pt) =
       let inner = [C.cexp|*$id:struct->$id:(closureRetvalStructField name)|]
-       in [C.cdecl|$ty:ty $id:name = $exp:inner;|]
+       in [C.cdecl|$ty:ty $id:name = $exp:(fromStorage pt inner);|]
     field name (ty, _, ScalarMem) =
       [C.cdecl|$ty:ty $id:name = $id:struct->$id:(closureRetvalStructField name);|]
     field name (ty, _, _) =
@@ -190,8 +213,8 @@
 compileWriteBackResVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.Stm]
 compileWriteBackResVals struct = zipWith field
   where
-    field name (_, _, Prim _) =
-      [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(C.toExp name noLoc);|]
+    field name (_, _, Prim pt) =
+      [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(toStorage pt (C.toExp name noLoc));|]
     field name (_, _, ScalarMem) =
       [C.cstm|$id:struct->$id:(closureRetvalStructField name) = $id:name;|]
     field name (_, _, _) =
@@ -286,6 +309,7 @@
         mapM_ GC.item =<< GC.declAllocatedMem
         mapM_ GC.item code'
         free_mem <- GC.freeAllocatedMem
+        mapM_ GC.item =<< GC.errorLabel
         GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|]
     pure
       [C.cedecl|int $id:s(void *args, typename int64_t iterations, int tid, struct scheduler_info info) {
@@ -307,7 +331,7 @@
   [(C.Type, C.Type, ValueType)] ->
   [VName] ->
   [(C.Type, C.Type, ValueType)] ->
-  GC.CompilerM Multicore s Name
+  GC.CompilerM Multicore s (Name, [C.Stm])
 prepareTaskStruct def name free_args free_ctypes retval_args retval_ctypes = do
   let makeStruct s =
         pure
@@ -321,8 +345,8 @@
   GC.decl [C.cdecl|struct $id:fstruct $id:fstruct';|]
   GC.stm [C.cstm|$id:fstruct'.ctx = ctx;|]
   GC.stms [C.cstms|$stms:(compileSetStructValues fstruct' free_args free_ctypes)|]
-  GC.stms [C.cstms|$stms:(compileSetRetvalStructValues fstruct' retval_args retval_ctypes)|]
-  pure fstruct
+  after <- compileSetRetvalStructValues fstruct' retval_args retval_ctypes
+  pure (fstruct, after)
 
 -- Generate a segop function for top_level and potentially nested SegOp code
 compileOp :: GC.OpCompiler Multicore s
@@ -346,7 +370,7 @@
 
   let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] params mempty seq_code
 
-  fstruct <-
+  (fstruct, read_retvals) <-
     prepareTaskStruct multicoreDef "task" free_args free_ctypes retval_args retval_ctypes
 
   fpar_task <- generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval
@@ -375,22 +399,24 @@
     Nothing ->
       GC.stm [C.cstm|$id:ftask_name.nested_fn=NULL;|]
 
+  failure <- GC.failureJump
   let ftask_err = fpar_task <> "_err"
       code =
         [C.citems|int $id:ftask_err = scheduler_prepare_task(&ctx->scheduler, &$id:ftask_name);
                   if ($id:ftask_err != 0) {
                     err = $id:ftask_err;
-                    goto cleanup;
+                    $stm:failure
                   }|]
 
   mapM_ GC.item code
+  GC.stms read_retvals
 compileOp (ParLoop s' body free) = do
   free_ctypes <- mapM paramToCType free
   let free_args = map paramName free
 
   let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] free mempty body
 
-  fstruct <-
+  (fstruct, _) <-
     prepareTaskStruct multicoreDef (s' <> "_parloop_struct") free_args free_ctypes mempty mempty
 
   ftask <- multicoreDef (s' <> "_parloop") $ \s -> do
@@ -406,6 +432,7 @@
         mapM_ GC.item =<< GC.declAllocatedMem
         free_mem <- GC.freeAllocatedMem
         mapM_ GC.item body'
+        mapM_ GC.item =<< GC.errorLabel
         GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|]
     pure
       [C.cedecl|static int $id:s(void *args,
@@ -432,6 +459,7 @@
 
   let ftask_err = ftask <> "_err"
       ftask_total = ftask <> "_total"
+  failure <- GC.failureJump
   code' <-
     benchmarkCode
       ftask_total
@@ -439,7 +467,7 @@
                                                            &$id:ftask_name);
                if ($id:ftask_err != 0) {
                  err = $id:ftask_err;
-                 goto cleanup;
+                 $stm:failure
                }|]
 
   mapM_ GC.item code'
diff --git a/src/Futhark/CodeGen/Backends/MulticoreISPC.hs b/src/Futhark/CodeGen/Backends/MulticoreISPC.hs
--- a/src/Futhark/CodeGen/Backends/MulticoreISPC.hs
+++ b/src/Futhark/CodeGen/Backends/MulticoreISPC.hs
@@ -707,6 +707,9 @@
   GC.stm [C.cstm|if (!$exp:e') { $items:err }|]
   where
     stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs
+compileCode OnFailure {} =
+  -- ISPC does not allow the goto.
+  error "compileCode: OnFailure in ISPC kernel."
 compileCode code =
   GC.compileCode code
 
@@ -790,6 +793,18 @@
                      $id:name.size = 0;
                      $id:name.references = NULL;|]
 
+-- | Does this code contain an 'OnFailure', not counting those in the tasks of
+-- nested 'SegOp's?
+hasOnFailure :: MCCode -> Bool
+hasOnFailure OnFailure {} = True
+hasOnFailure (x :>>: y) = hasOnFailure x || hasOnFailure y
+hasOnFailure (If _ x y) = hasOnFailure x || hasOnFailure y
+hasOnFailure (For _ _ x) = hasOnFailure x
+hasOnFailure (While _ x) = hasOnFailure x
+hasOnFailure (Op (ForEach _ _ _ body)) = hasOnFailure body
+hasOnFailure (Op (ForEachActive _ body)) = hasOnFailure body
+hasOnFailure _ = False
+
 -- | Can the given code produce an error? If so, we can't use foreach
 -- loops, since they don't allow for early-outs in error handling.
 mayProduceError :: MCCode -> Bool
@@ -800,6 +815,7 @@
 mayProduceError (Op (ForEachActive _ body)) = mayProduceError body
 mayProduceError (Op (ForEach _ _ _ body)) = mayProduceError body
 mayProduceError (Op SegOp {}) = True
+mayProduceError (OnFailure x y) = mayProduceError x || mayProduceError y
 mayProduceError Allocate {} = True
 mayProduceError Assert {} = True
 mayProduceError SetMem {} = True
@@ -822,7 +838,7 @@
 
   let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] params mempty seq_code
 
-  fstruct <-
+  (fstruct, read_retvals) <-
     MC.prepareTaskStruct sharedDef "task" free_args free_ctypes retval_args retval_ctypes
 
   fpar_task <- MC.generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval
@@ -874,6 +890,7 @@
                         struct $id:fstruct $tyqual:uniform * $tyqual:uniform args,
                         $tyqual:uniform int iterations);|]
 
+  failure <- GC.failureJump
   aos_name <- newVName "aos"
   GC.items
     [C.citems|
@@ -892,16 +909,23 @@
     $escstm:("#else")
     err = $id:schedn(ctx, &$id:(fstruct <> "_"), $exp:e');
     if (err != 0) {
-      goto cleanup;
+      $stm:failure
     }
     $escstm:("#endif")|]
+  GC.stms read_retvals
+compileOp (ISPCKernel body free)
+  | -- ISPC is too weak for our failure handling machinery.
+    hasOnFailure body =
+      local (\env -> env {GC.envOperations = MC.operations}) $
+        MC.compileOp $
+          ISPCKernel body free
 compileOp (ISPCKernel body free) = do
   free_ctypes <- mapM MC.paramToCType free
   let free_args = map paramName free
 
   let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] free mempty body
   -- Generate ISPC kernel
-  fstruct <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] []
+  (fstruct, _) <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] []
   let fstruct' = fstruct <> "_"
 
   ispcShim <- ispcDef "loop_ispc" $ \s -> do
@@ -911,7 +935,9 @@
           GC.collect $ do
             GC.decl [C.cdecl|$tyqual:uniform struct futhark_context * $tyqual:uniform ctx = $id:fstruct'->ctx;|]
             GC.items =<< compileGetStructVals fstruct free_args free_ctypes
-            body' <- GC.collect $ compileCode body
+            body' <- GC.collect $ do
+              compileCode body
+              GC.items =<< GC.errorLabel
             mapM_ GC.item decl_cached
             mapM_ GC.item =<< GC.declAllocatedMem
 
@@ -962,11 +988,12 @@
         }|]
 
   -- Generate C code to call into ISPC kernel
+  failure <- GC.failureJump
   GC.items
     [C.citems|
     err = $id:ispcShim(start, end, & $id:fstruct');
     if (err != 0) {
-      goto cleanup;
+      $stm:failure
     }|]
 compileOp (ForEach i from bound body) = do
   from' <- compileExp from
@@ -1089,18 +1116,13 @@
     free = freeIn bound
 findDeps (While cond x) = do
   local (<> freeIn cond) $ findDeps x
-findDeps (Op (SegOp _ free _ _ retvals _)) =
-  mapM_
-    ( \x ->
-        addDeps (paramName x) $
-          namesFromList $
-            map paramName free
-    )
-    retvals
 findDeps (Op (ForEach _ _ _ body)) =
   findDeps body
 findDeps (Op (ForEachActive _ body)) =
   findDeps body
+findDeps (OnFailure body handler) = do
+  findDeps body
+  findDeps handler
 findDeps (SetScalar name e) =
   addDeps name $ freeIn e
 findDeps (Call tars _ args) =
@@ -1130,15 +1152,20 @@
     deps' = M.map grow deps
 
 -- | Find roots of variance. These are memory blocks declared in
--- the current scope as well as loop indices of foreach loops.
+-- the current scope, loop indices of foreach loops, and the results of
+-- nested SegOps. A SegOp in a kernel is run once per program instance,
+-- each of which receives its result through a pointer to its own lane
+-- of the variable (see 'MC.compileSetRetvalStructValues').
 findVarying :: MCCode -> [VName]
 findVarying (x :>>: y) = findVarying x ++ findVarying y
 findVarying (If _ x y) = findVarying x ++ findVarying y
 findVarying (For _ _ x) = findVarying x
 findVarying (While _ x) = findVarying x
 findVarying (Op (ForEachActive _ body)) = findVarying body
+findVarying (OnFailure body handler) = findVarying body ++ findVarying handler
 findVarying (Op (ForEach idx _ _ body)) = idx : findVarying body
 findVarying (DeclareMem mem _) = [mem]
+findVarying (Op (SegOp _ _ _ _ retvals _)) = map paramName retvals
 findVarying _ = []
 
 -- | Analyze variability in a body of code and run an action with
@@ -1147,7 +1174,7 @@
 analyzeVariability code m = do
   let roots = findVarying code
   let deps = depsFixedPoint $ execVariabilityM $ findDeps code
-  let safelist = M.filter (\b -> all (`notNameIn` b) roots) deps
+  let safelist = M.filterWithKey (\v b -> v `notElem` roots && all (`notNameIn` b) roots) deps
   let safe = namesFromList $ M.keys safelist
   pre_state <- GC.getUserState
   GC.modifyUserState (\s -> s {sUniform = safe})
diff --git a/src/Futhark/CodeGen/Backends/MulticoreWASM.hs b/src/Futhark/CodeGen/Backends/MulticoreWASM.hs
--- a/src/Futhark/CodeGen/Backends/MulticoreWASM.hs
+++ b/src/Futhark/CodeGen/Backends/MulticoreWASM.hs
@@ -4,22 +4,14 @@
 -- JavaScript wrapper.
 module Futhark.CodeGen.Backends.MulticoreWASM
   ( compileProg,
-    runServer,
-    libraryExports,
-    GC.CParts (..),
-    GC.asLibrary,
-    GC.asExecutable,
-    GC.asServer,
   )
 where
 
-import Data.Maybe
 import Data.Text qualified as T
 import Futhark.CodeGen.Backends.GenericC qualified as GC
 import Futhark.CodeGen.Backends.GenericWASM
 import Futhark.CodeGen.Backends.MulticoreC qualified as MC
 import Futhark.CodeGen.Backends.MulticoreC.Boilerplate (generateBoilerplate)
-import Futhark.CodeGen.ImpCode.Multicore qualified as Imp
 import Futhark.CodeGen.ImpGen.Multicore qualified as ImpGen
 import Futhark.IR.MCMem
 import Futhark.MonadFreshNames
@@ -32,10 +24,11 @@
 -- * Generated C code (to be passed to Emscripten).
 --
 -- * JavaScript wrapper code that presents a nicer interface to the
---   Emscripten-produced code (this should be put in a @.class.js@
---   file by itself).
+--   Emscripten-produced code (passed to @emcc@ with
+--   @--extern-post-js@).
 --
--- * Options that should be passed to @emcc@.
+-- * The names of the C functions that must be exported from the
+--   WebAssembly module.
 compileProg ::
   (MonadFreshNames m) =>
   T.Text ->
@@ -58,20 +51,7 @@
   pure
     ( ws,
       ( prog'',
-        javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')),
+        javascriptWrapper (GC.cManifest prog''),
         "_futhark_context_config_set_num_threads" : emccExportNames (GC.cManifest prog'')
       )
     )
-
-fRepMyRep :: Imp.Definitions Imp.Multicore -> [JSEntryPoint]
-fRepMyRep prog =
-  let Imp.Functions fs = Imp.defFuns prog
-      function fun = do
-        Imp.EntryPoint n res args _ <- Imp.functionEntry fun
-        Just $
-          JSEntryPoint
-            { name = nameToString n,
-              parameters = map (extToString . snd) args,
-              ret = extToString $ snd res
-            }
-   in mapMaybe (function . snd) fs
diff --git a/src/Futhark/CodeGen/Backends/PyOpenCL.hs b/src/Futhark/CodeGen/Backends/PyOpenCL.hs
--- a/src/Futhark/CodeGen/Backends/PyOpenCL.hs
+++ b/src/Futhark/CodeGen/Backends/PyOpenCL.hs
@@ -233,6 +233,11 @@
 callKernel (Imp.GetSizeMax v size_class) = do
   v' <- compileVar v
   stm $ Assign v' $ kernelConstToExp $ Imp.SizeMaxConst size_class
+callKernel (Imp.GetBookkeeping v b) = do
+  v' <- compileVar v
+  stm . Assign v' . Var $ case b of
+    Imp.BookkeepingCounters -> "self.bookkeeping_counters"
+    Imp.BookkeepingLocks -> "self.bookkeeping_locks"
 callKernel (Imp.LaunchKernel safety name shared_memory args num_threadblocks workgroup_size) = do
   num_threadblocks' <- mapM (fmap asLong . compileExp) num_threadblocks
   workgroup_size' <- mapM compileBlockDim workgroup_size
diff --git a/src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs b/src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs
--- a/src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs
+++ b/src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs
@@ -11,13 +11,15 @@
 import Futhark.CodeGen.Backends.GenericPython qualified as Py
 import Futhark.CodeGen.Backends.GenericPython.AST
 import Futhark.CodeGen.ImpCode.OpenCL
-  ( ErrorMsg (..),
+  ( Bookkeeping (..),
+    ErrorMsg (..),
     ErrorMsgPart (..),
     FailureMsg (..),
     KernelConst (..),
     KernelConstExp,
     ParamMap,
     PrimType (..),
+    bookkeepingSize,
     errorMsgArgTypes,
     untyped,
   )
@@ -57,6 +59,8 @@
   [text|
 size_heuristics=$size_heuristics
 self.global_failure_args_max = $max_num_args
+self.bookkeeping_counters_size = $num_counters
+self.bookkeeping_locks_size = $num_locks
 self.failure_msgs=$failure_msgs
 constants = $constants'
 program = initialise_opencl_object(self,
@@ -84,6 +88,8 @@
     types' = prettyText $ map (show . prettyString) types -- Looks enough like Python.
     sizes' = prettyText $ Py.sizeClassesToPython sizes
     max_num_args = prettyText $ foldl max 0 $ map (errorMsgNumArgs . failureError) failures
+    num_counters = prettyText $ bookkeepingSize BookkeepingCounters
+    num_locks = prettyText $ bookkeepingSize BookkeepingLocks
     failure_msgs = prettyText $ List $ map formatFailure failures
     onConstant (name, e) =
       Tuple
diff --git a/src/Futhark/CodeGen/Backends/SequentialWASM.hs b/src/Futhark/CodeGen/Backends/SequentialWASM.hs
--- a/src/Futhark/CodeGen/Backends/SequentialWASM.hs
+++ b/src/Futhark/CodeGen/Backends/SequentialWASM.hs
@@ -4,16 +4,9 @@
 -- JavaScript wrapper.
 module Futhark.CodeGen.Backends.SequentialWASM
   ( compileProg,
-    runServer,
-    libraryExports,
-    GC.CParts (..),
-    GC.asLibrary,
-    GC.asExecutable,
-    GC.asServer,
   )
 where
 
-import Data.Maybe
 import Data.Text qualified as T
 import Futhark.CodeGen.Backends.GenericC qualified as GC
 import Futhark.CodeGen.Backends.GenericWASM
@@ -31,10 +24,11 @@
 -- * Generated C code (to be passed to Emscripten).
 --
 -- * JavaScript wrapper code that presents a nicer interface to the
---   Emscripten-produced code (this should be put in a @.class.js@
---   file by itself).
+--   Emscripten-produced code (passed to @emcc@ with
+--   @--extern-post-js@).
 --
--- * Options that should be passed to @emcc@.
+-- * The names of the C functions that must be exported from the
+--   WebAssembly module.
 compileProg :: (MonadFreshNames m) => T.Text -> Prog SeqMem -> m (ImpGen.Warnings, (GC.CParts, T.Text, [String]))
 compileProg version prog = do
   (ws, prog') <- ImpGen.compileProg prog
@@ -49,23 +43,10 @@
       (DefaultSpace, [DefaultSpace])
       []
       prog'
-  pure (ws, (prog'', javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')), emccExportNames (GC.cManifest prog'')))
+  pure (ws, (prog'', javascriptWrapper (GC.cManifest prog''), emccExportNames (GC.cManifest prog'')))
   where
     operations :: GC.Operations Imp.Sequential ()
     operations =
       GC.defaultOperations
         { GC.opsCompiler = const $ pure ()
         }
-
-fRepMyRep :: Imp.Program -> [JSEntryPoint]
-fRepMyRep prog =
-  let Imp.Functions fs = Imp.defFuns prog
-      function (Imp.Function entry _ _ _ _) = do
-        Imp.EntryPoint n res args _doc <- entry
-        Just $
-          JSEntryPoint
-            { name = nameToString n,
-              parameters = map (extToString . snd) args,
-              ret = extToString $ snd res
-            }
-   in mapMaybe (function . snd) fs
diff --git a/src/Futhark/CodeGen/ImpCode.hs b/src/Futhark/CodeGen/ImpCode.hs
--- a/src/Futhark/CodeGen/ImpCode.hs
+++ b/src/Futhark/CodeGen/ImpCode.hs
@@ -146,7 +146,7 @@
 data Param
   = MemParam VName Space
   | ScalarParam VName PrimType
-  deriving (Eq, Show)
+  deriving (Eq, Ord, Show)
 
 -- | The name of a parameter.
 paramName :: Param -> VName
@@ -337,6 +337,12 @@
     -- out not to be true, then report a failure along with
     -- the given error message.
     Assert Exp (ErrorMsg Exp) (Loc, [Loc])
+  | -- | @OnFailure body handler@ executes @body@, and if it fails, executes
+    -- @handler@ before the failure propagates as usual. This is for code that
+    -- must still be executed after a failure, such as releasing a lock. The
+    -- handler must not fail. In a GPU kernel, only the failing thread executes
+    -- the handler, so @body@ must not contain barriers.
+    OnFailure (Code a) (Code a)
   | -- | Print the given value to the screen, somehow
     -- annotated with the given string as a description.  If
     -- no type/value pair, just print the string.  This has
@@ -390,6 +396,7 @@
     go f (If _ x y) = f x <> f y
     go f (For _ _ x) = f x
     go f (While _ x) = f x
+    go f (OnFailure x y) = f x <> f y
     go _ _ = mempty
 
     declared (DeclareMem mem space) =
@@ -415,6 +422,7 @@
     listify (If cond c1 c2) = [If cond (declsFirst c1) (declsFirst c2)]
     listify (For i e c) = [For i e (declsFirst c)]
     listify (While cond c) = [While cond (declsFirst c)]
+    listify (OnFailure c1 c2) = [OnFailure (declsFirst c1) (declsFirst c2)]
     listify c = [c]
     isDecl (DeclareScalar {}) = True
     isDecl (DeclareMem {}) = True
@@ -429,6 +437,7 @@
 calledFuncs f (If _ x y) = calledFuncs f x <> calledFuncs f y
 calledFuncs f (For _ _ x) = calledFuncs f x
 calledFuncs f (While _ x) = calledFuncs f x
+calledFuncs f (OnFailure x y) = calledFuncs f x <> calledFuncs f y
 calledFuncs _ _ = mempty
 
 -- | Compute call graph, as per 'calledFuncs', but also include
@@ -449,6 +458,7 @@
 foldProvenances f (For _ _ c) = foldProvenances f c
 foldProvenances f (While _ c) = foldProvenances f c
 foldProvenances f (If _ x y) = foldProvenances f x <> foldProvenances f y
+foldProvenances f (OnFailure x y) = foldProvenances f x <> foldProvenances f y
 foldProvenances f (Op x) = f x
 foldProvenances _ _ = mempty
 
@@ -643,6 +653,13 @@
     pretty dest <+> "<-" <+> pretty from <+> "@" <> pretty space
   pretty (Assert e msg _) =
     "assert" <> parens (commasep [pretty msg, pretty e])
+  pretty (OnFailure body handler) =
+    "on_failure"
+      <+> "{"
+      </> indent 2 (pretty body)
+      </> "} handler {"
+      </> indent 2 (pretty handler)
+      </> "}"
   pretty (Copy t shape (dst, dstspace) (dstoffset, dststrides) (src, srcspace) (srcoffset, srcstrides)) =
     ("lmadcopy_" <> pretty (length shape) <> "d_" <> pretty t)
       <> (parens . align)
@@ -732,6 +749,8 @@
     While cond <$> traverse f code
   traverse f (If cond x y) =
     If cond <$> traverse f x <*> traverse f y
+  traverse f (OnFailure x y) =
+    OnFailure <$> traverse f x <*> traverse f y
   traverse f (Op kernel) =
     Op <$> f kernel
   traverse _ Skip =
@@ -779,6 +798,7 @@
 declaredIn (x :>>: y) = declaredIn x <> declaredIn y
 declaredIn (For i _ body) = oneName i <> declaredIn body
 declaredIn (While _ body) = declaredIn body
+declaredIn (OnFailure body handler) = declaredIn body <> declaredIn handler
 declaredIn _ = mempty
 
 instance FreeIn EntryPoint where
@@ -837,6 +857,8 @@
     freeIn' cond <> freeIn' t <> freeIn' f
   freeIn' (Assert e msg _) =
     freeIn' e <> foldMap freeIn' msg
+  freeIn' (OnFailure body handler) =
+    freeIn' body <> freeIn' handler
   freeIn' (Op op) =
     freeIn' op
   freeIn' (Meta {}) =
diff --git a/src/Futhark/CodeGen/ImpCode/GPU.hs b/src/Futhark/CodeGen/ImpCode/GPU.hs
--- a/src/Futhark/CodeGen/ImpCode/GPU.hs
+++ b/src/Futhark/CodeGen/ImpCode/GPU.hs
@@ -7,6 +7,8 @@
     KernelConst (..),
     KernelConstExp,
     HostOp (..),
+    Bookkeeping (..),
+    bookkeepingSize,
     KernelOp (..),
     Fence (..),
     AtomicOp (..),
@@ -48,8 +50,29 @@
   | GetSize VName Name SizeClass
   | CmpSizeLe VName Name SizeClass Exp
   | GetSizeMax VName SizeClass
+  | -- | Bind the given device memory to one of the bookkeeping buffers of the
+    -- context.
+    GetBookkeeping VName Bookkeeping
   deriving (Show)
 
+-- | The bookkeeping buffers are device buffers of 32-bit integers that belong
+-- to the context and are shared by all kernels. They are zero whenever no
+-- kernel is running, and a kernel that uses them must leave them zeroed when it
+-- succeeds, and the context zeroes them again after a failure. As kernels run
+-- serially, they never compete for them. If we ever add support for multiple
+-- streams, we must have a set of buffers per stream.
+data Bookkeeping
+  = -- | Counters, e.g. for detecting the last block to finish.
+    BookkeepingCounters
+  | -- | Locks for operations that cannot be done atomically.
+    BookkeepingLocks
+  deriving (Eq, Ord, Show)
+
+-- | The number of 32-bit integers in a bookkeeping buffer.
+bookkeepingSize :: Bookkeeping -> Integer
+bookkeepingSize BookkeepingCounters = 1024
+bookkeepingSize BookkeepingLocks = 100151
+
 -- | The size of one dimension of a block.
 type BlockDim = Either Exp KernelConstExp
 
@@ -114,6 +137,10 @@
       <> parens (commasep [pretty key, pretty size_class])
   pretty (GetSizeMax dest size_class) =
     pretty dest <+> "<-" <+> "get_size_max" <> parens (pretty size_class)
+  pretty (GetBookkeeping dest BookkeepingCounters) =
+    pretty dest <+> "<-" <+> "bookkeeping_counters()"
+  pretty (GetBookkeeping dest BookkeepingLocks) =
+    pretty dest <+> "<-" <+> "bookkeeping_locks()"
   pretty (CmpSizeLe dest name size_class x) =
     pretty dest
       <+> "<-"
@@ -133,6 +160,8 @@
     freeIn' dest
   freeIn' (GetSize dest _ _) =
     freeIn' dest
+  freeIn' (GetBookkeeping dest _) =
+    freeIn' dest
 
 instance FreeIn Kernel where
   freeIn' kernel =
@@ -181,13 +210,14 @@
     -- warp. Only supported for CUDA and HIP.
     WarpSync
   | SharedAlloc VName (Count Bytes (TExp Int64))
-  | -- | Perform a barrier and also check whether any
-    -- threads have failed an assertion.  Make sure all
-    -- threads would reach all 'ErrorSync's if any of them
-    -- do.  A failing assertion will jump to the next
-    -- following 'ErrorSync', so make sure it's not inside
-    -- control flow or similar.
-    ErrorSync Fence
+  | -- | Perform a barrier and also check whether any threads have failed an
+    -- assertion. If so, all threads of the block execute the given code and
+    -- then stop. The code must not fail, and is usually empty. Make sure all
+    -- threads would reach all 'ErrorSync's if any of them do. A failing
+    -- assertion will jump to the next following 'ErrorSync' (unless it is in
+    -- the body of an 'OnFailure'), so make sure it's not inside control flow or
+    -- similar.
+    ErrorSync Fence (Code KernelOp)
   deriving (Show)
 
 -- | Atomic operations return the value stored before the update. This
@@ -257,10 +287,14 @@
     "warp_sync()"
   pretty (SharedAlloc name size) =
     pretty name <+> equals <+> "shared_alloc" <> parens (pretty size)
-  pretty (ErrorSync FenceLocal) =
-    "error_sync_local()"
-  pretty (ErrorSync FenceGlobal) =
-    "error_sync_global()"
+  pretty (ErrorSync f handler) =
+    sync f <> handler'
+    where
+      sync FenceLocal = "error_sync_local()"
+      sync FenceGlobal = "error_sync_global()"
+      handler' = case handler of
+        Skip -> mempty
+        _ -> " on_failure" <> brace (pretty handler)
   pretty (Atomic _ (AtomicAdd t old arr ind x)) =
     pretty old
       <+> "<-"
@@ -335,6 +369,7 @@
 instance FreeIn KernelOp where
   freeIn' (Atomic _ op) = freeIn' op
   freeIn' (SharedAlloc _ size) = freeIn' size
+  freeIn' (ErrorSync _ handler) = freeIn' handler
   freeIn' _ = mempty
 
 brace :: Doc a -> Doc a
diff --git a/src/Futhark/CodeGen/ImpCode/Kernels.hs b/src/Futhark/CodeGen/ImpCode/Kernels.hs
--- a/src/Futhark/CodeGen/ImpCode/Kernels.hs
+++ b/src/Futhark/CodeGen/ImpCode/Kernels.hs
@@ -5,6 +5,8 @@
     KernelArg (..),
     HostCode,
     HostOp (..),
+    Bookkeeping (..),
+    bookkeepingSize,
     KernelSafety (..),
     numFailureParams,
     FailureMsg (..),
@@ -17,7 +19,13 @@
 where
 
 import Futhark.CodeGen.ImpCode
-import Futhark.CodeGen.ImpCode.GPU (BlockDim, KernelConst (..), KernelConstExp)
+import Futhark.CodeGen.ImpCode.GPU
+  ( BlockDim,
+    Bookkeeping (..),
+    KernelConst (..),
+    KernelConstExp,
+    bookkeepingSize,
+  )
 import Futhark.IR.GPU.Sizes
 import Futhark.Util.Pretty
 
@@ -73,6 +81,7 @@
   | GetSize VName Name
   | CmpSizeLe VName Name Exp
   | GetSizeMax VName SizeClass
+  | GetBookkeeping VName Bookkeeping
   deriving (Show)
 
 instance Pretty HostOp where
diff --git a/src/Futhark/CodeGen/ImpCode/Multicore.hs b/src/Futhark/CodeGen/ImpCode/Multicore.hs
--- a/src/Futhark/CodeGen/ImpCode/Multicore.hs
+++ b/src/Futhark/CodeGen/ImpCode/Multicore.hs
@@ -206,6 +206,7 @@
     go f (If _ x y) = f x <> f y
     go f (For _ _ x) = f x
     go f (While _ x) = f x
+    go f (OnFailure x y) = f x <> f y
     go f (Op op) = goOp f op
     go _ _ = mempty
 
diff --git a/src/Futhark/CodeGen/ImpGen.hs b/src/Futhark/CodeGen/ImpGen.hs
--- a/src/Futhark/CodeGen/ImpGen.hs
+++ b/src/Futhark/CodeGen/ImpGen.hs
@@ -50,6 +50,7 @@
     lookupAcc,
     askAttrs,
     askProvenance,
+    bodyMayFail,
 
     -- * Building Blocks
     TV,
@@ -126,6 +127,7 @@
 import Control.Monad
 import Control.Monad.Reader
 import Control.Monad.State
+import Control.Monad.Writer (execWriter, tell)
 import Control.Parallel.Strategies
 import Data.Bifunctor (first)
 import Data.DList qualified as DL
@@ -134,6 +136,7 @@
 import Data.List.NonEmpty (NonEmpty (..))
 import Data.Map.Strict qualified as M
 import Data.Maybe
+import Data.Monoid (Any (..))
 import Data.Set qualified as S
 import Data.String
 import Data.Text qualified as T
@@ -1354,6 +1357,17 @@
 -- | The provenance of whatever we are currently generating code for.
 askProvenance :: ImpM rep r op Provenance
 askProvenance = asks envProvenance
+
+-- | Can executing this body fail? This is the case if it contains an
+-- assertion, or calls a non-builtin function, which we conservatively assume
+-- may fail. Assertions inside 'Op's are not considered.
+bodyMayFail :: (ASTRep rep) => Body rep -> Bool
+bodyMayFail = any (expMayFail . stmExp) . bodyStms
+  where
+    expMayFail (BasicOp Assert {}) = True
+    expMayFail (Apply fname _ _ _) = not $ isBuiltInFunction fname
+    expMayFail e = getAny $ execWriter $ walkExpM walker e
+    walker = identityWalker {walkOnBody = \_ -> tell . Any . bodyMayFail}
 
 -- | Wrap any code emitted in the enclosed section with the current provenance,
 -- if any.
diff --git a/src/Futhark/CodeGen/ImpGen/GPU.hs b/src/Futhark/CodeGen/ImpGen/GPU.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU.hs
@@ -239,9 +239,8 @@
     locksForInputs atomics ((c, (_, _, op)) : inputs')
       | Just (op_lam, _) <- op,
         AtomicLocking _ <- atomicUpdateLocking atomics op_lam = do
-          let num_locks = 100151
-          locks_arr <- genZeroes "withacc_locks" num_locks
-          let locks = Locks locks_arr num_locks
+          locks_arr <- bookkeepingArray Imp.BookkeepingLocks
+          let locks = Locks locks_arr $ fromInteger $ Imp.bookkeepingSize Imp.BookkeepingLocks
               extend env = env {hostLocks = M.insert c locks $ hostLocks env}
           localEnv extend $ locksForInputs atomics inputs'
       | otherwise =
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/Base.hs b/src/Futhark/CodeGen/ImpGen/GPU/Base.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/Base.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/Base.hs
@@ -18,6 +18,7 @@
     warpLocal,
     blockScan,
     blockLoop,
+    peeledKernelLoop,
     isActive,
     sKernel,
     sKernelThread,
@@ -31,7 +32,7 @@
     blockCoverSpace,
     fenceForArrays,
     updateAcc,
-    genZeroes,
+    bookkeepingArray,
     isPrimParam,
     kernelConstToExp,
     getSize,
@@ -107,9 +108,10 @@
     -- | A mapping from dimensions of nested SegOps to already
     -- computed local thread IDs.  Only valid in non-virtualised case.
     kernelLocalIdMap :: M.Map [SubExp] [Imp.TExp Int32],
-    -- | Mapping from dimensions of nested SegOps to how many
-    -- iterations the virtualisation loop needs.
-    kernelChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)
+    -- | Mapping from dimensions of nested SegOps to how many full block-sized
+    -- chunks the virtualisation loop has; see 'peeledKernelLoop'. Constant dimensions
+    -- are not included.
+    kernelFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32)
   }
 
 kernelGlobalThreadId, kernelLocalThreadId, kernelBlockId :: KernelConstants -> Imp.TExp Int32
@@ -180,15 +182,16 @@
             Nothing ->
               error $ "Missing locks for " ++ prettyString acc
 
--- | Generate a constant device array of 32-bit integer zeroes with
--- the given number of elements.  Initialised with a replicate.
-genZeroes :: Name -> Int -> CallKernelGen VName
-genZeroes desc n = genConstants $ do
-  counters_mem <- sAlloc (desc <> "_mem") (4 * fromIntegral n) (Space "device")
-  let shape = Shape [intConst Int64 (fromIntegral n)]
-  counters <- sArrayInMem desc int32 shape counters_mem
-  sReplicate counters $ intConst Int32 0
-  pure (namesFromList [counters_mem], counters)
+-- | Retrieve a bookkeeping array.
+bookkeepingArray :: Imp.Bookkeeping -> CallKernelGen VName
+bookkeepingArray b = do
+  mem <- sDeclareMem (desc <> "_mem") (Space "device")
+  sOp $ Imp.GetBookkeeping mem b
+  sArrayInMem desc int32 (Shape [intConst Int64 $ Imp.bookkeepingSize b]) mem
+  where
+    desc = case b of
+      Imp.BookkeepingCounters -> "counters"
+      Imp.BookkeepingLocks -> "locks"
 
 compileThreadExp :: ExpCompiler GPUMem KernelEnv Imp.KernelOp
 compileThreadExp (Pat [pe]) (BasicOp (Opaque _ se)) =
@@ -226,6 +229,28 @@
           i <- dPrimVE "i" $ chunk_i * num_threads + tid
           sWhen (i .<. n) $ f i
 
+-- | @peeledKernelLoop tid num_threads num_full_chunks n f@ invokes @f@ on the
+-- iterations @tid@, @tid+num_threads@, and so on, below @n@, where
+-- @num_full_chunks@ must be @n `quot` num_threads@. Only the final partial
+-- chunk is bounds checked, so the body is generated twice; see Note [Peeled
+-- block loops]. The body must contain thread-level code.
+peeledKernelLoop ::
+  (IntExp t) =>
+  Imp.TExp t ->
+  Imp.TExp t ->
+  Imp.TExp t ->
+  Imp.TExp t ->
+  (Imp.TExp t -> InKernelGen ()) ->
+  InKernelGen ()
+peeledKernelLoop tid num_threads num_full_chunks n f =
+  localOps threadOperations $ do
+    sFor "chunk_i" num_full_chunks $ \chunk_i ->
+      f =<< dPrimVE "i" (sMinAs (chunk_i * num_threads + tid) (n - 1))
+    i <- dPrimVE "i" $ num_full_chunks * num_threads + tid
+    sWhen (i .<. n) $ f i
+  where
+    sMinAs x y = TPrimExp $ BinOpExp (SMin $ expIntType x) (untyped x) (untyped y)
+
 -- | Assign iterations of a for-loop to threads in the threadblock.  The
 -- passed-in function is invoked with the (symbolic) iteration.  For
 -- multidimensional loops, use 'blockCoverSpace'.
@@ -236,11 +261,13 @@
   InKernelGen ()
 blockLoop n f = do
   constants <- kernelConstants <$> askEnv
-  kernelLoop
-    (kernelLocalThreadId constants `sExtAs` n)
-    (kernelBlockSize constants `sExtAs` n)
-    n
-    f
+  let tid = kernelLocalThreadId constants `sExtAs` n
+      num_threads = kernelBlockSize constants `sExtAs` n
+  if n == num_threads
+    then localOps threadOperations $ f tid
+    else do
+      num_full_chunks <- dPrimVE "num_full_chunks" $ n `quot` num_threads
+      peeledKernelLoop tid num_threads num_full_chunks n f
 
 -- | Iterate collectively though a multidimensional space, such that
 -- all threads in the block participate.  The passed-in function is
@@ -322,18 +349,17 @@
   Imp.TExp Int32 ->
   Imp.TExp Bool ->
   [VName] ->
-  InKernelGen () ->
+  Imp.Fence ->
   Lambda GPUMem ->
   InKernelGen ()
-inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs barrier scan_lam = do
+inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs fence scan_lam = do
   skip_threads <- dPrim "skip_threads"
   let actual_params = lambdaParams scan_lam
       (x_params, y_params) =
         splitAt (length actual_params `div` 2) actual_params
       y_to_x =
         forM_ (zip x_params y_params) $ \(x, y) ->
-          when (isPrimParam x) $
-            copyDWIM (paramName x) [] (Var (paramName y)) []
+          copyDWIM (paramName x) [] (Var (paramName y)) []
 
   -- Set initial y values
   sComment "read input for in-block scan" $
@@ -366,25 +392,32 @@
               $ compileBody' x_params
               $ lambdaBody scan_lam
 
-      maybeBarrier =
-        sWhen
-          (lockstep_width .<=. tvExp skip_threads)
-          barrier
+      -- See Note [Failing block-level operators].
+      maybeBarrier
+        | may_fail = sOp $ Imp.ErrorSync fence mempty
+        | otherwise =
+            sWhen
+              (lockstep_width .<=. tvExp skip_threads)
+              barrier
 
+      warpLocal'
+        | may_fail = id
+        | otherwise = warpLocal
+
   sComment "in-block scan (hopefully no barriers needed)" $ do
     skip_threads <-- 1
     sWhile (tvExp skip_threads .<. block_size) $ do
       thread_active <-
         dPrimVE "thread_active" $ tvExp skip_threads .<=. in_block_id .&&. active
 
-      warpLocal $ do
+      warpLocal' $ do
         sWhen thread_active . sComment "read operands" $
           zipWithM_ (readParam (sExt64 $ tvExp skip_threads)) x_params arrs
         sComment "perform operation" $ op_to_x thread_active
 
       maybeBarrier
 
-      warpLocal . sWhen thread_active . sComment "write result" $
+      warpLocal' . sWhen thread_active . sComment "write result" $
         sequence_ $
           zipWith3 writeResult x_params y_params arrs
 
@@ -397,9 +430,15 @@
     ltid32 = kernelLocalThreadId constants
     ltid = sExt64 ltid32
     array_scan = not $ all primType $ lambdaReturnType scan_lam
+    barrier = sOp $ Imp.Barrier fence
+    may_fail = bodyMayFail $ lambdaBody scan_lam
 
-    readInitial p arr =
+    readInitial p arr = do
       copyDWIMFix (paramName p) [] (Var arr) [ltid]
+      -- An array-typed element is also read @arrs_full_size@ elements along,
+      -- by 'readParam', so put it there too.
+      unless (isPrimParam p) $
+        copyDWIMFix arr [ltid + arrs_full_size] (Var arr) [ltid]
 
     readParam behind p arr
       | isPrimParam p =
@@ -408,10 +447,38 @@
           copyDWIMFix (paramName p) [] (Var arr) [ltid - behind + arrs_full_size]
 
     writeResult x y arr = do
-      when (isPrimParam x) $
-        copyDWIMFix arr [ltid] (Var $ paramName x) []
+      copyDWIMFix arr [ltid] (Var $ paramName x) []
+      unless (isPrimParam x) $
+        copyDWIMFix arr [ltid + arrs_full_size] (Var $ paramName x) []
       copyDWIM (paramName y) [] (Var $ paramName x) []
 
+-- | 'blockScan' splits the block into chunks of this size, which are scanned
+-- separately. The only requirement is that it should not be less than the
+-- square root of the block size. With 32, we will work on blocks of size 1024
+-- or smaller, which fits every device Troels has seen. Still, it would be nicer
+-- if it were a runtime parameter. Some day.
+blockScanChunkSize :: (Num a) => a
+blockScanChunkSize = 32
+
+-- | Scan the @w@ first elements of the given arrays, in place. @w@ must be at
+-- most the thread block size; you'll need to wrap this function in a loop if
+-- you want to scan larger arrays.
+--
+-- An array-typed element is kept twice: at index @i@, where the result is
+-- expected, and again at index @i + arrs_full_size@. It is the latter copy that
+-- a thread reads as its x operand, because the operator writes its result
+-- through its x parameter, which may well be the former. With @k@ chunks, the
+-- first @k@ elements are overwritten while computing the per-chunk carry-ins,
+-- so they are saved at indexes @arrs_full_size + k@ to @arrs_full_size + 2k@,
+-- whose copies are not needed by then. Only the @w@ first elements need be
+-- within the /shape/ of the arrays, but the indexes from @arrs_full_size@ to
+-- @arrs_full_size + w@ must also be backed by memory; see
+-- 'Futhark.CodeGen.ImpGen.GPU.Block.blockScanElems'.
+--
+-- The callers in "Futhark.CodeGen.ImpGen.GPU.SegScan.TwoPass" and
+-- "Futhark.CodeGen.ImpGen.GPU.SegRed" pass arrays whose elements /are/ the x
+-- and y parameters of the operator, so for them the copies that keep the two in
+-- step are redundant, but harmless.
 blockScan ::
   Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->
   Imp.TExp Int64 ->
@@ -419,7 +486,7 @@
   Lambda GPUMem ->
   [VName] ->
   InKernelGen ()
-blockScan seg_flag arrs_full_size w lam arrs = do
+blockScan seg_flag arrs_full_size w lam arrs = localOps threadOperations $ do
   constants <- kernelConstants <$> askEnv
   renamed_lam <- renameLambda lam
 
@@ -437,12 +504,7 @@
   -- scanned separately. Typically, these chunks are at most the
   -- lockstep width, which enables barrier-free execution inside them.
   --
-  -- We hardcode the chunk size here. The only requirement is that it
-  -- should not be less than the square root of the block size. With
-  -- 32, we will work on blocks of size 1024 or smaller, which fits
-  -- every device Troels has seen. Still, it would be nicer if it were
-  -- a runtime parameter. Some day.
-  let chunk_size = 32
+  let chunk_size = blockScanChunkSize
       simd_width = kernelWaveSize constants
       chunk_id = ltid32 `quot` chunk_size
       in_chunk_id = ltid32 - chunk_id * chunk_size
@@ -455,19 +517,18 @@
           chunk_size
           active
           arrs
-          barrier
+          barrier_fence
       array_scan = not $ all primType $ lambdaReturnType lam
-      barrier
-        | array_scan =
-            sOp $ Imp.Barrier Imp.FenceGlobal
-        | otherwise =
-            sOp $ Imp.Barrier fence
+      barrier_fence
+        | array_scan = Imp.FenceGlobal
+        | otherwise = fence
+      barrier = sOp $ Imp.Barrier barrier_fence
 
       errorsync
         | array_scan =
-            sOp $ Imp.ErrorSync Imp.FenceGlobal
+            sOp $ Imp.ErrorSync Imp.FenceGlobal mempty
         | otherwise =
-            sOp $ Imp.ErrorSync Imp.FenceLocal
+            sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
       writeBlockResult p arr =
         copyDWIMFix arr [sExt64 chunk_id] (Var $ paramName p) []
@@ -478,19 +539,24 @@
   doInChunkScan seg_flag ltid_in_bounds lam
   barrier
 
+  -- With more than one chunk, the first 'num_chunks' elements are overwritten
+  -- by the chunk results, which are then scanned to produce the carry-ins.
+  num_chunks <- dPrimVE "num_chunks" $ sExt32 $ w `ceilDiv` blockScanChunkSize
   let is_first_block = chunk_id .==. 0
+      several_chunks = num_chunks .>. 1
+      scans_carries = several_chunks .&&. ltid32 .<. num_chunks
   when array_scan $ do
     sComment "save correct values for first block" $
-      sWhen is_first_block $
+      sWhen scans_carries $
         forM_ (zip x_params arrs) $ \(x, arr) ->
           unless (isPrimParam x) $
-            copyDWIMFix arr [arrs_full_size + sExt64 chunk_size + ltid] (Var $ paramName x) []
+            copyDWIMFix arr [arrs_full_size + sExt64 num_chunks + ltid] (Var $ paramName x) []
 
     barrier
 
   let last_in_block = in_chunk_id .==. chunk_size - 1
   sComment "last thread of block 'i' writes its result to offset 'i'" $
-    sWhen (last_in_block .&&. ltid_in_bounds) $
+    sWhen (several_chunks .&&. last_in_block .&&. ltid_in_bounds) $
       everythingVolatile $
         zipWithM_ writeBlockResult x_params arrs
 
@@ -502,20 +568,20 @@
           flag_true (from * chunk_size + chunk_size - 1) (to * chunk_size + chunk_size - 1)
   sComment
     "scan the first block, after which offset 'i' contains carry-in for block 'i+1'"
-    $ doInChunkScan first_block_seg_flag (is_first_block .&&. ltid_in_bounds) renamed_lam
+    $ doInChunkScan first_block_seg_flag scans_carries renamed_lam
 
   errorsync
 
   when array_scan $ do
     sComment "move correct values for first block back a block" $
-      sWhen is_first_block $
+      sWhen scans_carries $
         forM_ (zip x_params arrs) $ \(x, arr) ->
           unless (isPrimParam x) $
             copyDWIMFix
               arr
               [arrs_full_size + ltid]
               (Var arr)
-              [arrs_full_size + sExt64 chunk_size + ltid]
+              [arrs_full_size + sExt64 num_chunks + ltid]
 
     barrier
 
@@ -541,8 +607,7 @@
 
       write_final_result =
         forM_ (zip x_params arrs) $ \(p, arr) ->
-          when (isPrimParam p) $
-            copyDWIMFix arr [ltid] (Var $ paramName p) []
+          copyDWIMFix arr [ltid] (Var $ paramName p) []
 
   sComment "carry-in for every block except the first" $
     localOps threadOperations $ do
@@ -550,17 +615,20 @@
       sComment "perform operation" op_to_x
       sComment "write final result" $ sUnless no_carry_in write_final_result
 
-  barrier
+  -- See Note [Failing block-level operators].
+  sOp $ Imp.ErrorSync barrier_fence mempty
 
   sComment "restore correct values for first block" $
-    sWhen (is_first_block .&&. ltid_in_bounds) $
-      forM_ (zip3 x_params y_params arrs) $ \(x, y, arr) ->
+    sWhen scans_carries $
+      forM_ (zip3 x_params y_params arrs) $ \(_x, y, arr) ->
         if isPrimParam y
           then copyDWIMFix arr [ltid] (Var $ paramName y) []
-          else copyDWIMFix (paramName x) [] (Var arr) [arrs_full_size + ltid]
+          else copyDWIMFix arr [ltid] (Var arr) [arrs_full_size + ltid]
 
   barrier
 
+-- | Reduce the @w@ first elements of the given arrays, leaving the result in
+-- element zero of each. The others are clobbered.
 blockReduce ::
   Imp.TExp Int32 ->
   Lambda GPUMem ->
@@ -580,26 +648,18 @@
   constants <- kernelConstants <$> askEnv
 
   let local_tid = kernelLocalThreadId constants
-
-      barrier
-        | all primType $ lambdaReturnType lam = sOp $ Imp.Barrier Imp.FenceLocal
-        | otherwise = sOp $ Imp.Barrier Imp.FenceGlobal
+      may_fail = bodyMayFail $ lambdaBody lam
 
       errorsync
-        | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal
-        | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal
+        | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+        | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal mempty
 
       readReduceArgument param arr = do
         let i = local_tid + tvExp offset
         copyDWIMFix (paramName param) [] (Var arr) [sExt64 i]
 
       writeReduceOpResult param arr =
-        when (isPrimParam param) $
-          copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []
-
-      writeArrayOpResult param arr =
-        unless (isPrimParam param) $
-          copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []
+        copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []
 
   let (reduce_acc_params, reduce_arr_params) =
         splitAt (length arrs) $ lambdaParams lam
@@ -622,7 +682,10 @@
         sComment "write result of operation" $
           zipWithM_ writeReduceOpResult reduce_acc_params arrs
 
-      wave_size = kernelWaveSize constants
+      -- See Note [Failing block-level operators].
+      wave_size
+        | may_fail = 1
+        | otherwise = kernelWaveSize constants
       tblock_size = kernelBlockSize constants
       wave_id = local_tid `quot` wave_size
       in_wave_id = local_tid - wave_id * wave_size
@@ -652,20 +715,14 @@
         arg_in_bounds .&&. is_first_thread_in_wave .&&. wave_not_skipped
       cross_wave_reductions =
         sWhile doing_cross_wave_reductions $ do
-          barrier
           offset <-- tvExp skip_waves * wave_size
           sWhen apply_in_cross_wave_iteration do_reduce
+          errorsync
           skip_waves <-- tvExp skip_waves * 2
 
-  in_wave_reductions
-  cross_wave_reductions
+  unless may_fail in_wave_reductions
   errorsync
-
-  unless (all isPrimParam reduce_acc_params) $
-    sComment "Copy array-typed operands to result array" $
-      sWhen (local_tid .==. 0) $
-        localOps threadOperations $
-          zipWithM_ writeArrayOpResult reduce_acc_params arrs
+  cross_wave_reductions
 
 compileThreadOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp
 compileThreadOp pat (Alloc size space) =
@@ -864,8 +921,11 @@
       acquire_fence
       dLParams acc_params
       bind_acc_params
-      op_body
-      do_hist
+      -- A thread whose operator fails must still release the lock, or other
+      -- threads would wait for it forever.
+      update <- collect $ op_body >> do_hist
+      on_failure <- collect release_lock
+      emit $ Imp.OnFailure update on_failure
       release_lock
       break_loop
   where
@@ -1005,7 +1065,7 @@
             kernelNumThreads = sExt32 (tblock_size' * num_tblocks'),
             kernelWaveSize = Imp.le32 wave_size,
             kernelLocalIdMap = mempty,
-            kernelChunkItersMap = mempty
+            kernelFullChunksMap = mempty
           }
 
   let set_constants = do
@@ -1095,7 +1155,7 @@
             kernelNumThreads = sExt32 (tblock_size' * num_tblocks'),
             kernelWaveSize = 0,
             kernelLocalIdMap = mempty,
-            kernelChunkItersMap = mempty
+            kernelFullChunksMap = mempty
           }
 
       wrapKernel m = do
@@ -1142,7 +1202,7 @@
         (tvExp phys_tblock_id + i * sExt32 (kernelNumBlocks constants))
     -- Make sure the virtual block is actually done before we let
     -- another virtual block have its way with it.
-    sOp $ Imp.ErrorSync Imp.FenceGlobal
+    sOp $ Imp.ErrorSync Imp.FenceGlobal mempty
 virtualiseBlocks _ _ m =
   m . tvExp . kernelBlockIdVar . kernelConstants =<< askEnv
 
@@ -1477,3 +1537,41 @@
   copyDWIMFix (patElemName pe) is what []
 compileThreadResult _ _ TileReturns {} =
   compilerBugS "compileThreadResult: TileReturns unhandled."
+
+-- Note [Peeled block loops]
+--
+-- When the threads of a block cover an iteration space larger than the block
+-- (virtualised SegOps), each thread handles one iteration per block-sized
+-- chunk, and only the final chunk can be partial. A bounds check in every chunk
+-- keeps the kernel compiler from overlapping the memory accesses of consecutive
+-- chunks, which makes memory-bound loops considerably slower.
+-- 'peeledKernelLoop' therefore runs the full chunks without a check, and
+-- generates the body again for the final chunk, with a bounds check.
+--
+-- A bounds check also tells the kernel compiler that the iteration is in
+-- bounds, which lets it remove checks in the body that follow from it, such as
+-- those of accumulator updates, which likewise keep memory accesses from
+-- overlapping. To retain that information, the iteration is clamped to @n-1@ in
+-- the full chunks, which does not change its value. This is a bit of a hack,
+-- but it seems robust.
+--
+-- For the same reason, the number of full chunks should be computed in the
+-- kernel when it is a constant, rather than be passed in as a kernel argument,
+-- so that the kernel compiler can unroll the loop. This is why
+-- 'kernelFullChunksMap' does not contain constant dimensions.
+
+-- Note [Failing block-level operators]
+--
+-- Nothing prevents reduction/scan operators from having arbitrary assertions
+-- inside them, which is a bit complicated to handle due to the intricate
+-- synchronisation we perform during collective block-level reductions and
+-- scans.
+--
+-- The problem is that we implement failures by jumping to the next
+-- 'Imp.ErrorSync' in the program text, so no other barrier, nor the warp-level
+-- synchronisation of 'warpLocal', must occur in between (see "Bounds Checking
+-- on GPU", HLPP '20). In 'blockReduce' and 'blockScan', we handle operators
+-- that may fail ('bodyMayFail') by skipping the warp-level phase (which is just
+-- an optimisation anyway) and following each application with an
+-- 'Imp.ErrorSync' in the same iteration. For operators that cannot fail, an
+-- 'Imp.ErrorSync' is just a barrier.
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/Block.hs b/src/Futhark/CodeGen/ImpGen/GPU/Block.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/Block.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/Block.hs
@@ -29,7 +29,7 @@
 import Futhark.IR.Mem.LMAD qualified as LMAD
 import Futhark.Transform.Rename
 import Futhark.Util (chunks, mapAccumLM, takeLast)
-import Futhark.Util.IntegralExp (ceilDiv, rem)
+import Futhark.Util.IntegralExp (ceilDiv, quot, rem)
 import Prelude hiding (quot, rem)
 
 -- | @flattenArray k flat arr@ flattens the outer @k@ dimensions of
@@ -103,6 +103,8 @@
   (Imp.TExp Int32 -> TV Int64 -> InKernelGen ()) ->
   InKernelGen ()
 blockChunkLoop w m = do
+  -- The loop body contains error syncs, and failures must not jump into it.
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
   constants <- kernelConstants <$> askEnv
   let max_chunk_size = sExt32 $ kernelBlockSize constants
   num_chunks <- dPrimVE "num_chunks" $ w `ceilDiv` max_chunk_size
@@ -123,6 +125,27 @@
 crossesSegment segment_size from to =
   (to - from) .>. (to `rem` sExt32 segment_size)
 
+-- | Given a chunk offset, adjust a segment predicate ('crossesSegment') defined
+-- in terms of whole-array indexes to instead be defined in terms of chunk-local
+-- indexes. Also returns whether the chunk begins a new segment. This is used to
+-- process segmented reductions and scans that are chunked.
+chunkSegFlags ::
+  Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->
+  Imp.TExp Int32 ->
+  ( Imp.TExp Bool,
+    Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool)
+  )
+chunkSegFlags seg_flag chunk_start =
+  ( maybe false (\flag_true -> flag_true (chunk_start - 1) chunk_start) seg_flag,
+    fmap shifted seg_flag
+  )
+  where
+    shifted flag_true from to =
+      flag_true (chunk_start + from) (chunk_start + to)
+
+-- | Scan the @w@ first elements of the given arrays in-place. The scan is done
+-- one block-sized chunk at a time, but @seg_flag@ is passed indexes relative to
+-- the whole array.
 virtualisedBlockScan ::
   Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->
   Imp.TExp Int32 ->
@@ -131,14 +154,9 @@
   InKernelGen ()
 virtualisedBlockScan seg_flag w lam arrs = do
   blockChunkLoop w $ \chunk_start chunk_size -> do
-    constants <- kernelConstants <$> askEnv
-    let ltid = kernelLocalThreadId constants
-        crosses_segment =
-          case seg_flag of
-            Nothing -> false
-            Just flag_true ->
-              flag_true (sExt32 (chunk_start - 1)) (sExt32 chunk_start)
-    sComment "possibly incorporate carry" $
+    ltid <- kernelLocalThreadId . kernelConstants <$> askEnv
+    let (crosses_segment, shifted) = chunkSegFlags seg_flag chunk_start
+    sComment "possibly incorporate carry" . localOps threadOperations $
       sWhen (chunk_start .>. 0 .&&. ltid .==. 0 .&&. bNot crosses_segment) $ do
         carry_idx <- dPrimVE "carry_idx" $ sExt64 chunk_start - 1
         applyRenamedLambda
@@ -150,10 +168,21 @@
 
     arrs_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) arrs
 
-    sOp $ Imp.ErrorSync Imp.FenceLocal
+    sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-    blockScan seg_flag (sExt64 w) (tvExp chunk_size) lam arrs_chunks
+    -- The chunks past this one are still live, so the second copy that
+    -- 'blockScan' keeps of an array-typed element must go beyond the whole
+    -- space, not merely beyond the chunk.
+    blockScan shifted (sExt64 w) (tvExp chunk_size) lam arrs_chunks
 
+-- | Semantically, a block-level statement finishes before the next one. This
+-- means that any statement that writes memory must be enclosed in barriers, as
+-- sibling statements may reuse the same memory. Our current approach to this is
+-- a bit naive and may result in duplicate barriers. If this ever turns out to
+-- be a performance problem, we'll do something more sophisticated.
+beforeBlockWrite :: InKernelGen ()
+beforeBlockWrite = sOp $ Imp.Barrier Imp.FenceLocal
+
 copyInBlock :: CopyCompiler GPUMem KernelEnv Imp.KernelOp
 copyInBlock pt destloc srcloc = do
   dest_space <- entryMemSpace <$> lookupMemory (memLocName destloc)
@@ -179,6 +208,7 @@
         (sliceMemLoc destloc destslice')
         (sliceMemLoc srcloc srcslice')
     _ -> do
+      beforeBlockWrite
       blockCoverSpace (map sExt32 dims) $ \is ->
         lmadCopy
           pt
@@ -244,6 +274,19 @@
 
           pure (Just l', f l' (Space "shared") local_subhistos)
 
+-- | The virtualisation that 'blockCoverSegSpace' will actually use for this
+-- index space.  Maybe we can statically detect that the requested one is really
+-- a 'SegNoVirtFull', and generate ever-so-slightly simpler code.  Covering an
+-- index space twice has to know this, as it decides whether the cover loops -
+-- and so introduces a scope of its own for the indexes.
+blockCoverVirt :: SegVirt -> SegSpace -> InKernelGen SegVirt
+blockCoverVirt virt space = do
+  tblock_size <- kernelBlockSize . kernelConstants <$> askEnv
+  pure $
+    if map pe64 (segSpaceDims space) == [tblock_size]
+      then SegNoVirtFull (SegSeqDims [])
+      else virt
+
 blockCoverSegSpace :: SegVirt -> SegSpace -> InKernelGen () -> InKernelGen ()
 blockCoverSegSpace virt space m = do
   let (ltids, dims) = unzip $ unSegSpace space
@@ -251,24 +294,23 @@
 
   constants <- kernelConstants <$> askEnv
   let tblock_size = kernelBlockSize constants
-  -- Maybe we can statically detect that this is actually a
-  -- SegNoVirtFull and generate ever-so-slightly simpler code.
-  let virt' = if dims' == [tblock_size] then SegNoVirtFull (SegSeqDims []) else virt
+  virt' <- blockCoverVirt virt space
   case virt' of
     SegVirt -> do
-      iters <- M.lookup dims . kernelChunkItersMap . kernelConstants <$> askEnv
-      case iters of
-        Nothing -> do
-          iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'
-          blockLoop iterations $ \i -> do
+      iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'
+      let onIteration i = do
             dIndexSpace (zip ltids dims') $ sExt64 i
             m
-        Just num_chunks -> localOps threadOperations $ do
-          let ltid = kernelLocalThreadId constants
-          sFor "chunk_i" num_chunks $ \chunk_i -> do
-            i <- dPrimVE "i" $ chunk_i * sExt32 tblock_size + ltid
-            dIndexSpace (zip ltids dims') $ sExt64 i
-            sWhen (inBounds (Slice (map (DimFix . le64) ltids)) dims') m
+      full_chunks <- M.lookup dims . kernelFullChunksMap . kernelConstants <$> askEnv
+      case full_chunks of
+        Nothing -> blockLoop iterations onIteration
+        Just num_full_chunks ->
+          peeledKernelLoop
+            (kernelLocalThreadId constants)
+            (sExt32 tblock_size)
+            num_full_chunks
+            iterations
+            onIteration
     SegNoVirt -> localOps threadOperations $ do
       zipWithM_ dPrimV_ ltids =<< localThreadIDs dims
       sWhen (isActive $ zip ltids dims) m
@@ -288,7 +330,8 @@
 -- The static arrays stuff does not work inside kernels.
 compileBlockExp (Pat [dest]) (BasicOp (ArrayVal vs t)) =
   compileBlockExp (Pat [dest]) (BasicOp (ArrayLit (map Constant vs) (Prim t)))
-compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) =
+compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) = do
+  beforeBlockWrite
   forM_ (zip [0 ..] es) $ \(i, e) ->
     copyDWIMFix (patElemName dest) [fromIntegral (i :: Int64)] e []
 compileBlockExp _ (BasicOp (UpdateAcc safety acc is vs)) = do
@@ -296,6 +339,7 @@
   sWhen (ltid .==. 0) $ updateAcc safety acc is vs
   sOp $ Imp.Barrier Imp.FenceLocal
 compileBlockExp (Pat [dest]) (BasicOp (Replicate ds se)) | ds /= mempty = do
+  beforeBlockWrite
   flat <- newVName "rep_flat"
   is <- replicateM (arrayRank dest_t) (newVName "rep_i")
   let is' = map le64 is
@@ -305,6 +349,7 @@
   where
     dest_t = patElemType dest
 compileBlockExp (Pat [dest]) (BasicOp (Iota n e s it)) = do
+  beforeBlockWrite
   n' <- toExp n
   e' <- toExp e
   s' <- toExp s
@@ -339,7 +384,7 @@
   -- Avoid that by always doing an error sync here.  Potential
   -- improvement: only do this if any errors are pending (this could
   -- also be handled in later codegen).
-  when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal
+  when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal mempty
   defCompileExp dest e
   where
     doSync Loop {} = True
@@ -362,192 +407,401 @@
 blockAlloc dest _ _ =
   error $ "Invalid target for in-kernel allocation: " ++ show dest
 
-compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp
-compileBlockOp pat (Alloc size space) =
-  blockAlloc pat size space
-compileBlockOp pat (Inner (SegOp (SegMap lvl space _ body))) = do
-  compileFlatId space
+-- | How many elements an operand array for 'blockScan' must be backed by, given
+-- the number of elements actually scanned and the return types of the operator.
+-- Array-typed elements are kept twice over; see 'blockScan'.
+blockScanElems :: Imp.TExp Int64 -> [Type] -> Imp.TExp Int64
+blockScanElems n ts
+  | all primType ts = n
+  | otherwise = 2 * n
 
-  blockCoverSegSpace (segVirt lvl) space $
-    compileStms mempty (bodyStms body) $
-      zipWithM_ (compileThreadResult space) (patElems pat) $
-        bodyResult body
-  sOp $ Imp.ErrorSync Imp.FenceLocal
-compileBlockOp pat (Inner (SegOp (SegScan lvl space _ body scans post_op))) = do
-  compileFlatId space
+-- | A shared memory array of @elems@ values of type @elem_t@, staging the
+-- operands of a 'blockScan' or 'blockReduce'. Note that 'blockScan' needs
+-- memory past the index space it scans, and that excess is deliberately left
+-- outside the shape (but is given by @elems@), as nothing but 'blockScan' may
+-- touch it.
+sharedStagingArray ::
+  Name -> Type -> Imp.TExp Int64 -> Shape -> InKernelGen VName
+sharedStagingArray desc elem_t elems shape = do
+  mem <-
+    sAlloc (desc <> "_mem") (Imp.bytes $ elems * unCount (typeSize elem_t)) $
+      Space "shared"
+  sArrayInMem desc (elemType elem_t) shape mem
 
-  let (ltids, dims) = unzip $ unSegSpace space
-      dims' = map pe64 dims
-      [scan] = scans
-      shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)
-      scan_ts = concatMap shpT scans
-      shpOfT t s =
-        arrayShape $
-          foldr (flip arrayOfRow) (arrayOfShape t s) $
-            segSpaceDims space
-      (scan_pars, map_pars) =
-        splitAt (length $ segBinOpNeutral scan) $ lambdaParams $ segPostOpLambda post_op
-      (body_scan_res, body_map_res) =
-        splitAt (length $ segBinOpNeutral scan) $ bodyResult body
+-- | Will a single thread be responsible for more than one element of this index
+-- space? If so, a value computed for one element in one phase of a multi-phase
+-- operation cannot be kept in a scalar variable until a later phase, as the
+-- thread will by then have overwritten it with the value belonging to some
+-- other element.
+severalElemsPerThread :: SegVirt -> Bool
+severalElemsPerThread SegVirt = True
+severalElemsPerThread SegNoVirt = False
+severalElemsPerThread (SegNoVirtFull (SegSeqDims seq_dims)) = not $ null seq_dims
 
-  scan_out <- forM scan_ts $ \(s, t) ->
-    sAllocArray "scan_out" (elemType t) (shpOfT t s) $ Space "shared"
+-- | Compile a block-level 'SegScan'. This proceeds in three phases, separated
+-- by barriers:
+--
+-- 1. The kernel body is run to produce the scan inputs as distinct arrays.
+--
+-- 2. The scan itself is performed.
+--
+-- 3. Finally the post operator is applied to the scan results.
+--
+-- Everything that has to survive a barrier lives in shared memory, and because
+-- the index space may be virtualised (such that a thread handles more than one
+-- element), that includes the map results of the kernel body.
+compileBlockScan ::
+  Pat LetDecMem ->
+  SegLevel ->
+  SegSpace ->
+  KernelBody GPUMem ->
+  [SegBinOp GPUMem] ->
+  SegPostOp GPUMem ->
+  InKernelGen ()
+compileBlockScan pat lvl space body scans post_op = do
+  compileFlatId space
 
-  dScope Nothing $ scopeOfLParams $ lambdaParams $ segPostOpLambda post_op
+  -- Whether a thread ends up handling several elements of the index space
+  -- decides both where the map results have to live, and how the post operator
+  -- gets at the indexes again.
+  virt <- blockCoverVirt (segVirt lvl) space
 
-  blockCoverSegSpace (segVirt lvl) space $
-    compileStms mempty (bodyStms body) $ do
-      forM_ (zip scan_out body_scan_res) $ \(dest, res) ->
-        copyDWIMFix
-          dest
-          (map Imp.le64 ltids)
-          (kernelResultSubExp res)
-          []
+  scan_arrs <- mapM scanArray scan_ts
+  map_dests <- mapM (mapResDest virt) map_ts
 
-      sComment "bind map results to post lambda params" $
-        forM_ (zip map_pars body_map_res) $ \(par, res) ->
-          copyDWIMFix
-            (paramName par)
-            []
-            (kernelResultSubExp res)
-            []
+  dScope Nothing $ scopeOfLParams $ lambdaParams post_lam
 
-  sOp $ Imp.ErrorSync Imp.FenceLocal
+  computeScanInputs virt scan_arrs map_dests
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+  doScan virt scan_arrs
+  applyPostOp virt scan_arrs map_dests
 
-  let seg_flag = case dims' of
-        [_] -> Nothing -- Nonsegmented.
-        _ -> Just $ crossesSegment $ last dims'
+  sOp $ Imp.Barrier Imp.FenceGlobal
+  where
+    (ltids, dims) = unzip $ unSegSpace space
+    dims' = map pe64 dims
+    [scan] = scans
+    post_lam = segPostOpLambda post_op
+    num_scan_res = length $ segBinOpNeutral scan
+    (scan_pars, map_pars) = splitAt num_scan_res $ lambdaParams post_lam
+    (body_scan_res, body_map_res) = splitAt num_scan_res $ bodyResult body
+    scan_ts = concatMap shpT scans
+    shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)
+    map_ts = map ((mempty,) . paramType) map_pars
 
-  -- blockScan needs to treat the scan output as a one-dimensional
-  -- array of scan elements, so we invent some new flattened arrays
-  -- here.
-  dims_flat <- dPrimV "dims_flat" $ product dims'
-  arrs_flat <-
-    mapM (flattenArray (length dims') dims_flat) scan_out
+    shapeFor s t =
+      arrayShape $ foldr (flip arrayOfRow) (arrayOfShape t s) dims
 
-  case segVirt lvl of
-    SegVirt ->
-      virtualisedBlockScan
-        seg_flag
-        (sExt32 $ tvExp dims_flat)
-        (segBinOpLambda scan)
-        arrs_flat
-    _ ->
-      blockScan
-        seg_flag
-        (product dims')
+    -- One value per element of the index space, plus whatever 'blockScan'
+    -- needs past that.
+    scanArray (s, t) =
+      sharedStagingArray
+        "scan_out"
+        (t `arrayOfShape` s)
+        (blockScanElems (product dims') (lambdaReturnType (segBinOpLambda scan)))
+        (shapeFor s t)
+
+    -- Map results are not operands of the scan, so one value per element of
+    -- the index space is all they need.
+    mapArray (s, t) =
+      sharedStagingArray
+        "map_out"
+        (t `arrayOfShape` s)
         (product dims')
-        (segBinOpLambda scan)
-        arrs_flat
+        (shapeFor s t)
 
-  -- FIXME: we actually need something like blockCoverSegSpacee here, although in
-  -- practice we currently do not generate virtualised scans.
-  sWhen (isActive $ zip ltids dims) . localOps threadOperations $ do
-    sComment "bind scan results to post lambda params" $
-      forM_ (zip scan_pars scan_out) $ \(par, acc) ->
-        copyDWIMFix (paramName par) [] (Var acc) (map Imp.le64 ltids)
+    -- Where does a map result produced by the kernel body live until the post
+    -- operator needs it? A shared memory array when a thread handles several
+    -- elements, as a scalar would by then have been overwritten; otherwise the
+    -- post operator parameter itself. Accumulators are phantoms in ImpGen, so
+    -- they need no storage at all.
+    mapResDest virt (s, t)
+      | isAcc t = pure Nothing
+      | severalElemsPerThread virt = Just <$> mapArray (s, t)
+      | otherwise = pure Nothing
 
-    let res = fmap resSubExp $ bodyResult $ lambdaBody $ segPostOpLambda post_op
-    sComment "compute post op." $
-      compileStms mempty (bodyStms $ lambdaBody $ segPostOpLambda post_op) $ do
-        sComment "write values" $
-          forM_ (zip (patElems pat) res) $ \(pe, subexp) ->
-            copyDWIMFix (patElemName pe) (map le64 ltids) subexp []
+    seg_flag = case dims' of
+      [_] -> Nothing -- Nonsegmented.
+      _ -> Just $ crossesSegment $ last dims'
 
-  sOp $ Imp.Barrier Imp.FenceGlobal
-compileBlockOp pat (Inner (SegOp (SegRed lvl space _ body ops))) = do
-  compileFlatId space
+    -- Run the kernel body for every element of the index space, stashing the
+    -- scan inputs - and the map results, if they cannot be kept in scalars -
+    -- in shared memory.
+    computeScanInputs virt scan_arrs map_dests =
+      blockCoverSegSpace virt space $
+        compileStms mempty (bodyStms body) $ do
+          let ltids' = map Imp.le64 ltids
+          forM_ (zip scan_arrs body_scan_res) $ \(arr, res) ->
+            copyDWIMFix arr ltids' (kernelResultSubExp res) []
 
-  let dims' = map pe64 dims
-      mkTempArr t =
-        sAllocArray "red_arr" (elemType t) (Shape dims <> arrayShape t) $ Space "shared"
+          sComment "stash map results for the post operator" $
+            forM_ (zip3 map_pars map_dests body_map_res) $ \(par, dest, res) ->
+              case dest of
+                Just arr -> copyDWIMFix arr ltids' (kernelResultSubExp res) []
+                Nothing -> copyDWIMFix (paramName par) [] (kernelResultSubExp res) []
 
-  tmp_arrs <- mapM mkTempArr $ concatMap (lambdaReturnType . segBinOpLambda) ops
-  blockCoverSegSpace (segVirt lvl) space $
-    compileStms mempty (bodyStms body) $ do
-      let (red_res, map_res) =
-            splitAt (segBinOpResults ops) $ bodyResult body
-      forM_ (zip tmp_arrs red_res) $ \(dest, res) ->
-        copyDWIMFix dest (map Imp.le64 ltids) (kernelResultSubExp res) []
-      zipWithM_ (compileThreadResult space) map_pes map_res
+    -- blockScan needs to treat the scan input as a one-dimensional array of
+    -- scan elements, so we invent some new flattened arrays here.
+    doScan virt scan_arrs = do
+      dims_flat <- dPrimV "dims_flat" $ product dims'
+      arrs_flat <- mapM (flattenArray (length dims') dims_flat) scan_arrs
+      case virt of
+        SegVirt ->
+          virtualisedBlockScan
+            seg_flag
+            (sExt32 $ tvExp dims_flat)
+            (segBinOpLambda scan)
+            arrs_flat
+        _ ->
+          blockScan
+            seg_flag
+            (product dims')
+            (product dims')
+            (segBinOpLambda scan)
+            arrs_flat
 
-  sOp $ Imp.ErrorSync Imp.FenceLocal
+    applyPostOp virt scan_arrs map_dests
+      | severalElemsPerThread virt =
+          blockCoverSegSpace virt space postOp
+      | otherwise =
+          -- In this case the ltids from computeScanInputs are still in scope.
+          sWhen (isActive $ zip ltids dims) $ localOps threadOperations postOp
+      where
+        ltids' = map Imp.le64 ltids
+        res = fmap resSubExp $ bodyResult $ lambdaBody post_lam
+        postOp = do
+          sComment "bind scan results to post lambda params" $
+            forM_ (zip scan_pars scan_arrs) $ \(par, arr) ->
+              copyDWIMFix (paramName par) [] (Var arr) ltids'
 
-  let tmps_for_ops = chunks (map (length . segBinOpNeutral) ops) tmp_arrs
-  case segVirt lvl of
-    SegVirt -> virtCase dims' tmps_for_ops
-    _ -> nonvirtCase dims' tmps_for_ops
+          sComment "bind map results to post lambda params" $
+            forM_ (zip map_pars map_dests) $ \(par, dest) ->
+              forM_ dest $ \arr ->
+                copyDWIMFix (paramName par) [] (Var arr) ltids'
+
+          sComment "compute post op." $
+            compileStms mempty (bodyStms $ lambdaBody post_lam) $
+              sComment "write values" $
+                forM_ (zip (patElems pat) res) $ \(pe, subexp) ->
+                  copyDWIMFix (patElemName pe) ltids' subexp []
+
+isEmptySpace :: SegSpace -> Imp.TExp Bool
+isEmptySpace = (.==. 0) . product . map pe64 . segSpaceDims
+
+-- | Compile a block-level 'SegRed'. Unless the index space is virtualised, the
+-- kernel body is first run for the entire index space, with the results stashed
+-- in shared memory, which is then reduced. A virtualised index space is instead
+-- handled one block-sized chunk at a time - chunks of the /flattened/ space,
+-- when there are several segments - so that we need no more shared memory than
+-- there are threads in the block.
+compileBlockRed ::
+  Pat LetDecMem ->
+  SegLevel ->
+  SegSpace ->
+  KernelBody GPUMem ->
+  [SegBinOp GPUMem] ->
+  InKernelGen ()
+compileBlockRed pat lvl space body ops = do
+  compileFlatId space
+
+  sIf (isEmptySpace space) emptyCase $ do
+    virt <- blockCoverVirt (segVirt lvl) space
+    case (virt, dims') of
+      (SegVirt, [dim']) -> chunkedCase dim'
+      (SegVirt, _) -> segmentedChunkedCase
+      (_, [dim']) -> stageWholeSpace (product dims') $ singleSegmentCase dim'
+      _ -> stageWholeSpace (blockScanElems (product dims') red_ts) segmentedCase
   where
     (ltids, dims) = unzip $ unSegSpace space
+    dims' = map pe64 dims
     (red_pes, map_pes) = splitAt (segBinOpResults ops) $ patElems pat
+    (body_red_res, body_map_res) =
+      splitAt (segBinOpResults ops) $ bodyResult body
+    red_ts = concatMap (lambdaReturnType . segBinOpLambda) ops
+    forOps :: [a] -> [[a]]
+    forOps = chunks $ map (length . segBinOpNeutral) ops
 
-    virtCase [dim'] tmps_for_ops = do
+    -- Shared memory for @elems@ values per reduction operand, declared with
+    -- the given shape.
+    sharedArrays elems shape =
+      forM red_ts $ \t ->
+        sharedStagingArray "red_arr" t elems (shape <> arrayShape t)
+
+    -- Write the reduction results, read at @src@ in the staging arrays, to
+    -- @dest@ in the destinations.
+    saveResults dest tmps_for_ops src =
+      sComment "Save result of reduction." $
+        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->
+          copyDWIM (patElemName pe) dest (Var arr) src
+
+    -- The result of the chunks so far accumulates at @dest_is@ in the
+    -- destination, so the carry is incorporated by folding the destination into
+    -- the chunk's first element; the reduction or scan that follows propagates
+    -- it through the rest of the chunk.
+    incorporateCarry due dest_is tmps_for_ops = do
       ltid <- kernelLocalThreadId . kernelConstants <$> askEnv
+      sComment "possibly incorporate carry" . localOps threadOperations $
+        sWhen (due .&&. ltid .==. 0) $
+          forM_ (zip3 ops tmps_for_ops (forOps red_pes)) $ \(op, tmps, pes) ->
+            applyRenamedLambda
+              (segBinOpLambda op)
+              (map (,[DimFix 0]) tmps)
+              ( map ((,dest_is) . Var . patElemName) pes
+                  ++ map ((,[DimFix 0]) . Var) tmps
+              )
+
+    -- Run the segop body for the point of the index space that the thread's
+    -- index variables are currently bound to, saving the reduction inputs at
+    -- @is@ in the given @tmp_arrs@. Map results go straight to their
+    -- destinations.
+    runBody tmp_arrs is =
+      compileStms mempty (bodyStms body) $ do
+        forM_ (zip tmp_arrs body_red_res) $ \(dest, res) ->
+          copyDWIMFix dest is (kernelResultSubExp res) []
+        zipWithM_ (compileThreadResult space) map_pes body_map_res
+
+    emptyCase = do
+      -- This is the result if there are no chunks at all.
+      sComment "set result to neutral element" $
+        blockCoverSpace (map pe64 $ init dims) $ \is ->
+          forM_ (zip red_pes $ concatMap segBinOpNeutral ops) $ \(pe, ne) ->
+            copyDWIMFix (patElemName pe) is ne []
+
+    -- Run the kernel body for the entire index space, staging the results in
+    -- shared memory, then reduce them with the passed-in continuation.
+    stageWholeSpace elems k = do
+      tmp_arrs <- sharedArrays elems $ Shape dims
+      blockCoverSegSpace (segVirt lvl) space $
+        runBody tmp_arrs (map Imp.le64 ltids)
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+      k $ forOps tmp_arrs
+
+    -- A single segment, virtualised. We run the kernel body for one block-sized
+    -- chunk at a time, reduce that chunk, and carry the result into the next
+    -- chunk.
+    chunkedCase dim' = do
+      tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv
+      ltid <- kernelLocalThreadId . kernelConstants <$> askEnv
+      -- A chunk is never larger than the block, nor than the index space
+      -- itself, so that is all the shared memory we need.
+      tmp_arrs <-
+        sharedArrays (sMin64 dim' (pe64 (unCount tblock_size))) $
+          Shape [unCount tblock_size]
+      let tmps_for_ops = forOps tmp_arrs
+          [ltid_name] = ltids
+
       blockChunkLoop (sExt32 dim') $ \chunk_start chunk_size -> do
-        sComment "possibly incorporate carry" $
-          sWhen (chunk_start .>. 0 .&&. ltid .==. 0) $
-            forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->
-              applyRenamedLambda
-                (segBinOpLambda op)
-                (map (,[DimFix $ sExt64 chunk_start]) tmps)
-                ( map ((,[]) . Var . patElemName) red_pes
-                    ++ map ((,[DimFix $ sExt64 chunk_start]) . Var) tmps
-                )
+        sComment "run the kernel body for this chunk" . localOps threadOperations $ do
+          dPrimV_ ltid_name $ sExt64 chunk_start + sExt64 ltid
+          sWhen (Imp.le64 ltid_name .<. dim') $
+            runBody tmp_arrs [sExt64 ltid]
 
-        sOp $ Imp.ErrorSync Imp.FenceLocal
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-        forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do
-          tmps_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) tmps
-          blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps_chunks
+        incorporateCarry (chunk_start .>. 0) [] tmps_for_ops
 
-        sOp $ Imp.ErrorSync Imp.FenceLocal
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-        sComment "Save result of reduction." $
-          forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->
-            copyDWIMFix (patElemName pe) [] (Var arr) [sExt64 chunk_start]
+        forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->
+          blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps
 
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+
+        saveResults [] tmps_for_ops [DimFix 0]
+
+        -- Everyone must be done reading the result before the next chunk
+        -- overwrites it.
+        sOp $ Imp.Barrier Imp.FenceLocal
+
+    -- Multiple segments, virtualised. We go through the flattened index space
+    -- one block-sized chunk at a time: run the segop body for the chunk, scan
+    -- the chunk with the segment flags, then let every thread whose element is
+    -- the last of its segment write that segment's result. The running value of
+    -- a segment that continues past the end of the chunk is carried into the
+    -- next chunk.
     --
-    virtCase dims' tmps_for_ops = do
-      dims_flat <- dPrimV "dims_flat" $ product dims'
+    -- Chunking the flattened space rather than taking whole segments at a time
+    -- keeps every thread busy no matter how the segment size relates to the
+    -- block size, and needs no separate strategy for segments larger than the
+    -- block.
+    segmentedChunkedCase = do
+      tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv
+      ltid <- kernelLocalThreadId . kernelConstants <$> askEnv
+      let w = product dims'
+          -- A chunk is never larger than the block, nor than the index space
+          -- itself, so that is all the shared memory we need.
+          chunk_elems = sMin64 w $ pe64 $ unCount tblock_size
+          seg_flag = crossesSegment $ last dims'
+      tmp_arrs <-
+        sharedArrays (blockScanElems chunk_elems red_ts) $
+          Shape [unCount tblock_size]
+      let tmps_for_ops = forOps tmp_arrs
 
-      forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do
-        tmps_flat <- mapM (flattenArray (length dims') dims_flat) tmps
-        virtualisedBlockScan
-          (Just $ crossesSegment $ last dims')
-          (sExt32 $ tvExp dims_flat)
-          (segBinOpLambda op)
-          tmps_flat
+      blockChunkLoop (sExt32 w) $ \chunk_start chunk_size -> do
+        flat_idx <- dPrimVE "flat_idx" $ sExt64 chunk_start + sExt64 ltid
+        in_bounds <- dPrimVE "in_bounds" $ flat_idx .<. w
+        dIndexSpace (zip ltids dims') flat_idx
 
-      sOp $ Imp.ErrorSync Imp.FenceLocal
+        sComment "run segred body for this chunk" . localOps threadOperations $
+          sWhen in_bounds $
+            runBody tmp_arrs [sExt64 ltid]
 
-      sComment "Save result of reduction." $
-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->
-          copyDWIM
-            (patElemName pe)
-            []
-            (Var arr)
-            (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-      sOp $ Imp.Barrier Imp.FenceLocal
+        let (crosses_segment, shifted) =
+              chunkSegFlags (Just seg_flag) chunk_start
 
-    -- Nonsegmented case (or rather, a single segment) - this we can
-    -- handle directly with a block-level reduction.
-    nonvirtCase [dim'] tmps_for_ops = do
+        -- Thread zero is at @chunk_start@, so its segment index is that of the
+        -- segment the carry belongs to - the one the preceding chunk ended in.
+        incorporateCarry
+          (chunk_start .>. 0 .&&. bNot crosses_segment)
+          (map (DimFix . Imp.le64) (init ltids))
+          tmps_for_ops
+
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+
+        -- Scan the currente chunk.
+        forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->
+          blockScan shifted chunk_elems (tvExp chunk_size) (segBinOpLambda op) tmps
+
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+
+        -- A thread whose element is the last of its segment holds that
+        -- segment's final result. The thread holding the last element of the
+        -- chunk holds the running result of a segment that continues into the
+        -- next chunk, which that chunk then reads back as its carry.
+        ends_segment <-
+          dPrimVE "ends_segment" $ Imp.le64 (last ltids) .==. last dims' - 1
+        ends_chunk <-
+          dPrimVE "ends_chunk" $ sExt64 ltid .==. tvExp chunk_size - 1
+        localOps threadOperations $
+          sWhen (in_bounds .&&. (ends_segment .||. ends_chunk)) $
+            saveResults
+              (map (DimFix . Imp.le64) (init ltids))
+              tmps_for_ops
+              [DimFix $ sExt64 ltid]
+
+        -- Everyone must be done reading the chunk before the next one
+        -- overwrites it, and the carry must be visible to the next chunk.
+        fence <- fenceForArrays $ map patElemName red_pes
+        sOp $ Imp.Barrier fence
+
+    -- A single segment - this we can handle directly with a block-level
+    -- reduction.
+    singleSegmentCase dim' tmps_for_ops = do
       forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->
         blockReduce (sExt32 dim') (segBinOpLambda op) tmps
-      sOp $ Imp.ErrorSync Imp.FenceLocal
-      sComment "Save result of reduction." $
-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->
-          copyDWIMFix (patElemName pe) [] (Var arr) [0]
-      sOp $ Imp.ErrorSync Imp.FenceLocal
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+      saveResults [] tmps_for_ops [DimFix 0]
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-    -- Segmented intra-block reductions are turned into (regular)
-    -- segmented scans.  It is possible that this can be done
+    -- Multiple segments, with the whole index space staged in shared memory.
+    -- These are turned into (regular) segmented scans, of which we then pick
+    -- the last element of each segment. It is possible that this can be done
     -- better, but at least this approach is simple.
-    nonvirtCase dims' tmps_for_ops = do
-      -- blockScan operates on flattened arrays.  This does not
-      -- involve copying anything; merely playing with the index
-      -- function.
+    segmentedCase tmps_for_ops = do
+      -- The scans operate on flattened arrays. This does not involve copying
+      -- anything; merely playing with the index function.
       dims_flat <- dPrimV "dims_flat" $ product dims'
 
       forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do
@@ -559,19 +813,39 @@
           (segBinOpLambda op)
           tmps_flat
 
-      sOp $ Imp.ErrorSync Imp.FenceLocal
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
-      sComment "Save result of reduction." $
-        forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->
-          copyDWIM
-            (patElemName pe)
-            []
-            (Var arr)
-            (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])
+      saveResults [] tmps_for_ops lastInSegment
 
       sOp $ Imp.Barrier Imp.FenceLocal
-compileBlockOp pat (Inner (SegOp (SegHist lvl space _ kbody ops))) = do
+
+    lastInSegment =
+      map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1]
+
+compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp
+compileBlockOp pat (Alloc size space) =
+  blockAlloc pat size space
+compileBlockOp pat (Inner (SegOp op)) = do
+  beforeBlockWrite
+  compileBlockSegOp pat op
+compileBlockOp pat _ =
+  compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat
+
+compileBlockSegOp :: Pat LetDecMem -> SegOp SegLevel GPUMem -> InKernelGen ()
+compileBlockSegOp pat (SegMap lvl space _ body) = do
   compileFlatId space
+
+  blockCoverSegSpace (segVirt lvl) space $
+    compileStms mempty (bodyStms body) $
+      zipWithM_ (compileThreadResult space) (patElems pat) $
+        bodyResult body
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+compileBlockSegOp pat (SegScan lvl space _ body scans post_op) =
+  compileBlockScan pat lvl space body scans post_op
+compileBlockSegOp pat (SegRed lvl space _ body ops) =
+  compileBlockRed pat lvl space body ops
+compileBlockSegOp pat (SegHist lvl space _ kbody ops) = do
+  compileFlatId space
   let (ltids, dims) = unzip $ unSegSpace space
 
   -- We don't need the red_pes, because it is guaranteed by our type
@@ -613,9 +887,7 @@
                   copyDWIMFix (paramName p) [] v is
                 do_op (bin_is ++ is)
 
-  sOp $ Imp.ErrorSync Imp.FenceLocal
-compileBlockOp pat _ =
-  compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
 -- | The operations for block-level kernels.
 blockOperations :: Operations GPUMem KernelEnv Imp.KernelOp
@@ -744,7 +1016,7 @@
 -- | Various useful precomputed information for block-level SegOps.
 data Precomputed = Precomputed
   { pcSegOpSizes :: SegOpSizes,
-    pcChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)
+    pcFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32)
   }
 
 -- | Find the sizes of nested parallelism in a t'SegOp' body.
@@ -773,13 +1045,16 @@
 precomputeConstants :: Count BlockSize (Imp.TExp Int64) -> Stms GPUMem -> CallKernelGen Precomputed
 precomputeConstants tblock_size stms = do
   let sizes = segOpSizes stms
-  iters_map <- M.fromList <$> mapM mkMap (S.toList sizes)
-  pure $ Precomputed sizes iters_map
+  -- Constant sizes are left to the kernel; see Note [Peeled block loops].
+  full_chunks_map <-
+    M.fromList <$> mapM mkMap (filter (any (isJust . subExpVar)) (S.toList sizes))
+  pure $ Precomputed sizes full_chunks_map
   where
     mkMap dims = do
       let n = product $ map Imp.pe64 dims
-      num_chunks <- dPrimVE "num_chunks" $ sExt32 $ n `ceilDiv` unCount tblock_size
-      pure (dims, num_chunks)
+      num_full_chunks <-
+        dPrimVE "num_full_chunks" $ sExt32 $ n `quot` unCount tblock_size
+      pure (dims, num_full_chunks)
 
 -- | Make use of various precomputed constants.
 precomputedConstants :: Precomputed -> InKernelGen a -> InKernelGen a
@@ -791,7 +1066,7 @@
           { kernelConstants =
               (kernelConstants env)
                 { kernelLocalIdMap = new_ids,
-                  kernelChunkItersMap = pcChunkItersMap pre
+                  kernelFullChunksMap = pcFullChunksMap pre
                 }
           }
   localEnv f m
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs
@@ -196,21 +196,19 @@
       [Imp.TExp Int64] -> InKernelGen ()
     )
 prepareAtomicUpdateGlobal l segments dests slug =
-  -- We need a separate lock array if the operators are not all of a
-  -- particularly simple form that permits pure atomic operations.
+  -- We need locks if the operators are not all of a particularly
+  -- simple form that permits pure atomic operations.
   case (l, slugAtomicUpdate slug) of
     (_, AtomicPrim f) -> pure (l, f (Space "global") dests)
     (_, AtomicCAS f) -> pure (l, f (Space "global") dests)
     (Just l', AtomicLocking f) -> pure (l, f l' (Space "global") dests)
     (Nothing, AtomicLocking f) -> do
-      -- The number of locks used here is too low, but since we are
-      -- currently forced to inline a huge list, I'm keeping it down
-      -- for now.  Some quick experiments suggested that it has little
-      -- impact anyway (maybe the locking case is just too slow).
-      --
-      -- A fun solution would also be to use a simple hashing
-      -- algorithm to ensure good distribution of locks.
-      let num_locks = 100151
+      -- The lock for a bucket is chosen by its flat index modulo the
+      -- number of locks.  More locks, or a simple hashing algorithm
+      -- that distributes the buckets better, might reduce contention,
+      -- but quick experiments suggested that it has little impact
+      -- (maybe the locking case is just too slow).
+      let num_locks = Imp.bookkeepingSize Imp.BookkeepingLocks
           dims =
             map pe64 $
               shapeDims segments
@@ -218,7 +216,7 @@
                 ++ [tvSize (slugNumSubhistos slug)]
                 ++ shapeDims (histShape (slugOp slug))
 
-      locks <- genZeroes "hist_locks" num_locks
+      locks <- bookkeepingArray Imp.BookkeepingLocks
       let l' = Locking locks 0 1 0 (pure . (`rem` fromIntegral num_locks) . flattenIndex dims)
       pure (Just l', f l' (Space "global") dests)
 
@@ -781,7 +779,7 @@
                           copyDWIMFix (paramName p) [] v is
                         do_op (bucket_is ++ is)
 
-        sOp $ Imp.ErrorSync Imp.FenceGlobal
+        sOp $ Imp.ErrorSync Imp.FenceGlobal mempty
 
         sComment "Compact the multiple shared memory subhistograms to result in global memory" $
           onSlugs $ \slug dests hist_H_chk histo_dims _histo_size bins_per_thread -> do
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs
@@ -9,13 +9,8 @@
 --   kernel, with the final threadblock to write a result (tracked via
 --   an atomic counter) performing the final reduction as well.
 --
--- * Instead of depending on storage layout transformations to handle
---   non-commutative reductions efficiently, we slide a
---   @tblocksize@-sized window over the input, and perform a parallel
---   reduction for each window.  This sacrifices the notion of
---   efficient sequentialisation, but is sometimes faster and
---   definitely simpler and more predictable (and uses less auxiliary
---   storage).
+-- * We have various strategies for handling non-commutative reductions. See
+--   Note [Stage one strategies].
 --
 -- For segmented reductions we use the approach from "Strategies for
 -- Regular Segmented Reductions on GPU" (FHPC '17).  This involves
@@ -30,10 +25,8 @@
 --   value per segment.
 --
 --   Each thread /can/ read multiple elements, which will greatly
---   increase performance; however, if the reduction is
---   non-commutative we will have to use a less efficient traversal
---   (with interim block-wide reductions) to enable coalesced memory
---   accesses, just as in the non-segmented case.
+--   increase performance.  Non-commutative reductions use the same
+--   strategies as in the non-segmented case.
 --
 -- * Small: is used to let each block process *multiple* segments
 --   within a block. We will only use this approach when we can
@@ -41,13 +34,6 @@
 --   cases, we would allocate a /whole/ block per segment with the
 --   large strategy, but at most 50% of the threads in the block would
 --   have any element to read, which becomes highly inefficient.
---
--- An optimization specfically targeted at non-segmented and large-segments
--- segmented reductions with non-commutative is made: The stage one main loop is
--- essentially stripmined by a factor *chunk*, inserting collective copies via
--- shared memory of each reduction parameter going into the intra-block (partial)
--- reductions. This saves a factor *chunk* number of intra-block reductions at
--- the cost of some overhead in collective copies.
 module Futhark.CodeGen.ImpGen.GPU.SegRed
   ( compileSegRed,
     compileSegRed',
@@ -56,7 +42,6 @@
 where
 
 import Control.Monad
-import Data.List (zip4)
 import Data.Map qualified as M
 import Data.Maybe
 import Futhark.CodeGen.ImpCode.GPU qualified as Imp
@@ -72,6 +57,11 @@
 forM2_ :: (Monad m) => [a] -> [b] -> (a -> b -> m c) -> m ()
 forM2_ xs ys f = forM_ (zip xs ys) (uncurry f)
 
+-- | The total size of values of the given types in 32-bit words, where
+-- smaller types occupy a full word.
+primWords :: [PrimType] -> Int
+primWords = sum . map ((`div` 4) . max 4 . primByteSize)
+
 -- | Given available register and a list of parameter types, compute
 -- the largest available chunk size given the parameters for which we
 -- want chunking and the available resources.
@@ -91,7 +81,7 @@
       sizes = map primByteSize types'
 
       sum_sizes = sum sizes
-      sum_sizes' = sum (map (sMax64 4 . primByteSize) types') `quot` 4
+      sum_sizes' = fromIntegral $ primWords types'
       max_size = maximum sizes
 
       mem_constraint = max k_mem sum_sizes `quot` max_size
@@ -153,6 +143,45 @@
   all primType (lambdaReturnType $ segBinOpLambda segbinop)
     && shapeRank (segBinOpShape segbinop) == 0
 
+-- | How the first stage of a reduction distributes the input among the threads.
+-- For elaboration on the strategies, see Note [Stage one strategies].
+data StageOne
+  = -- | Thread @t@ of @T@ handles elements @t@, @t+T@, @t+2T@, and so on, not
+    -- combining the per-thread results until all input has been processed.
+    -- Requires a commutative operator.
+    StageOneStrided
+  | -- | Each block handles a sequence of contiguous tiles, using collective
+    -- copies through shared memory and a block-wide reduction per tile.
+    -- Requires primitive (non-array) operands.
+    StageOneTiled
+  | -- | Each thread handles a contiguous range of the input.
+    StageOneContiguous
+  | -- | Each block slides a window of @tblock_size@ elements across a
+    -- contiguous range of the input, with a block-wide reduction per window.
+    StageOneWindow
+  deriving (Eq)
+
+-- | The largest total size of the reduction operands, in 32-bit words, for
+-- which we use 'StageOneTiled'. Heuristically chosen. It is difficult to make
+-- this a tuning parameter, as the strategy is a compile-time choice.
+maxTiledOperandWords :: Int
+maxTiledOperandWords = 16
+
+stageOneStrategy :: [SegBinOp GPUMem] -> StageOne
+stageOneStrategy segbinops
+  | all ((== Commutative) . segBinOpComm) segbinops =
+      StageOneStrided
+  | not $ all isPrimSegBinOp segbinops =
+      StageOneWindow
+  | operand_words <= maxTiledOperandWords =
+      StageOneTiled
+  | otherwise =
+      StageOneContiguous
+  where
+    operand_words =
+      primWords . map elemType $
+        concatMap (lambdaReturnType . segBinOpLambda) segbinops
+
 -- | Like 'compileSegRed', but where the body is a monadic action.
 compileSegRed' ::
   Pat LetDecMem ->
@@ -175,19 +204,19 @@
         (compileReduction (chunk_v, chunk_const) largeSegmentsReduction)
   where
     compileReduction chunk f =
-      f pat num_tblocks tblock_size chunk space segbinops map_body_cont
+      f pat num_tblocks tblock_size strategy chunk space segbinops map_body_cont
 
     param_types = map paramType $ concatMap paramOf segbinops
 
     num_tblocks = gridNumBlocks grid
     tblock_size = gridBlockSize grid
 
-    chunk_const =
-      if Noncommutative `elem` map segBinOpComm segbinops
-        && all isPrimSegBinOp segbinops
-        then getRedChunkSize param_types
-        else Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)
+    strategy = stageOneStrategy segbinops
 
+    chunk_const
+      | strategy == StageOneTiled = getRedChunkSize param_types
+      | otherwise = Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)
+
 -- | Prepare intermediate arrays for the reduction.  Prim-typed
 -- arguments go in shared memory (so we need to do the allocation of
 -- those arrays inside the kernel), while array-typed arguments go in
@@ -198,14 +227,14 @@
 -- For more info about the intermediate arrays used for the different reduction
 -- kernels, see note [IntermArrays].
 makeIntermArrays ::
+  StageOne ->
   Imp.TExp Int64 ->
   SubExp ->
   SubExp ->
   [SegBinOp GPUMem] ->
   InKernelGen [SegRedIntermediateArrays]
-makeIntermArrays tblock_id tblock_size chunk segbinops
-  | Noncommutative <- mconcat (map segBinOpComm segbinops),
-    all isPrimSegBinOp segbinops =
+makeIntermArrays strategy tblock_id tblock_size chunk segbinops
+  | strategy == StageOneTiled =
       noncommPrimSegRedInterms
   | otherwise =
       generalSegRedInterms tblock_id tblock_size segbinops
@@ -282,10 +311,10 @@
 -- | Arrays for storing block results.
 --
 -- The block-result arrays have an extra dimension because they are
--- also used for keeping vectorised accumulators for first-stage
--- reduction, if necessary.  If necessary, this dimension has size
--- tblock_size, and otherwise 1.  When actually storing block results,
--- the first index is set to 0.
+-- also used for keeping the per-thread accumulators for first-stage
+-- reduction of non-primitive operands.  In that case, this dimension
+-- has size tblock_size, and otherwise 1.  When actually storing block
+-- results, the first index is set to 0.
 groupResultArrays ::
   SubExp ->
   SubExp ->
@@ -308,6 +337,7 @@
   Pat LetDecMem ->
   Count NumBlocks SubExp ->
   Count BlockSize SubExp ->
+  StageOne ->
   (TV Int64, Imp.KernelConstExp) ->
   SegSpace ->
   [SegBinOp GPUMem] ->
@@ -315,7 +345,7 @@
   CallKernelGen ()
 
 nonsegmentedReduction :: DoCompileSegRed
-nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do
+nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do
   let (gtids, dims) = unzip $ unSegSpace space
       chunk = tvExp chunk_v
       num_tblocks_se = unCount num_tblocks
@@ -324,7 +354,7 @@
       global_tid = Imp.le64 $ segFlat space
       n = pe64 $ last dims
 
-  counters <- genZeroes "counters" $ length segbinops
+  counters <- bookkeepingArray Imp.BookkeepingCounters
 
   reds_block_res_arrs <- groupResultArrays num_tblocks_se tblock_size_se segbinops
 
@@ -341,7 +371,7 @@
     let ltid = kernelLocalThreadId constants
     let tblock_id = kernelBlockId constants
 
-    interms <- makeIntermArrays (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops
+    interms <- makeIntermArrays strategy (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops
     sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared"
 
     -- Since this is the nonsegmented case, all outer segment IDs must
@@ -355,6 +385,7 @@
         zip3 segbinops interms reds_block_res_arrs
     new_lambdas <-
       reductionStageOne
+        strategy
         gtids
         n
         global_tid
@@ -369,22 +400,18 @@
             (map (length . segBinOpNeutral) segbinops)
             segred_pes
 
-    forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $
-      \(pes, slug, new_lambda, i) ->
-        reductionStageTwo
-          pes
-          tblock_id
-          [0]
-          0
-          (sExt64 $ kernelNumBlocks constants)
-          slug
-          new_lambda
-          counters
-          sync_arr
-          (fromInteger i)
+    reductionStageTwo
+      tblock_id
+      [0]
+      0
+      (sExt64 $ kernelNumBlocks constants)
+      (zip3 segred_pess slugs new_lambdas)
+      counters
+      sync_arr
+      0
 
 smallSegmentsReduction :: DoCompileSegRed
-smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ space segbinops map_body_cont = do
+smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ _ space segbinops map_body_cont = do
   let (gtids, dims) = unzip $ unSegSpace space
       dims' = map pe64 dims
       segment_size = last dims'
@@ -453,7 +480,7 @@
           in_bounds
           out_of_bounds
 
-      sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier.
       let crossesSegment from to =
             (sExt64 to - sExt64 from) .>. (sExt64 to `rem` segment_size)
       sWhen (segment_size .>. 0) $
@@ -495,7 +522,7 @@
       sOp $ Imp.Barrier Imp.FenceLocal
 
 largeSegmentsReduction :: DoCompileSegRed
-largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do
+largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do
   let (gtids, dims) = unzip $ unSegSpace space
       dims' = map pe64 dims
       num_segments = product $ init dims'
@@ -537,13 +564,13 @@
   -- However, we exploit the fact that the number of segments being
   -- reduced at any point in time is limited by the number of
   -- threadblocks. If we bound the number of threadblocks, we can get away
-  -- with using that many counters.  FIXME: Is this limit checked
-  -- anywhere?  There are other places in the compiler that will fail
-  -- if the block count exceeds the maximum block size, which is at
-  -- most 1024 anyway.
-  let counters_per_op = 1024
-  counters <-
-    genZeroes "counters" $ fromIntegral $ length segbinops * counters_per_op
+  -- with using that many counters, so we use the bookkeeping counters (see
+  -- 'Imp.Bookkeeping'), indexed by the segment modulo their number.  FIXME:
+  -- Is this limit checked anywhere?  There are other places in the compiler
+  -- that will fail if the block count exceeds the maximum block size, which
+  -- is at most 1024 anyway.
+  counters <- bookkeepingArray Imp.BookkeepingCounters
+  let num_counters = Imp.bookkeepingSize Imp.BookkeepingCounters
 
   let attrs =
         (defKernelAttrs num_tblocks tblock_size)
@@ -555,7 +582,7 @@
     let tblock_id = sExt64 $ kernelBlockId constants
         ltid = kernelLocalThreadId constants
 
-    interms <- makeIntermArrays tblock_id tblock_size_se (tvSize chunk_v) segbinops
+    interms <- makeIntermArrays strategy tblock_id tblock_size_se (tvSize chunk_v) segbinops
     sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared"
 
     -- We probably do not have enough actual threadblocks to cover the
@@ -583,6 +610,7 @@
           zip3 segbinops interms reds_block_res_arrs
       new_lambdas <-
         reductionStageOne
+          strategy
           gtids
           n
           global_tid
@@ -598,23 +626,15 @@
               segred_pes
 
           multiple_blocks_per_segment =
-            forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $
-              \(pes, slug, new_lambda, i) -> do
-                let counter_idx =
-                      fromIntegral (i * counters_per_op)
-                        + flat_segment_id
-                          `rem` fromIntegral counters_per_op
-                reductionStageTwo
-                  pes
-                  virttblock_id
-                  (map Imp.le64 segment_gtids)
-                  first_block_for_segment
-                  blocks_per_segment
-                  slug
-                  new_lambda
-                  counters
-                  sync_arr
-                  counter_idx
+            reductionStageTwo
+              virttblock_id
+              (map Imp.le64 segment_gtids)
+              first_block_for_segment
+              blocks_per_segment
+              (zip3 segred_pess slugs new_lambdas)
+              counters
+              sync_arr
+              (flat_segment_id `rem` fromIntegral num_counters)
 
           one_block_per_segment =
             sComment "first thread in block saves final result to memory" $
@@ -633,7 +653,9 @@
   { slugOp :: SegBinOp GPUMem,
     -- | Intermediate arrays needed for the given reduction.
     slugInterms :: SegRedIntermediateArrays,
-    -- | Place(s) to store block accumulator(s) in stage 1 reduction.
+    -- | Per-thread accumulator(s) for the stage one reduction.  After
+    -- each block-wide reduction, the accumulator(s) of thread 0 hold
+    -- the result for the block.
     slugAccs :: [(VName, [Imp.TExp Int64])],
     -- | Global memory destination(s) for the final result(s) for this
     -- particular reduction.
@@ -652,10 +674,10 @@
     mkAcc p block_res_arr
       | Prim t <- paramType p,
         shapeRank (segBinOpShape op) == 0 = do
-          block_res_acc <- dPrimS (baseName (paramName p) <> "_block_res_acc") t
-          pure (block_res_acc, [])
+          thread_acc <- dPrimS (baseName (paramName p) <> "_thread_acc") t
+          pure (thread_acc, [])
       -- if this is a non-primitive reduction, the global mem result array will
-      -- double as accumulator.
+      -- double as the per-thread accumulators.
       | otherwise =
           pure (block_res_arr, [sExt64 ltid, sExt64 tblock_id])
 
@@ -674,9 +696,6 @@
 slugShape :: SegBinOpSlug -> Shape
 slugShape = segBinOpShape . slugOp
 
-slugsComm :: [SegBinOpSlug] -> Commutativity
-slugsComm = mconcat . map (segBinOpComm . slugOp)
-
 slugSplitParams :: SegBinOpSlug -> ([LParam GPUMem], [LParam GPUMem])
 slugSplitParams slug = splitAt (length (slugNeutral slug)) $ slugParams slug
 
@@ -689,7 +708,43 @@
 slugCollCopyArrs :: SegBinOpSlug -> [VName]
 slugCollCopyArrs = collCopyArrs . slugInterms
 
+-- | When 'sequentialStageOneBody' performs the block-wide reduction.
+data BlockReduceAt = ReducePerIteration | ReduceAfterLoop
+  deriving (Eq)
+
+-- | Each thread sequentially reduces @q@ elements, with block-wide reductions
+-- as determined by 'BlockReduceAt'. The function maps the iteration number to
+-- the index of the element.
+sequentialStageOneBody ::
+  [SegBinOpSlug] ->
+  DoSegBody ->
+  TV Int64 ->
+  Imp.TExp Int64 ->
+  Imp.TExp Int64 ->
+  BlockReduceAt ->
+  InKernelGen () ->
+  (Imp.TExp Int64 -> Imp.TExp Int64) ->
+  InKernelGen ()
+sequentialStageOneBody slugs body_cont glb_ind_var q n reduce_at doBlockReduce index = do
+  sFor "i" q $ \i -> do
+    glb_ind_var <-- index i
+
+    sWhen (tvExp glb_ind_var .<. n) $
+      sComment "apply map function(s)" $
+        body_cont $ \all_red_res -> do
+          let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res
+
+          forM2_ slugs maps_res $ \slug map_res ->
+            sLoopNest (slugShape slug) $ \vec_is ->
+              applyRedOp slug vec_is map_res
+
+    when (reduce_at == ReducePerIteration) doBlockReduce
+
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+  when (reduce_at == ReduceAfterLoop) doBlockReduce
+
 reductionStageOne ::
+  StageOne ->
   [VName] ->
   Imp.TExp Int64 ->
   Imp.TExp Int64 ->
@@ -699,39 +754,39 @@
   [SegBinOpSlug] ->
   DoSegBody ->
   InKernelGen [Lambda GPUMem]
-reductionStageOne gtids n global_tid q chunk threads_per_segment slugs body_cont = do
+reductionStageOne strategy gtids n global_tid q chunk threads_per_segment slugs body_cont = do
   constants <- kernelConstants <$> askEnv
   let glb_ind_var = mkTV (last gtids)
       ltid = sExt64 $ kernelLocalThreadId constants
 
   dScope Nothing $ scopeOfLParams $ concatMap slugParams slugs
 
-  sComment "ne-initialise the outer (per-block) accumulator(s)" $ do
+  sComment "ne-initialise the per-thread accumulator(s)" $ do
     forM_ slugs $ \slug ->
       forM2_ (slugAccs slug) (slugNeutral slug) $ \(acc, acc_is) ne ->
         sLoopNest (slugShape slug) $ \vec_is ->
           copyDWIMFix acc (acc_is ++ vec_is) ne []
 
   new_lambdas <- mapM (renameLambda . slugLambda) slugs
-  let tblock_size = sExt32 $ kernelBlockSize constants
   let doBlockReduce =
         forM2_ slugs new_lambdas $ \slug new_lambda -> do
           let accs = slugAccs slug
           let params = slugParams slug
+          let tblock_size = sExt32 $ kernelBlockSize constants
           sLoopNest (slugShape slug) $ \vec_is -> do
             let block_red_arrs = slugBlockRedArrs slug
-            sComment "store accs. prims go in lmem; non-prims in params (in global mem)" $
+            sComment "store reduction accs" $
               forM_ (zip3 block_red_arrs accs params) $
                 \(arr, (acc, acc_is), p) ->
                   if isPrimParam p
                     then copyDWIMFix arr [ltid] (Var acc) (acc_is ++ vec_is)
                     else copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)
 
-            sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.
+            sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier.
             blockReduce tblock_size new_lambda block_red_arrs
             sOp $ Imp.Barrier Imp.FenceLocal
 
-            sComment "thread 0 updates per-block acc(s); rest reset to ne" $ do
+            sComment "thread 0 keeps the block result in its accumulator; rest reset to neutral" $ do
               sIf
                 (ltid .==. 0)
                 ( forM2_ accs (lambdaParams new_lambda) $
@@ -743,8 +798,8 @@
                       copyDWIMFix acc (acc_is ++ vec_is) ne []
                 )
 
-  case (slugsComm slugs, all (isPrimSegBinOp . slugOp) slugs) of
-    (Noncommutative, True) ->
+  case strategy of
+    StageOneTiled ->
       noncommPrimParamsStageOneBody
         slugs
         body_cont
@@ -754,74 +809,43 @@
         n
         chunk
         doBlockReduce
-    _ ->
-      generalStageOneBody
-        slugs
-        body_cont
-        glb_ind_var
-        global_tid
-        q
-        n
-        threads_per_segment
-        doBlockReduce
+    StageOneStrided ->
+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $
+        \i -> global_tid + threads_per_segment * i
+    StageOneContiguous ->
+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $
+        \i -> global_tid * q + i
+    StageOneWindow -> do
+      let window_size = kernelBlockSize constants
+      block_base_offset <-
+        dPrimVE "block_base_offset" $ (global_tid `quot` window_size) * q * window_size
+      sequentialStageOneBody slugs body_cont glb_ind_var q n ReducePerIteration doBlockReduce $
+        \i -> block_base_offset + i * window_size + ltid
 
   pure new_lambdas
 
-generalStageOneBody ::
-  [SegBinOpSlug] ->
-  DoSegBody ->
-  TV Int64 ->
-  Imp.TExp Int64 ->
-  Imp.TExp Int64 ->
-  Imp.TExp Int64 ->
-  Imp.TExp Int64 ->
-  InKernelGen () ->
+-- | Combine the accumulators of a reduction with the given next values. The
+-- vector indexes select the elements of a vectorised reduction, and are
+-- appended to the indexes of both.
+applyRedOp ::
+  SegBinOpSlug ->
+  [Imp.TExp Int64] ->
+  [(SubExp, [Imp.TExp Int64])] ->
   InKernelGen ()
-generalStageOneBody slugs body_cont glb_ind_var global_tid q n threads_per_segment doBlockReduce = do
-  let is_comm = slugsComm slugs == Commutative
-
-  constants <- kernelConstants <$> askEnv
-  let tblock_size = kernelBlockSize constants
-      ltid = sExt64 $ kernelLocalThreadId constants
-
-  -- this block's id within its designated segment, and this block's initial
-  -- global offset.
-  tblock_id_in_segment <- dPrimVE "tblock_id_in_segment" $ global_tid `quot` tblock_size
-  block_base_offset <- dPrimVE "block_base_offset" $ tblock_id_in_segment * q * tblock_size
-
-  sFor "i" q $ \i -> do
-    block_offset <- dPrimVE "block_offset" $ block_base_offset + i * tblock_size
-    glb_ind_var
-      <-- if is_comm
-        then global_tid + threads_per_segment * i
-        else block_offset + ltid
-
-    sWhen (tvExp glb_ind_var .<. n) $
-      sComment "apply map function(s)" $
-        body_cont $ \all_red_res -> do
-          let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res
-
-          forM2_ slugs maps_res $ \slug map_res ->
-            sLoopNest (slugShape slug) $ \vec_is -> do
-              let (acc_params, next_params) = slugSplitParams slug
-              sComment "load accumulator(s)" $
-                forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->
-                  copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)
-              sComment "load next value(s)" $
-                forM2_ next_params map_res $ \p (res, res_is) ->
-                  copyDWIMFix (paramName p) [] res (res_is ++ vec_is)
-              sComment "apply reduction operator(s)"
-                $ compileStms mempty (bodyStms $ slugBody slug)
-                $ sComment "store in accumulator(s)"
-                $ forM2_
-                  (slugAccs slug)
-                  (map resSubExp $ bodyResult $ slugBody slug)
-                $ \(acc, acc_is) se ->
-                  copyDWIMFix acc (acc_is ++ vec_is) se []
-
-    unless is_comm doBlockReduce
-  sOp $ Imp.ErrorSync Imp.FenceLocal
-  when is_comm doBlockReduce
+applyRedOp slug vec_is next_values = do
+  let (acc_params, next_params) = slugSplitParams slug
+  sComment "load accumulator(s)" $
+    forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->
+      copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)
+  sComment "load next value(s)" $
+    forM2_ next_params next_values $ \p (v, v_is) ->
+      copyDWIMFix (paramName p) [] v (v_is ++ vec_is)
+  sComment "apply reduction operator(s)" $
+    compileStms mempty (bodyStms $ slugBody slug) $
+      sComment "store in accumulator(s)" $
+        forM2_ (slugAccs slug) (map resSubExp $ bodyResult $ slugBody slug) $
+          \(acc, acc_is) se ->
+            copyDWIMFix acc (acc_is ++ vec_is) se []
 
 noncommPrimParamsStageOneBody ::
   [SegBinOpSlug] ->
@@ -870,7 +894,7 @@
                 copyDWIMFix priv_chunk [k] ne []
         )
 
-    sOp $ Imp.ErrorSync Imp.FenceLocal
+    sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
     sComment "effectualize collective copies in shared memory" $ do
       forM_ slugs $ \slug -> do
@@ -890,58 +914,42 @@
 
           sOp $ Imp.Barrier Imp.FenceLocal
 
-    sComment "per-thread sequential reduction of private chunk(s)" $ do
+    sComment "per-thread sequential reduction of private chunk(s)" $
       chunkLoop $ \k ->
-        forM_ slugs $ \slug -> do
-          let accs = map fst $ slugAccs slug
-          let (acc_ps, next_ps) = slugSplitParams slug
-          let ps_accs_chunks = zip4 acc_ps next_ps accs (slugPrivChunks slug)
-
-          sComment "load params for all reductions" $ do
-            forM_ ps_accs_chunks $ \(acc_p, next_p, acc, priv_chunk) -> do
-              copyDWIM (paramName acc_p) [] (Var acc) []
-              copyDWIMFix (paramName next_p) [] (Var priv_chunk) [k]
-
-          sComment "apply reduction operator(s)" $ do
-            let binop_ress = map resSubExp $ bodyResult $ slugBody slug
-            compileStms mempty (bodyStms $ slugBody slug) $
-              forM2_ accs binop_ress $ \acc binop_res ->
-                copyDWIM acc [] binop_res []
+        forM_ slugs $ \slug ->
+          applyRedOp slug [] [(Var c, [k]) | c <- slugPrivChunks slug]
     doLMemBlockReduce
-  sOp $ Imp.ErrorSync Imp.FenceLocal
+  sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
+-- | The second stage of a reduction where each segment is handled by more than
+-- one block. Each block writes its partial results for all the operators, and
+-- the last block of the segment to do so combines them into the final results.
+-- The blocks find out which of them is last by incrementing a counter for the
+-- segment, which the last block then resets.
 reductionStageTwo ::
-  [PatElem LetDecMem] ->
   Imp.TExp Int32 ->
   [Imp.TExp Int64] ->
   Imp.TExp Int64 ->
   Imp.TExp Int64 ->
-  SegBinOpSlug ->
-  Lambda GPUMem ->
+  [([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem)] ->
   VName ->
   VName ->
   Imp.TExp Int64 ->
   InKernelGen ()
-reductionStageTwo segred_pes tblock_id segment_gtids first_block_for_segment blocks_per_segment slug new_lambda counters sync_arr counter_idx = do
+reductionStageTwo tblock_id segment_gtids first_block_for_segment blocks_per_segment reds counters sync_arr counter_idx = do
   constants <- kernelConstants <$> askEnv
 
   let ltid32 = kernelLocalThreadId constants
-      ltid = sExt64 ltid32
-      tblock_size = kernelBlockSize constants
 
-  let (acc_params, next_params) = slugSplitParams slug
-      nes = slugNeutral slug
-      red_arrs = slugBlockRedArrs slug
-      block_res_arrs = blockResArrs slug
-
   old_counter <- dPrim "old_counter"
   (counter_mem, _, counter_offset) <- fullyIndexArray counters [counter_idx]
-  sComment "first thread in block saves block result to global memory" $
+  sComment "first thread in block saves block results to global memory" $
     sWhen (ltid32 .==. 0) $ do
-      forM_ (take (length nes) $ zip block_res_arrs (slugAccs slug)) $ \(v, (acc, acc_is)) ->
-        writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is
-      -- Increment the counter, thus stating that our result is
-      -- available.
+      forM_ reds $ \(_, slug, _) ->
+        forM_ (take (length (slugNeutral slug)) $ zip (blockResArrs slug) (slugAccs slug)) $
+          \(v, (acc, acc_is)) ->
+            writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is
+      -- Increment the counter, thus declaring that our results are available.
       sOp
         $ Imp.Atomic DefaultSpace
         $ Imp.AtomicAdd
@@ -950,8 +958,8 @@
           counter_mem
           counter_offset
         $ untyped (1 :: Imp.TExp Int32)
-      -- Now check if we were the last block to write our result.  If
-      -- so, it is our responsibility to produce the final result.
+      -- Now check if we were the last block to write our results.  If
+      -- so, it is our responsibility to produce the final results.
       sWrite sync_arr [0] $ untyped $ tvExp old_counter .==. sExt32 (blocks_per_segment - 1)
 
   sOp $ Imp.Barrier Imp.FenceGlobal
@@ -959,7 +967,7 @@
   is_last_block <- dPrim "is_last_block"
   copyDWIMFix (tvVar is_last_block) [] (Var sync_arr) [0]
   sWhen (tvExp is_last_block) $ do
-    -- The final block has written its result (and it was
+    -- The final block has written its results (and it was
     -- us!), so read in all the block results and perform the
     -- final stage of the reduction.  But first, we reset the
     -- counter so it is ready for next time.  This is done
@@ -972,74 +980,93 @@
             untyped $
               sExt32 (negate blocks_per_segment)
 
-    sLoopNest (slugShape slug) $ \vec_is -> do
-      unless (null $ slugShape slug) $
-        sOp (Imp.Barrier Imp.FenceLocal)
+    mapM_ (finalReduction segment_gtids first_block_for_segment blocks_per_segment) reds
 
-      -- There is no guarantee that the number of threadblocks for the
-      -- segment is less than the threadblock size, so each thread may
-      -- have to read multiple elements.  We do this in a sequential
-      -- way that may induce non-coalesced accesses, but the total
-      -- number of accesses should be tiny here.
-      --
-      -- TODO: here we *could* insert a collective copy of the num_tblocks
-      -- per-block results. However, it may not be beneficial, since num_tblocks
-      -- is not necessarily larger than tblock_size, meaning the number of
-      -- uncoalesced reads here may be insignificant. In fact, if we happen to
-      -- have a num_tblocks < tblock_size, then the collective copy would add
-      -- unnecessary overhead. Also, this code is only executed by a single
-      -- block.
-      sComment "read in the per-block-results" $ do
-        read_per_thread <-
-          dPrimVE "read_per_thread" $
-            blocks_per_segment `ceilDiv` sExt64 tblock_size
+-- | Combine the partial results that the blocks of a segment have written for
+-- one operator. Executed by a single block.
+finalReduction ::
+  [Imp.TExp Int64] ->
+  Imp.TExp Int64 ->
+  Imp.TExp Int64 ->
+  ([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem) ->
+  InKernelGen ()
+finalReduction segment_gtids first_block_for_segment blocks_per_segment (segred_pes, slug, new_lambda) = do
+  constants <- kernelConstants <$> askEnv
 
-        forM2_ acc_params nes $ \p ne ->
-          copyDWIM (paramName p) [] ne []
+  let ltid32 = kernelLocalThreadId constants
+      ltid = sExt64 ltid32
+      tblock_size = kernelBlockSize constants
 
-        sFor "i" read_per_thread $ \i -> do
-          block_res_id <-
-            dPrimVE "block_res_id" $
-              ltid * read_per_thread + i
-          index_of_block_res <-
-            dPrimVE "index_of_block_res" $
-              first_block_for_segment + block_res_id
+  let (acc_params, next_params) = slugSplitParams slug
+      nes = slugNeutral slug
+      red_arrs = slugBlockRedArrs slug
+      block_res_arrs = blockResArrs slug
 
-          sWhen (block_res_id .<. blocks_per_segment) $ do
-            forM2_ next_params block_res_arrs $
-              \p block_res_arr ->
-                copyDWIMFix
-                  (paramName p)
-                  []
-                  (Var block_res_arr)
-                  ([0, index_of_block_res] ++ vec_is)
+  sLoopNest (slugShape slug) $ \vec_is -> do
+    unless (null $ slugShape slug) $
+      sOp (Imp.Barrier Imp.FenceLocal)
 
-            compileStms mempty (bodyStms $ slugBody slug) $
-              forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->
-                copyDWIMFix (paramName p) [] se []
+    -- There is no guarantee that the number of threadblocks for the segment is
+    -- less than the threadblock size, so each thread may have to read multiple
+    -- elements. We do this in a sequential way that may induce non-coalesced
+    -- accesses, but the total number of accesses should be tiny here.
+    --
+    -- TODO: here we *could* insert a collective copy of the num_tblocks
+    -- per-block results. However, it may not be beneficial, since num_tblocks
+    -- is not necessarily larger than tblock_size, meaning the number of
+    -- uncoalesced reads here may be insignificant. In fact, if we happen to
+    -- have a num_tblocks < tblock_size, then the collective copy would add
+    -- unnecessary overhead. Also, this code is only executed by a single block.
+    sComment "read in the per-block-results" $ do
+      read_per_thread <-
+        dPrimVE "read_per_thread" $
+          blocks_per_segment `ceilDiv` sExt64 tblock_size
 
-      forM2_ acc_params red_arrs $ \p arr ->
-        when (isPrimParam p) $
-          copyDWIMFix arr [ltid] (Var $ paramName p) []
+      forM2_ acc_params nes $ \p ne ->
+        copyDWIM (paramName p) [] ne []
 
-      sOp $ Imp.ErrorSync Imp.FenceLocal
+      sFor "i" read_per_thread $ \i -> do
+        block_res_id <-
+          dPrimVE "block_res_id" $
+            ltid * read_per_thread + i
+        index_of_block_res <-
+          dPrimVE "index_of_block_res" $
+            first_block_for_segment + block_res_id
 
-      sComment "reduce the per-block results" $ do
-        blockReduce (sExt32 tblock_size) new_lambda red_arrs
+        sWhen (block_res_id .<. blocks_per_segment) $ do
+          forM2_ next_params block_res_arrs $ \p block_res_arr ->
+            copyDWIMFix
+              (paramName p)
+              []
+              (Var block_res_arr)
+              ([0, index_of_block_res] ++ vec_is)
 
-        sComment "and back to memory with the final result" $
-          sWhen (ltid32 .==. 0) $
-            forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->
-              copyDWIMFix
-                (patElemName pe)
-                (segment_gtids ++ vec_is)
-                (Var $ paramName p)
-                []
+          compileStms mempty (bodyStms $ slugBody slug) $
+            forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->
+              copyDWIMFix (paramName p) [] se []
 
+    forM2_ acc_params red_arrs $ \p arr ->
+      when (isPrimParam p) $
+        copyDWIMFix arr [ltid] (Var $ paramName p) []
+
+    sOp $ Imp.ErrorSync Imp.FenceLocal mempty
+
+    sComment "reduce the per-block results" $ do
+      blockReduce (sExt32 tblock_size) new_lambda red_arrs
+
+      sComment "and back to memory with the final result" $
+        sWhen (ltid32 .==. 0) $
+          forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->
+            copyDWIMFix
+              (patElemName pe)
+              (segment_gtids ++ vec_is)
+              (Var $ paramName p)
+              []
+
 -- Note [IntermArrays]
 --
--- Intermediate memory for the nonsegmented and large segments non-commutative
--- reductions with all primitive parameters:
+-- Intermediate memory for the nonsegmented and large segments reductions using
+-- 'StageOneTiled' (see Note [Stage one strategies]):
 --
 --   These kernels need shared memory for 1) the initial collective copy, 2) the
 --   (virtualized) block reductions, and (TODO: this one not implemented yet!)
@@ -1078,9 +1105,63 @@
 --   Aside from shared memory, these kernels also require a CHUNK-sized array of
 --   thread-private register memory per reduction operator.
 --
--- For all other reductions, ie. commutative reductions, reductions with at
--- least one non-primitive operator, and small segments reductions:
+-- For all other reductions, ie. those using 'StageOneStrided',
+-- 'StageOneContiguous', or 'StageOneWindow', and small segments
+-- reductions:
 --
 --   These kernels use shared memory only for the intra-block reductions, and
 --   since they do not use chunking or CHUNK, they all require onlly `tblock_size
 --   * max elem_sizes` bytes of shared memory and no thread-private register mem.
+
+-- Note [Stage one strategies]
+--
+-- In the first stage of a nonsegmented or large-segments reduction, each
+-- thread reduces some of the input elements, after which the per-thread
+-- results are combined by a block-wide reduction.  The strategy decides which
+-- elements a thread reduces.
+--
+-- A commutative operator allows thread @t@ of @T@ to reduce elements @t@,
+-- @t+T@, @t+2T@, and so on ('StageOneStrided'). This gives coalesced memory
+-- accesses and a single block-wide reduction at the end, and we always pick
+-- this strategy for any commutative operator.
+--
+-- A non-commutative operator requires that each thread reduces a contiguous
+-- range of elements.  There are three ways of achieving this:
+--
+-- - 'StageOneContiguous': each thread reduces a contiguous range of @q@
+--   elements, followed by a single block-wide reduction. Order is preserved
+--   because consecutive threads have consecutive ranges, the block-wide
+--   reduction combines the threads in order, and stage two combines the blocks
+--   in order. This is similar to 'StageOneStrided', but results in noncoalesced
+--   reads. However, this strategy can still be a good choice for complex
+--   noncommutative reductions that need fairly little input from global memory,
+--   but have large operands.
+--
+-- - 'StageOneTiled': each block processes a sequence of contiguous tiles of
+--   @tblock_size * chunk@ elements. The map function is applied to the elements
+--   of a tile with coalesced accesses, its results are stored in thread-private
+--   chunks, and the chunks are transposed through shared memory (the collective
+--   copies), such that each thread obtains @chunk@ consecutive elements, which
+--   it reduces sequentially. Since the elements a thread obtains from
+--   consecutive tiles are not adjacent, each tile ends with a block-wide
+--   reduction, whose result thread 0 keeps in its accumulator. The chunk
+--   size amortises the block-wide reductions, but is bounded by the private
+--   storage of @chunk@ copies of the operands (see 'getRedChunkSize'). This
+--   requires non-array operands.
+--
+-- - 'StageOneWindow': each block slides a window of @tblock_size@ elements
+--   across a contiguous range of @q * tblock_size@ elements, with coalesced
+--   accesses and a block-wide reduction per window, whose result is kept in
+--   the accumulator of thread 0.  This is like 'StageOneTiled' with a chunk
+--   size of 1, but needs no private copies of the operands, and so works for
+--   non-primitive operands.  It shares its code with 'StageOneStrided',
+--   except for the indexing and the placement of the block-wide reduction.
+--
+-- 'StageOneTiled' is preferable for cheap operators, where the reduction is
+-- bound by memory bandwidth.  'StageOneContiguous' is preferable for expensive
+-- operators, where the block-wide reductions (which apply the operator several
+-- times per element in each wave, often divergently) dominate.  We have no
+-- good static measure of operator cost, so we use the size of the operands as
+-- a crude proxy: we use 'StageOneTiled' when the operands are primitive and
+-- take up at most 'maxTiledOperandWords' 32-bit words in total.  The threshold
+-- is a heuristic.
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs
@@ -3,8 +3,6 @@
 -- the scan and the chosen abckend.
 module Futhark.CodeGen.ImpGen.GPU.SegScan (compileSegScan) where
 
-import Control.Monad
-import Data.Maybe
 import Futhark.CodeGen.ImpCode.GPU qualified as Imp
 import Futhark.CodeGen.ImpGen hiding (compileProg)
 import Futhark.CodeGen.ImpGen.GPU.Base
@@ -37,17 +35,6 @@
               (concatMap (bodyResult . lambdaBody) lams)
         }
 
-bodyHas :: (Exp GPUMem -> Bool) -> Body GPUMem -> Bool
-bodyHas f = any (f' . stmExp) . bodyStms
-  where
-    f' e
-      | f e = True
-      | otherwise = isNothing $ walkExpM walker e
-    walker =
-      identityWalker
-        { walkOnBody = const $ guard . not . bodyHas f
-        }
-
 canBeSinglePass :: [SegBinOp GPUMem] -> Maybe (SegBinOp GPUMem)
 canBeSinglePass scan_ops =
   if all ok scan_ops
@@ -57,9 +44,7 @@
     ok op =
       segBinOpShape op == mempty
         && all primType (lambdaReturnType (segBinOpLambda op))
-        && not (bodyHas isAssert (lambdaBody (segBinOpLambda op)))
-    isAssert (BasicOp Assert {}) = True
-    isAssert _ = False
+        && not (bodyMayFail (lambdaBody (segBinOpLambda op)))
 
 -- | Compile 'SegScan' instance to host-level code with calls to
 -- various kernels.
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs
@@ -136,14 +136,17 @@
 statusA = 1
 statusP = 2
 
+-- | Scan the look-back window held in the given flag and value arrays, using
+-- the first 32 threads, which form a single warp. Basically a kind of segmented
+-- scan, optimised for only the cases handled by single pass scan (in
+-- particular, no array operands).
 inBlockScanLookback ::
   KernelConstants ->
-  Imp.TExp Int64 ->
   VName ->
   [VName] ->
   Lambda GPUMem ->
   InKernelGen ()
-inBlockScanLookback constants arrs_full_size flag_arr arrs scan_lam = do
+inBlockScanLookback constants flag_arr arrs scan_lam = do
   flg_x :: TV Int8 <- dPrim "flg_x"
   flg_y :: TV Int8 <- dPrim "flg_y"
   let flg_param_x = Param mempty (tvVar flg_x) (MemPrim p_int8)
@@ -155,43 +158,26 @@
   dLParams (lambdaParams scan_lam)
 
   skip_threads <- dPrim "skip_threads"
-  let in_block_thread_active =
-        tvExp skip_threads .<=. in_block_id
-      actual_params = lambdaParams scan_lam
+  let thread_active = tvExp skip_threads .<=. ltid32
       (x_params, y_params) =
-        splitAt (length actual_params `div` 2) actual_params
+        splitAt (length (lambdaParams scan_lam) `div` 2) $ lambdaParams scan_lam
       y_to_x =
-        forM_ (zip x_params y_params) $ \(x, y) ->
-          when (primType (paramType x)) $
-            copyDWIM (paramName x) [] (Var (paramName y)) []
-      y_to_x_flg =
-        copyDWIM (tvVar flg_x) [] (Var (tvVar flg_y)) []
-
-  -- Set initial y values
-  sComment "read input for in-block scan" $ do
-    zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)
-    -- Since the final result is expected to be in x_params, we may
-    -- need to copy it there for the first thread in the block.
-    sWhen (in_block_id .==. 0) $ do
-      y_to_x
-      y_to_x_flg
-
-  when array_scan barrier
-
-  let op_to_x = do
+        forM_ (zip (flg_param_x : x_params) (flg_param_y : y_params)) $ \(x, y) ->
+          copyDWIM (paramName x) [] (Var (paramName y)) []
+      op_to_x =
         sIf
           (flg_y_exp .==. statusP_e .||. flg_y_exp .==. statusX_e)
-          ( do
-              y_to_x_flg
-              y_to_x
-          )
+          y_to_x
           (compileBody' x_params $ lambdaBody scan_lam)
 
-  sComment "in-block scan (hopefully no barriers needed)" $ do
+  sComment "read input for warp scan" $
+    zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)
+
+  sComment "warp scan" $ do
     skip_threads <-- 1
 
-    sWhile (tvExp skip_threads .<. block_size) $ do
-      warpLocal . sWhen in_block_thread_active $ do
+    sWhile (tvExp skip_threads .<. warp_size) $ do
+      warpLocal . sWhen thread_active $ do
         sComment "read operands" $
           zipWithM_
             (readParam (sExt64 $ tvExp skip_threads))
@@ -199,7 +185,7 @@
             (flag_arr : arrs)
         sComment "perform operation" op_to_x
 
-      warpLocal . sWhen in_block_thread_active . sComment "write result" $
+      warpLocal . sWhen thread_active . sComment "write result" $
         sequence_ $
           zipWith3
             writeResult
@@ -210,33 +196,18 @@
       skip_threads <-- tvExp skip_threads * 2
   where
     p_int8 = IntType Int8
-    block_size = 32
-    block_id = ltid32 `quot` block_size
-    in_block_id = ltid32 - block_id * block_size
+    warp_size = 32
     ltid32 = kernelLocalThreadId constants
     ltid = sExt64 ltid32
-    gtid = sExt64 $ kernelGlobalThreadId constants
-    array_scan = not $ all primType $ lambdaReturnType scan_lam
-    barrier
-      | array_scan =
-          sOp $ Imp.Barrier Imp.FenceGlobal
-      | otherwise =
-          sOp $ Imp.Barrier Imp.FenceLocal
 
-    readInitial p arr
-      | primType $ paramType p =
-          copyDWIMFix (paramName p) [] (Var arr) [ltid]
-      | otherwise =
-          copyDWIMFix (paramName p) [] (Var arr) [gtid]
-    readParam behind p arr
-      | primType $ paramType p =
-          copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]
-      | otherwise =
-          copyDWIMFix (paramName p) [] (Var arr) [gtid - behind + arrs_full_size]
+    readInitial p arr =
+      copyDWIMFix (paramName p) [] (Var arr) [ltid]
 
+    readParam behind p arr =
+      copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]
+
     writeResult x y arr = do
-      when (isPrimParam x) $
-        copyDWIMFix arr [ltid] (Var $ paramName x) []
+      copyDWIMFix arr [ltid] (Var $ paramName x) []
       copyDWIM (paramName y) [] (Var $ paramName x) []
 
 -- | Calculate the number of u64 words needed to store n bits
@@ -371,9 +342,6 @@
     tvSize <$> dPrimV "num_virt_blocks" (n `ceilDiv` (tblock_size_e * chunk))
   let num_virt_blocks_e = pe64 num_virt_blocks
 
-  num_virt_threads <-
-    dPrimVE "num_virt_threads" $ num_virt_blocks_e * tblock_size_e
-
   let (gtids, dims) = unzip $ unSegSpace space
       dims' = map pe64 dims
       segmented = length dims' > 1
@@ -392,7 +360,8 @@
           <$> sAllocArray "aggregates" ty (Shape [num_virt_blocks]) (Space "device")
           <*> sAllocArray "incprefixes" ty (Shape [num_virt_blocks]) (Space "device")
 
-  global_id <- genZeroes "global_dynid" 1
+  -- The first counter hands out the dynamic IDs.
+  global_id <- bookkeepingArray Imp.BookkeepingCounters
 
   let attrs' =
         attrs
@@ -555,7 +524,16 @@
 
           sIf (virt_tid .<. n) in_bounds out_of_bounds
 
-      sOp $ Imp.ErrorSync Imp.FenceLocal
+      -- See Note [Failures in single-pass scan].
+      publish_on_failure <- collect . sWhen (ltid32 .==. 0) $ do
+        everythingVolatile $
+          forM_ (zip incprefixArrays scanop_nes) $ \(incprefixArray, ne) ->
+            copyDWIMFix incprefixArray [tvExp dyn_id] ne []
+        sOp global_fence
+        everythingVolatile $
+          copyDWIMFix statusFlags [tvExp dyn_id] (intConst Int8 statusP) []
+      sOp $ Imp.ErrorSync Imp.FenceLocal publish_on_failure
+
       sComment "Transpose scan inputs" $ do
         forM_ (zip transposedArrays scan_private_chunks) $ \(trans, priv) -> do
           sFor "i" chunk $ \i -> do
@@ -611,11 +589,11 @@
         blockScan
           crossesSegment
           tblock_size_e
-          num_virt_threads
+          tblock_size_e
           scan_op1
           prefixArrays
 
-        sOp $ Imp.ErrorSync Imp.FenceLocal
+        sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 
         let firstThread acc prefixes =
               copyDWIMFix (tvVar acc) [] (Var prefixes) [sExt64 tblock_size_e - 1]
@@ -725,7 +703,6 @@
                     lam' <- renameLambda scan_op1
                     inBlockScanLookback
                       constants
-                      num_virt_threads
                       warpscan
                       exchanges
                       lam'
@@ -846,7 +823,7 @@
                 lambdaParams $
                   segPostOpLambda post_op
 
-      sComment "Compute post op and write to global memory." $ do
+      sComment "Compute post op and write to global memory." $
         sFor "i" chunk $ \i -> do
           flat_idx <- dPrimVE "flat_idx" $ thd_offset + i * tblock_size_e
           sWhen (flat_idx .<. n) $ do
@@ -883,5 +860,28 @@
                       then pure ()
                       else copyDWIMFix (patElemName pe) (map le64 gtids) subexp []
 
-      sOp local_barrier
+      -- The block has published its prefix, so it may stop here.
+      sOp $ Imp.ErrorSync Imp.FenceLocal mempty
 {-# NOINLINE compileSegScan #-}
+
+-- Note [Failures in single-pass scan]
+--
+-- Normally, when a thread fails, its whole thread block stops at the next
+-- 'Imp.ErrorSync'. The single-pass scan cannot allow that before the block has
+-- published its prefix, as each block waits in the lookback for the status
+-- flags of the blocks with smaller dynamic IDs, so a block that stops before
+-- publishing its flag leaves every later block waiting forever. Stopping after
+-- publishing is fine, no block waits for the larger dynamic IDs that are then
+-- skipped. The block that fetches the last ID normally resets the dynamic ID
+-- counter, so after a failure the counter may be left nonzero. This is
+-- harmless, as the runtime system resets the bookkeeping buffers when it
+-- reports the failure.
+--
+-- The only failures before publishing are in the segop body (the "map part"),
+-- as 'canBeSinglePass' checks that the scan operator cannot fail. The
+-- 'Imp.ErrorSync' after the map therefore has a handler that publishes the
+-- neutral element as the block's inclusive prefix before the block stops. The
+-- later blocks then compute meaningless results, but as the failure is recorded
+-- globally, the results are never used. Failures in the post op reach the
+-- 'Imp.ErrorSync' at the end of the virtual block, after the prefix has been
+-- published.
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs
@@ -257,7 +257,7 @@
                           to' = sExt64 to + tvExp chunk_offset
                        in f from' to'
 
-              sOp $ Imp.ErrorSync fence
+              sOp $ Imp.ErrorSync fence mempty
 
               -- We need to avoid parameter name clashes.
               scan_op_renamed <- renameLambda scan_op
@@ -386,7 +386,7 @@
               -- reload carry (xParams) from the scan output written by the
               -- previous chunk.  Thread 0 reads the last element of the
               -- previous chunk, unless a segment boundary falls between the
-              -- chunks.
+              -- chunks, or that element is out of bounds.
               when use_global_carry $ do
                 crosses_seg <-
                   dPrimVE "crosses_seg" $
@@ -394,7 +394,11 @@
                       Nothing -> false
                       Just f -> f prev_chunk_last (prev_chunk_last + 1)
                 sIf
-                  (chunk_id .>. 0 .&&. kernelLocalThreadId constants .==. 0 .&&. bNot crosses_seg)
+                  ( (chunk_id .>. 0)
+                      .&&. (kernelLocalThreadId constants .==. 0)
+                      .&&. (prev_chunk_last .<. product dims')
+                      .&&. bNot crosses_seg
+                  )
                   ( do
                       let carry_is = unflattenIndex dims' prev_chunk_last
                       forM_ (zip scan_x_params pes) $ \(p, pe) ->
@@ -425,7 +429,7 @@
                   \(arr, se) ->
                     copyDWIMFix arr [localArrayIndex constants] se []
 
-              sOp $ Imp.ErrorSync fence
+              sOp $ Imp.ErrorSync fence mempty
 
               -- crossesSegment' maps block-local thread IDs to element
               -- indices, adjusting for the current chunk offset.
@@ -575,8 +579,9 @@
                     (Var $ paramName p)
                     []
 
-      sOp $ Imp.Barrier Imp.FenceLocal
-
+      -- Each thread reads only the elements it wrote itself, so no barrier is
+      -- needed, and a failure must be able to reach the 'Imp.ErrorSync' at the
+      -- end of the iteration without passing one.
       if isIdentityPostOp post_op
         then pure ()
         else do
@@ -607,8 +612,6 @@
                   forM_ (zip (patElems pat) res) $ \(pe, subexp) ->
                     copyDWIMFix (patElemName pe) (map Imp.le64 gtids) subexp []
 
-      sOp $ Imp.Barrier Imp.FenceGlobal
-
 isIdentityPostOp :: SegPostOp rep -> Bool
 isIdentityPostOp post_op =
   map resSubExp (bodyResult (lambdaBody lam))
@@ -636,12 +639,15 @@
 
   -- The stage-2 block size is a tunable/user-settable parameter.  It
   -- is independent of stage1_num_tblocks so the autotuner can treat
-  -- it as a fixed knob.
+  -- it as a fixed knob.  It may not exceed the number of stage-1
+  -- threads, because the memory of array-typed operator parameters
+  -- is allocated for that many threads.
   stage2_tblock_size_param <- getSize "segscan_stage2_tblock_size" SizeThreadBlock
   stage2_tblock_size <-
     fmap (Imp.Count . tvSize) $
       dPrimV "stage2_tblock_size" $
-        tvExp stage2_tblock_size_param
+        sMin64 (tvExp stage2_tblock_size_param) $
+          pe64 (unCount (kAttrNumBlocks attrs)) * pe64 (unCount (kAttrBlockSize attrs))
 
   let shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)
       scan_ts = concatMap shpT scans
diff --git a/src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs b/src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs
--- a/src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs
+++ b/src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs
@@ -126,6 +126,7 @@
     getSize (ImpOpenCL.CmpSizeLe _ v _) = Just v
     getSize (ImpOpenCL.GetSizeMax {}) = Nothing
     getSize (ImpOpenCL.LaunchKernel {}) = Nothing
+    getSize (ImpOpenCL.GetBookkeeping {}) = Nothing
     directUseInFun fun = mapMaybe getSize $ toList $ functionBody fun
     direct_uses = map (second directUseInFun) $ unFunctions $ defFuns defs
 
@@ -158,18 +159,11 @@
 data KernelState = KernelState
   { kernelSharedMemory :: [SharedMemoryUse],
     kernelFailures :: [FailureMsg],
-    kernelNextSync :: Int,
-    -- | Has a potential failure occurred sine the last
-    -- ErrorSync?
-    kernelSyncPending :: Bool,
     kernelHasBarriers :: Bool
   }
 
 newKernelState :: [FailureMsg] -> KernelState
-newKernelState failures = KernelState mempty failures 0 False False
-
-errorLabel :: KernelState -> String
-errorLabel = ("error_" ++) . show . kernelNextSync
+newKernelState failures = KernelState mempty failures False
 
 data ToOpenCL = ToOpenCL
   { clGPU :: M.Map KernelName (KernelSafety, T.Text),
@@ -195,6 +189,7 @@
 codeMayFail f (For _ _ x) = codeMayFail f x
 codeMayFail f (While _ x) = codeMayFail f x
 codeMayFail f (If _ x y) = codeMayFail f x || codeMayFail f y
+codeMayFail f (OnFailure x y) = codeMayFail f x || codeMayFail f y
 codeMayFail _ _ = False
 
 hostOpMayFail :: ImpGPU.HostOp -> Bool
@@ -235,6 +230,8 @@
   pure $ ImpOpenCL.CmpSizeLe v key x
 onHostOp _ (ImpGPU.GetSizeMax v size_class) =
   pure $ ImpOpenCL.GetSizeMax v size_class
+onHostOp _ (ImpGPU.GetBookkeeping v b) =
+  pure $ ImpOpenCL.GetBookkeeping v b
 
 genGPUCode ::
   Env ->
@@ -303,7 +300,8 @@
 calledInHostOp _ = mempty
 
 calledInKernelOp :: KernelOp -> S.Set Name
-calledInKernelOp = const mempty
+calledInKernelOp (ErrorSync _ handler) = calledFuncs calledInKernelOp handler
+calledInKernelOp _ = mempty
 
 ensureDeviceFuns :: ImpGPU.KernelCode -> OnKernelM [Name]
 ensureDeviceFuns code = do
@@ -347,6 +345,7 @@
           -- No need to free, as we cannot allocate memory in kernels.
           mapM_ GC.item =<< GC.declAllocatedMem
           mapM_ GC.item body
+          GC.items =<< GC.errorLabel
       kstate = GC.compUserState cstate
 
       (kernel_consts, (const_defs, const_undefs)) =
@@ -359,17 +358,32 @@
   let (use_params, unpack_params) =
         unzip $ mapMaybe useAsParam $ kernelUses kernel
 
-  -- The local_failure variable is an int despite only really storing
-  -- a single bit of information, as some OpenCL implementations
-  -- (e.g. AMD) does not like byte-sized shared memory (and the others
-  -- likely pad to a whole word anyway).
-  let (safety, error_init)
+  -- The local_failure variable is an int despite only really storing a single
+  -- bit of information, as some OpenCL implementations (e.g. AMD) does not like
+  -- byte-sized shared memory (and the others likely pad to a whole word
+  -- anyway). The CUDA/HIP prelude #defines __local to nothing, so we need
+  -- special handling here to get a shared-memory local variable. Ugly...
+  let cuda_local_failure_decl =
+        [C.citem|$escstm:("__shared__ volatile int local_failure;")|]
+      local_failure_decl =
+        case target of
+          TargetOpenCL -> [C.citem|volatile __local int local_failure;|]
+          TargetCUDA -> cuda_local_failure_decl
+          TargetHIP -> cuda_local_failure_decl
+
+      -- No thread may write a failure before all threads have cleared it.
+      local_failure_barrier
+        | kernelHasBarriers kstate = [C.citems|barrier(CLK_LOCAL_MEM_FENCE);|]
+        | otherwise = []
+
+      (safety, error_init)
         -- We conservatively assume that any called function can fail.
         | not $ null called =
             ( SafetyFull,
-              [C.citems|volatile __local int local_failure;
+              [C.citems|$item:local_failure_decl
                         // Harmless for all threads to write this.
-                        local_failure = 0;|]
+                        local_failure = 0;
+                        $items:local_failure_barrier|]
             )
         | length (kernelFailures kstate) == length failures =
             if kernelFailureTolerant kernel
@@ -388,14 +402,14 @@
               else
                 ( SafetyFull,
                   [C.citems|
-                     volatile __local int local_failure;
+                     $item:local_failure_decl
                      if (failure_is_an_option) {
                        int failed = *global_failure >= 0;
                        if (failed) {
                          return;
                        }
                      }
-                     // All threads write this value - it looks like CUDA has a compiler bug otherwise.
+                     // Harmless for all threads to write this.
                      local_failure = 0;
                      barrier(CLK_LOCAL_MEM_FENCE);
                   |]
@@ -457,8 +471,6 @@
                     $items:error_init
                     $items:kernel_body
 
-                    $id:(errorLabel kstate): return;
-
                     $items:const_undefs
                 }|]
   modify $ \s ->
@@ -565,18 +577,6 @@
     useToArg (ScalarUse v pt) = Just $ ValueKArg (LeafExp v pt) pt
     useToArg ConstUse {} = Nothing
 
-nextErrorLabel :: GC.CompilerM KernelOp KernelState String
-nextErrorLabel =
-  errorLabel <$> GC.getUserState
-
-incErrorLabel :: GC.CompilerM KernelOp KernelState ()
-incErrorLabel =
-  GC.modifyUserState $ \s -> s {kernelNextSync = kernelNextSync s + 1}
-
-pendingError :: Bool -> GC.CompilerM KernelOp KernelState ()
-pendingError b =
-  GC.modifyUserState $ \s -> s {kernelSyncPending = b}
-
 hasCommunication :: ImpGPU.KernelCode -> Bool
 hasCommunication = any communicates
   where
@@ -642,16 +642,15 @@
       GC.modifyUserState $ \s ->
         s {kernelSharedMemory = (name', size) : kernelSharedMemory s}
       GC.stm [C.cstm|$id:name = (__local unsigned char*) $id:name';|]
-    kernelOps (ErrorSync f) = do
-      label <- nextErrorLabel
-      pending <- kernelSyncPending <$> GC.getUserState
-      when pending $ do
-        pendingError False
-        GC.stm [C.cstm|$id:label: barrier($exp:(fence f));|]
-        GC.stm [C.cstm|if (local_failure) { return; }|]
+    kernelOps (ErrorSync f handler) = do
+      error_label <- GC.errorLabel
+      unless (null error_label) $ do
+        handler' <- GC.collect $ GC.compileCode handler
+        GC.items error_label
+        GC.stm [C.cstm|barrier($exp:(fence f));|]
+        GC.stm [C.cstm|if (local_failure) { $items:handler' return; }|]
       GC.stm [C.cstm|barrier($exp:(fence f));|]
       GC.modifyUserState $ \s -> s {kernelHasBarriers = True}
-      incErrorLabel
     kernelOps (Atomic space aop) = atomicOps space aop
 
     atomicCast s t = do
@@ -814,16 +813,20 @@
     kernelReadScalar =
       GC.readScalarPointerWithQuals pointerQuals
 
-    whatNext = do
-      label <- nextErrorLabel
-      pendingError True
-      pure $
-        if has_communication
-          then [C.citems|local_failure = 1; goto $id:label;|]
-          else
-            if mode == FunMode
-              then [C.citems|return 1;|]
-              else [C.citems|return;|]
+    -- A failing thread jumps to the next error label, which is either at an
+    -- 'ErrorSync', before the handler of an 'OnFailure', or at the end of the
+    -- kernel. Since 'local_failure' is set before the jump, the handler of an
+    -- 'OnFailure' can propagate the failure with a plain jump. Device functions
+    -- instead return on failure, which would skip such a handler, so they must
+    -- not contain an 'OnFailure'.
+    whatNext
+      | mode == FunMode = pure [C.citems|return 1;|]
+      | has_communication = do
+          failure <- GC.failureJump
+          pure [C.citems|local_failure = 1; $stm:failure|]
+      | otherwise = do
+          failure <- GC.failureJump
+          pure [C.citems|$stm:failure|]
 
     callInKernel dests fname args
       | functionMayFail fname env = do
@@ -897,10 +900,12 @@
 typesInCode (If (TPrimExp e) c1 c2) =
   typesInExp e <> typesInCode c1 <> typesInCode c2
 typesInCode (Assert e _ _) = typesInExp e
+typesInCode (OnFailure body handler) = typesInCode body <> typesInCode handler
 typesInCode (Meta _) = mempty
 typesInCode (DebugPrint _ v) = maybe mempty typesInExp v
 typesInCode (TracePrint msg) = foldMap typesInExp msg
 typesInCode (GetUserParam _ _ def) = typesInExp (untyped def)
+typesInCode (Op (ErrorSync _ handler)) = typesInCode handler
 typesInCode Op {} = mempty
 
 typesInExp :: Exp -> S.Set PrimType
diff --git a/src/Futhark/CodeGen/ImpGen/Multicore/Base.hs b/src/Futhark/CodeGen/ImpGen/Multicore/Base.hs
--- a/src/Futhark/CodeGen/ImpGen/Multicore/Base.hs
+++ b/src/Futhark/CodeGen/ImpGen/Multicore/Base.hs
@@ -38,6 +38,7 @@
 import Futhark.Error
 import Futhark.IR.MCMem
 import Futhark.Transform.Rename
+import Futhark.Util (nubOrd)
 import Prelude hiding (quot, rem)
 
 -- | Is there an atomic t'BinOp' corresponding to this t'BinOp'?
@@ -107,13 +108,15 @@
     _ -> pure $ product $ init ns_64
 
 -- When the SegRed's return value is a scalar
--- we perform a call by value-result in the segop function
+-- we perform a call by value-result in the segop function.  Several
+-- array results may reside in the same memory block (e.g. after
+-- short-circuiting), so the parameters are deduplicated.
 getReturnParams :: Pat LetDecMem -> SegOp () MCMem -> MulticoreGen [Imp.Param]
 getReturnParams pat SegRed {} =
   -- It's a good idea to make sure any prim values are initialised, as
   -- we will load them (redundantly) in the task code, and
   -- uninitialised values are UB.
-  fmap concat . forM (patElems pat) $ \pe -> do
+  fmap (nubOrd . concat) . forM (patElems pat) $ \pe -> do
     case patElemType pe of
       Prim pt -> patElemName pe <~~ ValueExp (blankPrimValue pt)
       _ -> pure ()
@@ -149,6 +152,7 @@
 isLoadBalanced (a Imp.:>>: b) = isLoadBalanced a && isLoadBalanced b
 isLoadBalanced (Imp.For _ _ a) = isLoadBalanced a
 isLoadBalanced (Imp.If _ a b) = isLoadBalanced a && isLoadBalanced b
+isLoadBalanced (Imp.OnFailure a b) = isLoadBalanced a && isLoadBalanced b
 isLoadBalanced Imp.While {} = False
 isLoadBalanced (Imp.Op (Imp.ParLoop _ code _)) = isLoadBalanced code
 isLoadBalanced (Imp.Op (Imp.ForEachActive _ a)) = isLoadBalanced a
@@ -464,8 +468,11 @@
     sUnless (lock_acquired .==. 0) $ do
       dLParams acc_params
       bind_acc_params
-      op_body
-      do_hist
+      -- A thread whose operator fails must still release the lock, or other
+      -- threads would wait for it forever.
+      update <- collect $ op_body >> do_hist
+      on_failure <- collect release_lock
+      emit $ Imp.OnFailure update on_failure
       release_lock
   where
     writeArray bucket arr val = copyDWIMFix arr bucket val []
diff --git a/src/Futhark/CodeGen/RTS/JavaScript.hs b/src/Futhark/CodeGen/RTS/JavaScript.hs
--- a/src/Futhark/CodeGen/RTS/JavaScript.hs
+++ b/src/Futhark/CodeGen/RTS/JavaScript.hs
@@ -4,7 +4,7 @@
 module Futhark.CodeGen.RTS.JavaScript
   ( serverJs,
     valuesJs,
-    wrapperclassesJs,
+    wrappersJs,
   )
 where
 
@@ -19,6 +19,6 @@
 valuesJs :: T.Text
 valuesJs = $(embedStringFile "rts/javascript/values.js")
 
--- | @rts/javascript/wrapperclasses.js@
-wrapperclassesJs :: T.Text
-wrapperclassesJs = $(embedStringFile "rts/javascript/wrapperclasses.js")
+-- | @rts/javascript/wrappers.js@
+wrappersJs :: T.Text
+wrappersJs = $(embedStringFile "rts/javascript/wrappers.js")
diff --git a/src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs b/src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs
--- a/src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs
+++ b/src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs
@@ -1326,8 +1326,12 @@
                in -- It is not safe to short circuit if some other pattern
                   -- element is aliased to this one, as this indicates the
                   -- two pattern elements reference the same physical
-                  -- value somehow.
+                  -- value somehow.  Nor is it safe if the destination is
+                  -- itself bound by the pattern (e.g. the existential memory
+                  -- of a 'Match' or 'Loop'), as then it is not in scope where
+                  -- @b@ is created, although it is in @scope td_env@.
                   if any ((`nameIn` aliasesOf patel) . patElemName) pes
+                    || x_mem `elem` map patElemName pes
                     then failed
                     else case M.lookup b vtab of
                       Nothing ->
diff --git a/src/Futhark/Pass/ExpandAllocations.hs b/src/Futhark/Pass/ExpandAllocations.hs
--- a/src/Futhark/Pass/ExpandAllocations.hs
+++ b/src/Futhark/Pass/ExpandAllocations.hs
@@ -161,7 +161,7 @@
           -- allocations inside the update function.
           lvl = SegThread SegNoVirt Nothing
           (op_lam', lam_allocs) =
-            extractLambdaAllocations (lvl, [0]) bound_outside mempty op_lam
+            extractLambdaAllocations (lvl, [0], Nothing) bound_outside mempty op_lam
           variantAlloc (_, Var v, _) = v `notNameIn` bound_outside
           variantAlloc _ = False
           (variant_allocs, invariant_allocs) = M.partition variantAlloc lam_allocs
@@ -178,7 +178,9 @@
       let num_is = shapeRank shape
           is = map paramName $ take num_is $ lambdaParams op_lam
       (alloc_stms, alloc_offsets) <-
-        genericExpandedInvariantAllocations (const $ const (shape, map le64 is)) invariant_allocs
+        genericExpandedInvariantAllocations
+          (const $ const (map pe64 $ shapeDims shape, map le64 is))
+          invariant_allocs
 
       scope <- askScope
       let scope' = scopeOf op_lam <> scope <> scopeOf alloc_stms
@@ -217,7 +219,7 @@
   ExpandM (Stms GPUMem, (SegLevel, [Lambda GPUMem], KernelBody GPUMem))
 transformScanRed lvl space ops kbody = do
   bound_outside <- asks $ namesFromList . M.keys
-  let user = (lvl, [le64 $ segFlat space])
+  let user = (lvl, [le64 $ segFlat space], Nothing)
       (kbody', kbody_allocs) =
         extractKernelBodyAllocations user bound_outside bound_in_kernel kbody
       (ops', ops_allocs) =
@@ -343,8 +345,9 @@
 type Exp64 = TPrimExp Int64 VName
 
 -- | Identifying the spot where an allocation occurs in terms of its
--- level and unique thread ID.
-type User = (SegLevel, [Exp64])
+-- level and unique thread ID. For a 'SegThreadInBlock' user, also the
+-- size of its index space, if that is bound outside the kernel.
+type User = (SegLevel, [Exp64], Maybe Exp64)
 
 -- | A description of allocations that have been extracted, and how
 -- much memory (and which space) is needed.
@@ -404,7 +407,7 @@
    in (set_stms (stmsFromList stms) body, allocs)
 
 expandable :: User -> Space -> Bool
-expandable (SegBlock {}, _) (Space "shared") = False
+expandable (SegBlock {}, _, _) (Space "shared") = False
 expandable _ ScalarSpace {} = False
 expandable _ _ = True
 
@@ -447,13 +450,22 @@
       tell allocs
       pure body'
 
-    onOp (_, user_ids) (Inner (SegOp op)) =
+    onOp (_, user_ids, _) (Inner (SegOp op)) =
       Inner . SegOp <$> mapSegOpM (opMapper user'') op
       where
+        dims = segSpaceDims $ segSpace op
         user'' =
-          (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))])
+          (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))], extent)
+        extent
+          | SegThreadInBlock {} <- segLevel op,
+            all boundOutside dims =
+              Just $ product $ map pe64 dims
+          | otherwise = Nothing
     onOp _ op = pure op
 
+    boundOutside (Var v) = v `nameIn` bound_outside
+    boundOutside Constant {} = True
+
     opMapper user' =
       identitySegOpMapper
         { mapOnSegBinOpLambda = onLambda user',
@@ -472,7 +484,7 @@
       pure lam {lambdaBody = body}
 
 genericExpandedInvariantAllocations ::
-  (User -> Space -> (Shape, [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap)
+  (User -> Space -> ([Exp64], [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap)
 genericExpandedInvariantAllocations getNumUsers invariant_allocs = do
   -- We expand the invariant allocations by adding an inner dimension
   -- equal to the number of kernel threads.
@@ -485,22 +497,20 @@
           allocpat = Pat [PatElem mem $ MemMem space]
       total_size <-
         letExp "total_size" <=< toExp . product $
-          pe64 per_thread_size : map pe64 (shapeDims num_users)
+          pe64 per_thread_size : num_users
       letBind allocpat $ Op $ Alloc (Var total_size) space
       pure $ M.singleton mem $ newBase user space
 
     newBaseThread user space _old_shape =
-      let (users_shape, user_ids) = getNumUsers user space
-          dims = map pe64 (shapeDims users_shape)
+      let (dims, user_ids) = getNumUsers user space
        in ( flattenIndex dims user_ids,
             product dims
           )
 
-    newBase user@(SegThreadInBlock {}, _) space = newBaseThread user space
-    newBase user@(SegThread {}, _) space = newBaseThread user space
-    newBase user@(SegBlock {}, _) space = \_old_shape ->
-      let (users_shape, user_ids) = getNumUsers user space
-          dims = map pe64 (shapeDims users_shape)
+    newBase user@(SegThreadInBlock {}, _, _) space = newBaseThread user space
+    newBase user@(SegThread {}, _, _) space = newBaseThread user space
+    newBase user@(SegBlock {}, _, _) space = \_old_shape ->
+      let (dims, user_ids) = getNumUsers user space
        in ( flattenIndex dims user_ids,
             product dims
           )
@@ -514,16 +524,20 @@
 expandedInvariantAllocations num_threads (Count num_tblocks) (Count tblock_size) =
   genericExpandedInvariantAllocations getNumUsers
   where
-    getNumUsers (SegThread {}, [gtid]) _ = (Shape [num_threads], [gtid])
-    getNumUsers (SegThread {}, [gid, ltid]) _ = (Shape [num_tblocks, tblock_size], [gid, ltid])
-    getNumUsers (SegThreadInBlock {}, [gtid]) _ = (Shape [num_threads], [gtid])
-    getNumUsers (SegThreadInBlock {}, [_gid, ltid]) (Space "shared") =
-      (Shape [tblock_size], [ltid])
-    getNumUsers (SegThreadInBlock {}, [gid, ltid]) (Space "device") =
-      (Shape [num_tblocks, tblock_size], [gid, ltid])
-    getNumUsers (SegBlock {}, [gid]) _ = (Shape [num_tblocks], [gid])
+    getNumUsers (SegThread {}, [gtid], _) _ = ([pe64 num_threads], [gtid])
+    getNumUsers (SegThread {}, [gid, ltid], _) _ =
+      ([pe64 num_tblocks, pe64 tblock_size], [gid, ltid])
+    getNumUsers (SegThreadInBlock {}, [gtid], _) _ = ([pe64 num_threads], [gtid])
+    getNumUsers (SegThreadInBlock {}, [_gid, ltid], extent) (Space "shared") =
+      ([inBlockUsers extent], [ltid])
+    getNumUsers (SegThreadInBlock {}, [gid, ltid], extent) (Space "device") =
+      ([pe64 num_tblocks, inBlockUsers extent], [gid, ltid])
+    getNumUsers (SegBlock {}, [gid], _) _ = ([pe64 num_tblocks], [gid])
     getNumUsers user space = error $ "getNumUsers: unhandled " ++ show (user, space)
 
+    -- The number of threads in a block that use an in-block allocation.
+    inBlockUsers = maybe (pe64 tblock_size) (sMin64 (pe64 tblock_size))
+
 expandedVariantAllocations ::
   SubExp ->
   SegSpace ->
@@ -770,9 +784,9 @@
 
 offsetMemoryInLambda :: RebaseMap -> Lambda GPUMem -> OffsetM (Lambda GPUMem)
 offsetMemoryInLambda offsets lam = do
-  body <- inScopeOf lam $ offsetMemoryInBody offsets $ lambdaBody lam
-  let params = map (offsetMemoryInParam offsets) $ lambdaParams lam
-  pure $ lam {lambdaBody = body, lambdaParams = params}
+  let lam' = lam {lambdaParams = map (offsetMemoryInParam offsets) (lambdaParams lam)}
+  body <- inScopeOf lam' $ offsetMemoryInBody offsets $ lambdaBody lam'
+  pure $ lam' {lambdaBody = body}
 
 -- A loop may have memory parameters, and those memory blocks may
 -- be expanded.  We assume (but do not check - FIXME) that if the
diff --git a/src/Futhark/Pass/ExtractMulticore.hs b/src/Futhark/Pass/ExtractMulticore.hs
--- a/src/Futhark/Pass/ExtractMulticore.hs
+++ b/src/Futhark/Pass/ExtractMulticore.hs
@@ -123,6 +123,9 @@
   where
     transformInput (shape, arrs, op) =
       (shape,arrs,) <$> traverse (bitraverse transformLambda pure) op
+transformStm stm@(Let _ aux (Op _))
+  | "sequential" `inAttrs` stmAuxAttrs aux =
+      pure $ oneStm $ runIdentity $ rephraseStm toMC stm
 transformStm (Let pat aux (Op op)) =
   fmap (certify (stmAuxCerts aux)) <$> transformSOAC pat (stmAuxAttrs aux) op
 
@@ -145,8 +148,10 @@
 
 sequentialiseBody :: Body SOACS -> ExtractM (Body MC)
 sequentialiseBody = pure . runIdentity . rephraseBody toMC
-  where
-    toMC = injectSOACS OtherOp
+
+-- | Keep SOACs as they are, to be compiled to sequential code.
+toMC :: Rephraser Identity SOACS MC
+toMC = injectSOACS OtherOp
 
 transformFunDef :: FunDef SOACS -> ExtractM (FunDef MC)
 transformFunDef (FunDef entry attrs name rettype params body) = do
diff --git a/src/Futhark/Pass/Flatten.hs b/src/Futhark/Pass/Flatten.hs
--- a/src/Futhark/Pass/Flatten.hs
+++ b/src/Futhark/Pass/Flatten.hs
@@ -170,7 +170,9 @@
           (\i pe -> DistResult (ResTag i) (DistType [] (Rank 0) (patElemType pe)) (patElemName pe))
           [0 ..]
           (patElems nested_pat)
-  env <- flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form)
+  env <-
+    freshenSOAC res $
+      flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form)
   forM_ (zip (patNames pat) res) $ \(pat_v, r) ->
     case resVar (distResTag r) env of
       Regular v ->
@@ -321,7 +323,8 @@
           ~[pe] = patElems pat
       flattenBasicOp ops segments env (inps, res', pe, aux, op)
     Op (Screma w arrs form) ->
-      flattenScrema ops segments env inps res (pat, aux) (w, arrs, form)
+      freshenSOAC res $
+        flattenScrema ops segments env inps res (pat, aux) (w, arrs, form)
     Match scrutinees cases defaultCase rt ->
       flattenMatch ops segments env inps res aux scrutinees cases defaultCase rt
     Apply name args rettype s ->
@@ -333,9 +336,11 @@
     WithAcc inputs lam ->
       flattenWithAcc ops segments env inps res pat aux inputs lam
     Op (Hist w hist_inputs hist_ops bucket_fun) ->
-      flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun)
+      freshenSOAC res $
+        flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun)
     Op (FlatMap w arrs lam) ->
-      flattenFlatMapNested ops segments env inps res aux w arrs lam
+      freshenSOAC res $
+        flattenFlatMapNested ops segments env inps res aux w arrs lam
     Op (Stream {}) -> error "transformDistStm: Stream should have been removed"
     Op (JVP {}) -> error "Unhandled JVP"
     Op (VJP {}) -> error "Unhandled VJP"
@@ -378,6 +383,39 @@
   env' <- foldM (flattenDistStm ops segments) env dstms
   concat <$> zipWithM (liftRegResult lvl segments w inputs env') rettype result
 
+-- | Flatten a SOAC, copying those of its results that would otherwise alias
+-- something. This is because a SOAC result is fresh, and so must be the results
+-- of applying flattening to the SOAC, which is not the case if the body is e.g.
+-- a slice or a transpose.
+freshenSOAC :: [DistResult] -> FlattenM DistEnv -> FlattenM DistEnv
+freshenSOAC res m = do
+  (env, stms) <- collectStms m
+  addStms stms
+  let vs = map (repArray . (`resVar` env) . distResTag) res
+      -- Transitive aliases of each result, including names bound in 'stms'.
+      als = bodyAliases $ analyseBody mempty $ Body () stms (varsRes vs)
+      bound = boundByStms stms
+  snd <$> foldM (freshen bound) (mempty, env) (zip3 res vs als)
+  where
+    -- Only the data array of an irregular representation can be consumed.
+    repArray (Regular v) = v
+    repArray (Irregular irreg) = irregularD irreg
+
+    freshen bound (taken, env) (r, v, v_als) = do
+      v_t <- lookupType v
+      let v_all = oneName v <> v_als
+          aliases_outside = v_all `namesSubtract` bound /= mempty
+      case v_t of
+        Array {}
+          | aliases_outside || v_all `namesIntersect` taken -> do
+              v' <- letExp (baseName v <> "_fresh") $ BasicOp $ Replicate mempty $ Var v
+              env' <- insertRepM (distResTag r) (replaceArray v' $ resVar (distResTag r) env) env
+              pure (taken, env')
+        _ -> pure (taken <> v_all, env)
+
+    replaceArray v' (Regular _) = Regular v'
+    replaceArray v' (Irregular irreg) = Irregular irreg {irregularD = v'}
+
 -- | A lifted function must return fresh, non-aliasing arrays (as it
 -- corresponds to 'map f'; see 'liftRetType').  This is not
 -- automatically the case: a result may alias a parameter (when a value
@@ -570,11 +608,6 @@
   pure $ Lambda params ret body'
 
 transformStm :: Attrs -> FunHasParallelism -> FunSizeParams -> Stm SOACS -> FlattenM ()
-transformStm attrs funHasParallelism funSizeParams (Let pat aux (Op soac))
-  | "sequential_outer" `inAttrs` stmAuxAttrs aux = do
-      scope <- askScope
-      stms <- runBuilderT_ (FOT.transformSOAC pat soac) (castScope scope)
-      transformStms attrs funHasParallelism funSizeParams $ fmap (certify (stmAuxCerts aux)) stms
 transformStm _ _ _ stm
   | "sequential" `inAttrs` stmAuxAttrs (stmAux stm) = addStm $ soacsStmToGPU stm
 transformStm _ _ _ (Let pat aux (Op (Hist w arrs ops bucket_fun))) =
diff --git a/src/Futhark/Pass/Flatten/Builtins.hs b/src/Futhark/Pass/Flatten/Builtins.hs
--- a/src/Futhark/Pass/Flatten/Builtins.hs
+++ b/src/Futhark/Pass/Flatten/Builtins.hs
@@ -4,6 +4,7 @@
   ( BuiltinFn (..),
     builtinFunDef,
     determineReduceOp,
+    getGridSize,
     genUniformSegHist,
     mkSegSpace,
     segMap,
@@ -62,24 +63,17 @@
 
 data ThreadRecommendation = ManyThreads | NoRecommendation SegVirt
 
-numberOfBlocks ::
+-- | Compute a sensible grid size given this many (virtual) threads.
+getGridSize ::
   (MonadBuilder m, Op (Rep m) ~ HostOp inner (Rep m)) =>
   Name ->
   SubExp ->
-  SubExp ->
-  m (SubExp, SubExp)
-numberOfBlocks desc w tblock_size = do
+  Count BlockSize SubExp ->
+  m (Count NumBlocks SubExp)
+getGridSize desc num_threads tblock_size = do
   max_num_tblocks_key <- nameFromText . prettyText <$> newVName (desc <> "_num_tblocks")
-  num_tblocks <-
-    letSubExp "num_tblocks" $
-      Op $
-        SizeOp $
-          CalcNumBlocks w max_num_tblocks_key tblock_size
-  num_threads <-
-    letSubExp "num_threads" $
-      BasicOp $
-        BinOp (Mul Int64 OverflowUndef) num_tblocks tblock_size
-  pure (num_tblocks, num_threads)
+  fmap Count . letSubExp "num_tblocks" . Op . SizeOp $
+    CalcNumBlocks num_threads max_num_tblocks_key (unCount tblock_size)
 
 -- | Like 'segThread', but cap the thread count to the input size.
 -- This is more efficient for small kernels, e.g. summing a small
@@ -104,8 +98,8 @@
       let grid = KernelGrid (Count usable_groups) (Count tblock_size)
       pure $ SegThread SegNoVirt (Just grid)
     NoRecommendation v -> do
-      (num_tblocks, _) <- numberOfBlocks desc w tblock_size
-      let grid = KernelGrid (Count num_tblocks) (Count tblock_size)
+      num_tblocks <- getGridSize desc w $ Count tblock_size
+      let grid = KernelGrid num_tblocks (Count tblock_size)
       pure $ SegThread v (Just grid)
 
 -- FIXME: We use segThreadCapped here because otherwise we may get
diff --git a/src/Futhark/Pass/Flatten/Incremental.hs b/src/Futhark/Pass/Flatten/Incremental.hs
--- a/src/Futhark/Pass/Flatten/Incremental.hs
+++ b/src/Futhark/Pass/Flatten/Incremental.hs
@@ -55,7 +55,6 @@
 
     -- * Levels
     defaultSegLevel,
-    inBlockSegLevel,
     allowVersioning,
 
     -- * Various queries
@@ -84,9 +83,6 @@
 defaultSegLevel :: SegLevel
 defaultSegLevel = SegThread SegVirt Nothing
 
-inBlockSegLevel :: SegLevel
-inBlockSegLevel = SegThreadInBlock SegNoVirt
-
 allowVersioning :: SegLevel -> Bool
 allowVersioning SegThreadInBlock {} = False
 allowVersioning _ = True
@@ -191,18 +187,30 @@
 onlyExploitIntra attrs =
   "only_intra" `inAttrs` flatteningAttrs attrs
 
+onlyExploitOuter :: Attrs -> Bool
+onlyExploitOuter attrs =
+  "only_outer" `inAttrs` flatteningAttrs attrs
+
 mayExploitOuter :: Attrs -> Bool
 mayExploitOuter attrs =
-  not $ "no_outer" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'
+  not $ "no_outer" `inAttrs` attrs' || "only_full" `inAttrs` attrs'
   where
     attrs' = flatteningAttrs attrs
 
 mayExploitIntra :: Attrs -> Bool
 mayExploitIntra attrs =
-  not $ "no_intra" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'
+  not $ "no_intra" `inAttrs` attrs' || "only_full" `inAttrs` attrs'
   where
     attrs' = flatteningAttrs attrs
 
+-- | Is this one of the flattening attributes that select which versions are
+-- generated? These are the ones an enclosing @only_full@ overrides.
+isVersioningAttr :: Attr -> Bool
+isVersioningAttr (AttrComp f [x]) =
+  f `elem` ["flattening", "incremental_flattening"]
+    && x `elem` ["no_outer", "no_intra", "only_intra", "only_outer", "only_full"]
+isVersioningAttr _ = False
+
 intraBlockAlternative ::
   Intrablock.IntrablockResult ->
   FlattenM (SubExp, Body GPU)
@@ -215,7 +223,7 @@
       BasicOp $
         CmpOp
           (CmpSle Int64)
-          (Intrablock.intraThreadBlockSize intra)
+          (unCount (Intrablock.intraThreadBlockSize intra))
           max_tblock_size
   (intra_suff, _) <-
     sufficientParallelism
@@ -263,7 +271,7 @@
     _
       | parallel_fun_inside ->
           full []
-      | "sequential_inner" `inAttrs` attrs -> do
+      | onlyExploitOuter attrs -> do
           outer_body <- mkOuterBody
           kernelAlternatives desc result_ts outer_body []
     Nothing
@@ -317,7 +325,7 @@
 scanRedAlternatives desc result_ts attrs parallel_fun_inside allow_versioning ws mkFullBody mkOuterBody
   | parallel_fun_inside =
       fullAlternative
-  | "sequential_inner" `inAttrs` attrs =
+  | onlyExploitOuter attrs =
       outerAlternative
   | mayExploitOuter attrs && allow_versioning =
       fullWithOuterAlternative
@@ -375,16 +383,14 @@
           0
       where
         attrs = stmAuxAttrs $ stmAux stm
-        sequential_inner = "sequential_inner" `inAttrs` attrs
 
         zeroIfTooSmall (Constant (IntValue x))
           | intToInt64 x < 32 = 0
         zeroIfTooSmall _ = 1
 
-        mapLike w lam' =
-          if sequential_inner
-            then 0
-            else max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam'))
+        mapLike w lam'
+          | onlyExploitOuter attrs = 0
+          | otherwise = max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam'))
 
 -- | A lambda is worth sequentialising if it contains enough nested parallelism
 -- of an interesting kind, or if distributing it would fragment sequential
@@ -404,7 +410,7 @@
           0 :: Int
       | Op (Screma _ _ form@(ScremaForm lam' _ _ _)) <- stmExp stm,
         isJust $ isMapSOAC form =
-          if sequential_inner
+          if onlyExploitOuter attrs
             then 0
             else bodyInterest (depth + 1) (lambdaBody lam')
       | Loop _ _ body <- stmExp stm =
@@ -432,7 +438,6 @@
           0
       where
         attrs = stmAuxAttrs $ stmAux stm
-        sequential_inner = "sequential_inner" `inAttrs` attrs
 
 bodyHasParallelism :: FunHasParallelism -> Body SOACS -> Bool
 bodyHasParallelism funHasParallelism =
@@ -521,15 +526,17 @@
           <$> preprocessStms scope stms
       pure $ mkBody stms' $ varsRes $ patNames pat
 
--- | Add the flattening attributes of the enclosing context to a statement. A
--- statement that carries flattening attributes of its own is left alone, as
--- those are more specific.
+-- | Add the flattening attributes of the enclosing context to a statement, and
+-- to the statements nested in its sequential control flow (see Note [Flattening
+-- attribute propagation]).
 propagateAttrs :: Attrs -> Stm SOACS -> Stm SOACS
 propagateAttrs attrs stm
   | attrs' == mempty = stm
-  | flatteningAttrs (stmAuxAttrs (stmAux stm)) == mempty =
-      stm {stmAux = (stmAux stm) {stmAuxAttrs = attrs' <> stmAuxAttrs (stmAux stm)}}
-  | otherwise = stm
+  | otherwise =
+      stm
+        { stmAux = (stmAux stm) {stmAuxAttrs = stm_attrs},
+          stmExp = propagateInExp stm_attrs $ stmExp stm
+        }
   where
     -- 'flatteningAttrs' strips the enclosing 'flattening', which has to be put
     -- back for the attributes to be recognised on the statement.
@@ -537,17 +544,44 @@
       mconcat . mapAttrs (oneAttr . AttrComp "flattening" . pure) $
         flatteningAttrs attrs
 
+    own = stmAuxAttrs $ stmAux stm
+
+    stm_attrs
+      | "only_full" `inAttrs` flatteningAttrs attrs =
+          attrs' <> mconcat (mapAttrs keepNonVersioning own)
+      | flatteningAttrs own == mempty =
+          attrs' <> own
+      | otherwise =
+          own
+
+    keepNonVersioning attr
+      | isVersioningAttr attr = mempty
+      | otherwise = oneAttr attr
+
+-- | Propagate attributes into the bodies of sequential control flow, but not
+-- into SOAC lambdas.
+propagateInExp :: Attrs -> Exp SOACS -> Exp SOACS
+propagateInExp attrs (Loop merge form body) =
+  Loop merge form $ propagateInBody attrs body
+propagateInExp attrs (Match ses cases defbody dec) =
+  Match ses (map (fmap $ propagateInBody attrs) cases) (propagateInBody attrs defbody) dec
+propagateInExp attrs (WithAcc inputs lam) =
+  WithAcc inputs lam {lambdaBody = propagateInBody attrs $ lambdaBody lam}
+propagateInExp _ e = e
+
+propagateInBody :: Attrs -> Body SOACS -> Body SOACS
+propagateInBody attrs body =
+  body {bodyStms = fmap (propagateAttrs attrs) (bodyStms body)}
+
 -- | Impose outside flattening attributes on a statement. Only SOACs are
--- affected, and only those that carry no flattening attributes of their own, as
--- those are more specific.
+-- affected, following the rules of 'propagateAttrs'.
 imposeAttrs :: Attrs -> Stm SOACS -> Stm SOACS
 imposeAttrs attrs stm
   | Op {} <- stmExp stm = propagateAttrs attrs stm
   | otherwise = stm
 
--- | Propagate incremental flattening attributes to the statements of
--- a map lambda body. Statements that carry their own incremental
--- flattening attributes are left alone.
+-- | Propagate flattening attributes to the statements of a map lambda body,
+-- following the rules of 'propagateAttrs'.
 propagateVersioningAttrs :: Attrs -> Lambda SOACS -> Lambda SOACS
 propagateVersioningAttrs attrs lam
   | flatteningAttrs attrs == mempty = lam
@@ -558,3 +592,15 @@
               { bodyStms = fmap (propagateAttrs attrs) (bodyStms (lambdaBody lam))
               }
         }
+
+-- Note [Flattening attribute propagation]
+--
+-- A flattening attribute on a SOAC applies to every SOAC nested inside it,
+-- including inside loops, branches and with_acc. A nested statement keeps its
+-- own flattening attributes, except that an enclosing @only_full@ replaces its
+-- versioning attributes ('isVersioningAttr').
+--
+-- Attributes are copied onto a SOAC's body only once it has been preprocessed
+-- ('propagateVersioningAttrs', 'factorScremaForParallelism'), as preprocessing
+-- may rewrite statements and lose them. Nested SOAC lambdas are reached when
+-- their own SOAC is flattened.
diff --git a/src/Futhark/Pass/Flatten/Intrablock.hs b/src/Futhark/Pass/Flatten/Intrablock.hs
--- a/src/Futhark/Pass/Flatten/Intrablock.hs
+++ b/src/Futhark/Pass/Flatten/Intrablock.hs
@@ -10,8 +10,10 @@
 
 import Control.Monad
 import Control.Monad.RWS
-import Control.Monad.Writer
+import Control.Monad.State
+import Data.Bitraversable
 import Data.Map qualified as M
+import Data.Maybe
 import Data.Set qualified as S
 import Futhark.Analysis.PrimExp.Convert
 import Futhark.IR.GPU hiding (HistOp)
@@ -21,7 +23,6 @@
 import Futhark.Pass.Flatten.General
 import Futhark.Pass.Flatten.PreProcess (preprocessLambda)
 import Futhark.Tools
-import Futhark.Transform.FirstOrderTransform qualified as FOT
 import Futhark.Transform.Rename
 import Futhark.Transform.ToGPU
 import Prelude hiding (log)
@@ -34,13 +35,14 @@
 data IntrablockResult = IntrablockResult
   { intraMinPar :: SubExp,
     intraAvailPar :: SubExp,
-    intraThreadBlockSize :: SubExp,
+    intraThreadBlockSize :: Count BlockSize SubExp,
     intraPreludeStms :: Stms GPU,
     intraKernelStms :: Stms GPU,
     intraResultNames :: [VName]
   }
 
 type InBlockMapTransformer =
+  SegLevel ->
   Pat Type ->
   SubExp ->
   [VName] ->
@@ -51,9 +53,10 @@
 foldBinOp' _ [] = eSubExp $ intConst Int64 1
 foldBinOp' bop (x : xs) = foldBinOp bop x xs
 
--- | Extract the intra-block parallelism used by the body.
-findParallelism :: KernelBody GPU -> [[SubExp]]
-findParallelism = S.toList . execWriter . onKernelBody
+-- | Extract the intra-block parallelism used by the body, paired with whether
+-- it is virtualised.
+findParallelism :: KernelBody GPU -> [([SubExp], SegVirt)]
+findParallelism = S.toList . flip execState mempty . onKernelBody
   where
     onKernelBody = mapM_ onStm . bodyStms
     onBody = mapM_ onStm . bodyStms
@@ -64,37 +67,39 @@
           walkOnOp = onOp
         }
     onOp (SegOp op) = do
-      tell $ S.singleton $ segSpaceDims $ segSpace op
+      modify $ S.insert (segSpaceDims $ segSpace op, segVirt (segLevel op))
       onKernelBody $ segBody op
     onOp _ = pure ()
 
-computeThreadBlockSize :: [[SubExp]] -> [[SubExp]] -> FlattenM (SubExp, SubExp)
-computeThreadBlockSize wss_min wss_avail = do
-  ws_min <-
-    mapM (letSubExp "one_intra_par_min" <=< foldBinOp' (Mul Int64 OverflowUndef)) $
-      filter (not . null) wss_min
-  ws_avail <-
-    mapM (letSubExp "one_intra_par_avail" <=< foldBinOp' (Mul Int64 OverflowUndef)) $
-      filter (not . null) wss_avail
+onlyNonVirt :: [(a, SegVirt)] -> [a]
+onlyNonVirt = mapMaybe f
+  where
+    f (_, SegVirt) = Nothing
+    f (ws, _) = Just ws
 
-  -- The amount of parallelism available *in the worst case* is
-  -- equal to the smallest parallel loop, or *at least* 1.
+computeThreadBlockSize ::
+  [([SubExp], SegVirt)] ->
+  FlattenM (SubExp, Count BlockSize SubExp)
+computeThreadBlockSize wss = do
+  let prod = letSubExp "one_intra_par" <=< foldBinOp' (Mul Int64 OverflowUndef)
+  ws <- mapM (bitraverse prod pure) $ filter (not . null . fst) wss
+
+  -- The amount of parallelism available *in the worst case* is equal to the
+  -- smallest parallel loop.
   intra_avail_par <-
-    letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) ws_avail
+    letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) (map fst ws)
 
-  tblock_size <- newVName "computed_tblock_size"
-  -- The group size is either the maximum of the minimum parallelism
-  -- exploited, or the desired parallelism (bounded by the max group
-  -- size) in case there is no minimum.
-  letBindNames [tblock_size]
-    =<< if null ws_min
-      then
-        eBinOp
-          (SMin Int64)
-          (eSubExp =<< letSubExp "max_tblock_size" (Op $ SizeOp $ GetSizeMax SizeThreadBlock))
-          (eSubExp intra_avail_par)
-      else foldBinOp' (SMax Int64) ws_min
-  pure (intra_avail_par, Var tblock_size)
+  -- If we have purely virtualised SegOps, then the thread block size can be set
+  -- freely, but otherwise it must fit all the non-virtualised SegOps. Note that
+  -- in the virtualised case, we do not cap the block size by the actual
+  -- exploitable parallelism. This is to ensure that the block size is
+  -- statically known when the kernel is compiled, which can result in much
+  -- better code generation.
+  tblock_size <-
+    case onlyNonVirt ws of
+      [] -> getSize "tblock_size" SizeThreadBlock
+      ws' -> letSubExp "tblock_size" =<< foldBinOp' (SMax Int64) (intConst Int64 1 : ws')
+  pure (intra_avail_par, Count tblock_size)
 
 -- | Check whether this result is actually something that is acceptable to use:
 -- the parallel dimensions must all be bound outside the kernel, as nonuniform
@@ -155,13 +160,18 @@
   let new_segments = segments <> pure w
       wss = findParallelism kbody
 
-  noNonuniformPar (freeIn wss) $ do
+  noNonuniformPar (freeIn $ map fst wss) $ do
     ((intra_avail_par, tblock_size, kspace, num_tblocks), prelude_stms) <-
       collectStms $ do
-        num_tblocks <-
-          letSubExp "intra_num_tblocks"
-            =<< foldBinOp' (Mul Int64 OverflowUndef) new_segments
-        (intra_avail_par, tblock_size) <- computeThreadBlockSize wss wss
+        (intra_avail_par, tblock_size) <-
+          computeThreadBlockSize wss
+        -- CalcNumBlocks needs a "thread count" to cap the actual number of
+        -- blocks. Tell a white lie here that is good enough, hopefully.
+        num_threads <-
+          letSubExp "num_threads" <=< toExp $
+            product $
+              pe64 (unCount tblock_size) : map pe64 new_segments
+        num_tblocks <- getGridSize "intrablock" num_threads tblock_size
         gtids <- mapM (const $ newVName "gtid") new_segments
         kspace <- mkSegSpace $ zip gtids new_segments
         pure (intra_avail_par, tblock_size, kspace, num_tblocks)
@@ -176,8 +186,8 @@
 
     let kbody' = kbody {bodyStms = read_input_stms <> bodyStms kbody}
         rts = map (length new_segments `stripArray`) result_ts
-        grid = KernelGrid (Count num_tblocks) (Count tblock_size)
-        lvl = SegBlock SegNoVirt (Just grid)
+        grid = KernelGrid num_tblocks tblock_size
+        lvl = SegBlock SegVirt (Just grid)
         kstm = Let nested_pat aux $ Op $ SegOp $ SegMap lvl kspace rts kbody'
 
     pure $
@@ -250,10 +260,14 @@
   lam' <- intrablockLambda map_in_block lam
   pure (shape, arrs, Just (lam', nes))
 
+bindPatToRes :: Pat Type -> [VName] -> FlattenM ()
+bindPatToRes pat res = forM_ (zip (patNames pat) res) $ \(pe_v, v) ->
+  letBindNames [pe_v] $ BasicOp $ SubExp $ Var v
+
 intrablockStm :: InBlockMapTransformer -> Stm SOACS -> FlattenM ()
 intrablockStm map_in_block stm@(Let pat aux e) = do
   scope <- askScope
-  let lvl = SegThreadInBlock SegNoVirt
+  let lvl = SegThreadInBlock SegVirt
 
   case e of
     Loop merge form loopbody ->
@@ -270,28 +284,19 @@
       inputs' <- mapM (intrablockWithAccInput map_in_block) inputs
       lam' <- intrablockLambda map_in_block lam
       certifying (stmAuxCerts aux) . letBind pat $ WithAcc inputs' lam'
-    Op soac
-      | "sequential_outer" `inAttrs` stmAuxAttrs aux ->
-          intrablockStms map_in_block . fmap (certify (stmAuxCerts aux))
-            =<< runBuilder_ (FOT.transformSOAC pat soac)
     Op (Screma w arrs form)
       | Just lam <- isMapSOAC form ->
-          map_in_block pat w arrs lam
+          map_in_block lvl pat w arrs lam
     Op (Screma w arrs form)
       | Just (post_lam, scans, mapfun) <- isMaposcanomapSOAC form,
-        -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileGroupOp
+        -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileBlockScan
         -- cannot handle multiple scan operators yet.
         Scan scanfun nes <- singleScan scans -> do
           let scanfun' = soacsLambdaToGPU scanfun
               mapfun' = soacsLambdaToGPU mapfun
               post_op = soacsLambdaToGPU post_lam
-          scan_res <- certifying (stmAuxCerts aux) $ genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ())
-          zipWithM_
-            ( \pe v ->
-                letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v
-            )
-            (patElems pat)
-            scan_res
+          bindPatToRes pat <=< certifying (stmAuxCerts aux) $
+            genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ())
     Op (Screma w arrs form)
       | Just (reds, map_lam) <- isRedomapSOAC form -> do
           let onRed red =
@@ -302,15 +307,8 @@
                  in Reduce comm (soacsLambdaToGPU red_lam) (redNeutral red)
               reds_gpu = map onRed reds
               map_lam' = soacsLambdaToGPU map_lam
-          (red_res, stms) <- runBuilder (genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ()))
-          certifying (stmAuxCerts aux) $ do
-            addStms stms
-            zipWithM_
-              ( \pe v ->
-                  letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v
-              )
-              (patElems pat)
-              red_res
+          bindPatToRes pat <=< certifying (stmAuxCerts aux) $
+            genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ())
     Op (Screma w arrs form) ->
       -- This screma is too complicated for us to immediately do
       -- anything, so split it up and try again.
@@ -318,24 +316,12 @@
         =<< runBuilderT (dissectScrema pat w form arrs) (scopeForSOACs scope)
     Op (Hist w arrs ops bucket_fun) -> do
       let bucket_fun' = soacsLambdaToGPU bucket_fun
-
-      (hist_res, stms) <- runBuilder (genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ()))
-      certifying (stmAuxCerts aux) $ do
-        addStms stms
-        zipWithM_
-          ( \pe v ->
-              letBindNames [patElemName pe] $
-                BasicOp $
-                  SubExp $
-                    Var v
-          )
-          (patElems pat)
-          hist_res
+      bindPatToRes pat <=< certifying (stmAuxCerts aux) $
+        genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ())
     Op (Stream w arrs accs lam) -> do
       types <- asksScope castScope
-      ((), stream_stms) <-
-        runBuilderT (sequentialStreamWholeArray pat w accs lam arrs) types
-      intrablockStms map_in_block stream_stms
+      intrablockStms map_in_block
+        =<< runBuilderT_ (sequentialStreamWholeArray pat w accs lam arrs) types
     _ ->
       addStm $ soacsStmToGPU stm
 
diff --git a/src/Futhark/Pass/Flatten/PreProcess.hs b/src/Futhark/Pass/Flatten/PreProcess.hs
--- a/src/Futhark/Pass/Flatten/PreProcess.hs
+++ b/src/Futhark/Pass/Flatten/PreProcess.hs
@@ -52,7 +52,8 @@
 rewriteSoacStm scope (Let pat aux (Op soac))
   | "sequential_outer" `inAttrs` stmAuxAttrs aux =
       Just $
-        preprocessStms scope =<< runSimplifiedBuilder scope (FOT.transformSOAC pat soac)
+        preprocessStms scope
+          =<< runSimplifiedBuilder scope (certifying (stmAuxCerts aux) $ FOT.transformSOAC pat soac)
 rewriteSoacStm scope (Let pat aux (Op (Stream w arrs nes lam))) = Just $ do
   stms <- runSimplifiedBuilder scope (auxing aux $ sequentialStreamWholeArray pat w nes lam arrs)
   preprocessStms scope stms
diff --git a/src/Futhark/Pass/Flatten/SOAC.hs b/src/Futhark/Pass/Flatten/SOAC.hs
--- a/src/Futhark/Pass/Flatten/SOAC.hs
+++ b/src/Futhark/Pass/Flatten/SOAC.hs
@@ -838,12 +838,13 @@
 
 flattenMapForInBlock ::
   FlattenOps ->
+  SegLevel ->
   Pat Type ->
   SubExp ->
   [VName] ->
   Lambda SOACS ->
   FlattenM ()
-flattenMapForInBlock ops pat w arrs map_lam = do
+flattenMapForInBlock ops lvl pat w arrs map_lam = do
   scope <- askScope
   lam <- preprocessLambda (castScope scope) map_lam
   let arrs' = zipWith MapArray arrs $ map paramType (lambdaParams lam)
@@ -851,7 +852,7 @@
         distributeMapWith ops' scope pat [w] arrs' lam
   transformDistributed ops' mempty [w] distributed
   where
-    ops' = atSegLevel inBlockSegLevel ops
+    ops' = atSegLevel lvl ops
 
 resultMapMode :: InnerMapMode -> DistInputs -> Type -> InnerMapMode
 resultMapMode SingleDim _ _ = SingleDim
@@ -1116,7 +1117,7 @@
 transformMap ops attrs [] _env _inps pat w arrs map_lam = do
   -- Top-level map (no enclosing segments). Preprocess the body and then
   -- propagate the attributes onto it, so they influence how the body is
-  -- versioned (e.g. only_inner reaching a Screma produced by interchanging a
+  -- versioned (e.g. only_full reaching a Screma produced by interchanging a
   -- 'sequential_outer' loop). Order matters: preprocessing may rewrite a body
   -- statement, so propagating first would lose the attributes on the rewritten
   -- form. XXX: this is arguably a bug in preprocessing.
