futhark 0.28.2 → 0.29.1
raw patch · 61 files changed
+3279/−2181 lines, 61 files
Files
- CHANGELOG.md +45/−0
- docs/installation.rst +6/−3
- docs/js-api.rst +133/−145
- docs/language-reference.rst +32/−28
- docs/man/futhark-wasm-multicore.rst +8/−5
- docs/man/futhark-wasm.rst +28/−4
- futhark.cabal +2/−2
- rts/c/backends/cuda.h +28/−1
- rts/c/backends/hip.h +28/−1
- rts/c/backends/opencl.h +34/−0
- rts/c/gpu.h +63/−54
- rts/cuda/prelude.cu +3/−1
- rts/javascript/server.js +332/−146
- rts/javascript/values.js +45/−50
- rts/javascript/wrapperclasses.js +0/−91
- rts/javascript/wrappers.js +567/−0
- rts/opencl/copy.cl +3/−3
- rts/opencl/transpose.cl +37/−29
- rts/python/opencl.py +32/−4
- rts/python/scalar.py +18/−4
- src/Futhark/Actions.hs +83/−73
- src/Futhark/Analysis/Interference.hs +4/−0
- src/Futhark/CodeGen/Backends/GPU.hs +12/−0
- src/Futhark/CodeGen/Backends/GenericC.hs +7/−3
- src/Futhark/CodeGen/Backends/GenericC/Code.hs +25/−5
- src/Futhark/CodeGen/Backends/GenericC/Fun.hs +3/−1
- src/Futhark/CodeGen/Backends/GenericC/Monad.hs +64/−13
- src/Futhark/CodeGen/Backends/GenericC/Types.hs +2/−2
- src/Futhark/CodeGen/Backends/GenericPython.hs +4/−0
- src/Futhark/CodeGen/Backends/GenericWASM.hs +26/−554
- src/Futhark/CodeGen/Backends/MulticoreC.hs +54/−26
- src/Futhark/CodeGen/Backends/MulticoreISPC.hs +42/−15
- src/Futhark/CodeGen/Backends/MulticoreWASM.hs +5/−25
- src/Futhark/CodeGen/Backends/PyOpenCL.hs +5/−0
- src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs +7/−1
- src/Futhark/CodeGen/Backends/SequentialWASM.hs +5/−24
- src/Futhark/CodeGen/ImpCode.hs +23/−1
- src/Futhark/CodeGen/ImpCode/GPU.hs +46/−11
- src/Futhark/CodeGen/ImpCode/Kernels.hs +10/−1
- src/Futhark/CodeGen/ImpCode/Multicore.hs +1/−0
- src/Futhark/CodeGen/ImpGen.hs +14/−0
- src/Futhark/CodeGen/ImpGen/GPU.hs +2/−3
- src/Futhark/CodeGen/ImpGen/GPU/Base.hs +183/−85
- src/Futhark/CodeGen/ImpGen/GPU/Block.hs +467/−192
- src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs +10/−12
- src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs +332/−251
- src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs +1/−16
- src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs +64/−64
- src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs +16/−10
- src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs +55/−50
- src/Futhark/CodeGen/ImpGen/Multicore/Base.hs +11/−4
- src/Futhark/CodeGen/RTS/JavaScript.hs +4/−4
- src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs +5/−1
- src/Futhark/Pass/ExpandAllocations.hs +42/−28
- src/Futhark/Pass/ExtractMulticore.hs +7/−2
- src/Futhark/Pass/Flatten.hs +42/−9
- src/Futhark/Pass/Flatten/Builtins.hs +10/−16
- src/Futhark/Pass/Flatten/Incremental.hs +73/−27
- src/Futhark/Pass/Flatten/Intrablock.hs +63/−77
- src/Futhark/Pass/Flatten/PreProcess.hs +2/−1
- src/Futhark/Pass/Flatten/SOAC.hs +4/−3
CHANGELOG.md view
@@ -5,6 +5,51 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [0.29.1]++### Added++* Better GPU code generation for reductions with very complicated operators.++* Optimised internal book-keeping of GPU operations, resulting in lower static+ memory usage for large programs.++### Changed++* The flattening attributes have changed somewhat. `#[sequential_inner]`+ replaced by `#[flattening(only_outer)]`, and `#[flattening(only_inner)]` has+ been renamed to `#[flattening(only_full)]`, which now also affects any nested+ parallelism.++### Fixed++* Invalid memory expansion in some rare cases.++* The ISPC backend would handle `f16` incorrectly in some cases.++* Fixed a very rare out-of-bounds access in scans with array operands on GPU+ backends.++* Invalid handling of aliasing in flattening. (#2595)++* Invalid short-circuiting could cause a compiler crash. (#2597)++* Execution errors with array-operator scans with nonstandard tuning settings.++* On GPU backends, a failing safety check in code fused with a scan could cause+ barrier divergence or deadlocks.++* On the `cuda` and `hip` backends, a failing safety check in a kernel with+ barriers stopped only the failing thread rather than its whole thread block,+ which could cause deadlocks.++* On GPU backends, reductions and scans with operators that contain assertions+ could cause barrier divergence.++* On GPU and multicore backends, a failing operator in a `reduce_by_index` or+ `hist` that is implemented with locks could leave a lock held, causing a+ deadlock.+ ## [0.28.2] ### Added
docs/installation.rst view
@@ -79,7 +79,7 @@ Tarballs of binary releases can be `found online <https://futhark-lang.org/releases/>`_, but are available only for-very few platforms (as of this writing, only GNU/Linux on x86_64).+very few platforms (GNU/Linux on x86_64 and aarch64). See the enclosed ``README.md`` for installation instructions. Furthermore, every day a program automatically clones the Git@@ -96,8 +96,11 @@ **Linux (x86_64)** `futhark-nightly-linux-x86_64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-x86_64.tar.xz>`_ -**macOS (ARM64)**- `futhark-nightly-macos-arm64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-arm64.tar.xz>`_+**Linux (Aarch64)**+ `futhark-nightly-linux-aarch64.tar.xz <https://futhark-lang.org/releases/futhark-nightly-linux-aarch64.tar.xz>`_++**macOS (AARCH64)**+ `futhark-nightly-macos-aarch64.zip <https://futhark-lang.org/releases/futhark-nightly-macos-aarch64.tar.xz>`_ **Windows (x86_64)** `futhark-nightly-windows-x86_64.zip <https://futhark-lang.org/releases/futhark-nightly-windows-x86_64.zip>`_
docs/js-api.rst view
@@ -3,69 +3,36 @@ JavaScript API Reference ======================== -The :ref:`futhark-wasm(1)`, :ref:`futhark-wasm-multicore(1)`, and-:ref:`futhark-webgpu(1)` compilers can generate JavaScript wrapper code-when used with ``--library``. This chapter describes the API exposed by-those wrappers.--The exact generated files and the top-level JavaScript interface differ-somewhat between the WASM backend and the WebGPU backend, so the-relevant differences are described below.+The :ref:`futhark-wasm(1)` and :ref:`futhark-wasm-multicore(1)` compilers can+generate JavaScript wrapper code when used with ``--library``. This chapter+describes the API exposed by those wrappers. -First a warning: **the JavaScript API is experimental**. It may-change incompatibly even in minor versions of the compiler.+First a warning: **the JavaScript API is experimental**. It may change+incompatibly even in minor versions of the compiler. A Futhark program ``futlib.fut`` compiled with ``--library`` produces-generated C code as well as JavaScript-facing runtime files. Some of-these files are backend-specific.--For the WASM backend, a build typically produces:--* ``futlib.c``, ``futlib.h``: implementation and header C files- generated by the compiler, similar to ``futhark c``.-* ``futlib.class.js``: an intermediate JavaScript build artifact.-* ``futlib.wasm``: the compiled WebAssembly module, which must be- present at runtime.-* ``futlib.mjs``: an ES module that exports the JavaScript wrapper API.--For the WebGPU backend, a build typically produces:--* ``futlib.c``, ``futlib.h``: implementation and header C files- generated by the compiler.-* ``futlib.js``: the generated backend runtime module.-* ``futlib.wrapper.js``: the JavaScript wrapper that exposes the- Futhark-facing API.-* ``futlib.json``: manifest data used by the generated wrapper.-* ``futlib.wasm``: a generated WebAssembly artifact used by the runtime.--The exact file set is backend-dependent and may change over time.+the following files: -The generated JavaScript wrapper exposes a ``FutharkModule`` class,-which is the preferred top-level interface for using a compiled Futhark-program from JavaScript. The module is initialised with the generated-backend runtime module.+* ``futlib.mjs``: an ES module that exports the JavaScript API.+* ``futlib.wasm``: the compiled WebAssembly module, which is loaded by+ ``futlib.mjs`` and must be in the same directory.+* ``futlib.c`` and ``futlib.class.js``: intermediate files, which are+ not needed at runtime. -The initialisation pattern is the same for the WASM-style backend and-the WebGPU backend. The only difference is how ``Module`` is made available.-For the WASM-style backend, the generated ``.mjs`` file exports it as the-default export, so it is imported together with ``FutharkModule``:+The module exports a ``FutharkContext`` class, which is the top-level+interface for using a compiled Futhark program from JavaScript: .. code-block:: javascript - import Module, { FutharkModule } from './futlib.mjs';-- const module = await Module();-- const fut = new FutharkModule();- await fut.init(module);--For the WebGPU backend, the generated runtime module is usually loaded-before the application code, for example:+ import { FutharkContext } from './futlib.mjs'; -.. code-block:: html+ const fut = new FutharkContext();+ await fut.init(); - <script src="futlib.js"></script>- <script src="main.js"></script>+In a browser, the module must be loaded by a ``<script+type="module">`` element (or imported by another module), and the+files must be served over HTTP. In Node.js, a CommonJS module can+load it with ``await import('./futlib.mjs')``. General concerns ----------------@@ -75,102 +42,110 @@ You are responsible for eventually freeing all objects produced by the API, using the appropriate methods. +Methods that may run Futhark code are asynchronous, and return a+promise. The asynchronous operations on a ``FutharkContext`` run one at+a time, in the order they were started, and while one is running, no+synchronous method may be called on the ``FutharkContext`` or its+values. Errors, including errors that occur while running Futhark+code, are reported by throwing an ``Error`` (or rejecting the promise+with one). The arguments of every method are checked, and an invalid+argument is reported with a ``TypeError`` or ``RangeError``.+ Top-level wrapper objects ------------------------- -The preferred top-level JavaScript interface is ``FutharkModule``. A-``FutharkModule`` object represents an instance of a compiled Futhark-program. It owns the underlying Futhark context and is used to construct-arrays, call entry points, and free resources.+The top-level JavaScript interface is ``FutharkContext``. A+``FutharkContext`` object represents an instance of a compiled Futhark+program, and corresponds to a context in the C API. It is used to+construct arrays, call entry points, and free resources. -.. js:class:: FutharkModule()+.. js:class:: FutharkContext() A bookkeeping class representing an instance of a compiled Futhark program. -.. js:function:: FutharkModule.init(module)-- Asynchronously initialise the ``FutharkModule`` object.-- For the WASM-style backend, the ``module`` argument is optional. If- omitted, the generated wrapper loads the WebAssembly module itself.-- For the WebGPU backend, ``module`` must be the generated backend- runtime module.--.. js:function:: FutharkModule.free()+.. js:function:: FutharkContext.init(options) - Frees the Futhark context and configuration associated with the- module. It is an error to use a ``FutharkArray`` or ``FutharkOpaque``- after the ``FutharkModule`` on which it was created has been freed.+ Asynchronously initialise the ``FutharkContext`` object. This must+ be done exactly once, before the object is used for anything else.+ Each ``FutharkContext`` loads its own instance of the WebAssembly+ module, so values cannot be passed between different+ ``FutharkContext`` objects. -.. js:attribute:: FutharkModule.entry+ The ``options`` argument is optional. If given, it must be an object+ with any of the following fields, which correspond to the+ configuration functions of the C API (see :ref:`c-api`): - Object containing the generated entry point functions. For example,- an entry point named ``main`` is available as ``fut.entry.main``.+ * ``debugging``, ``logging``, ``profiling``: booleans that enable+ debugging, logging, or profiling. Logging information is printed+ with ``console.error``. -.. js:attribute:: FutharkModule.types+ * ``tuning_params``: an object mapping the names of tuning parameters+ to their values. - Object containing generated type information and array constructor- objects for the array types used by the program.+ * ``num_threads``: the number of threads to use. Only supported by+ :ref:`futhark-wasm-multicore(1)`. -The backends also provides utility methods for synchronisation,-cache management, and profiling.+ With :ref:`futhark-wasm-multicore(1)`, ``init()`` fails unless+ ``SharedArrayBuffer`` is available, which in a browser requires the+ page to be *cross-origin isolated*. -.. js:function:: FutharkModule.context_sync()+.. js:function:: FutharkContext.free() - Wait for pending backend work associated with the context to finish.+ Frees the underlying Futhark context and configuration. It is an+ error to use a ``FutharkArray`` or ``FutharkOpaque`` after the+ ``FutharkContext`` on which it was created has been freed. -.. js:function:: FutharkModule.clear_caches()+.. js:attribute:: FutharkContext.entry - Clear backend caches associated with the context.+ Object containing the entry point functions. For example, an entry+ point named ``main`` is available as ``fut.entry.main``. -.. js:function:: FutharkModule.report()+.. js:attribute:: FutharkContext.types - Return a profiling report string.+ Object mapping the name of each array type used by the program, such+ as ``[]i32``, to its constructor object (see `Array construction`_). -.. js:function:: FutharkModule.pause_profiling()+The ``FutharkContext`` also provides utility methods for+synchronisation, cache management, and profiling. They are all+asynchronous. - Pause profiling.+.. js:function:: FutharkContext.context_sync() -.. js:function:: FutharkModule.unpause_profiling()+ Wait for pending work associated with the context to finish. - Resume profiling.+.. js:function:: FutharkContext.clear_caches() -WASM compatibility interface-~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ Clear caches associated with the context. -The WASM-style backend also keep the older ``FutharkContext`` interface-for backwards compatibility.+.. js:function:: FutharkContext.report() -.. js:function:: newFutharkContext()+ Return a profiling report string. - Asynchronously create a new ``FutharkContext`` object.+.. js:function:: FutharkContext.pause_profiling() -.. js:class:: FutharkContext()+ Pause profiling. - Backwards-compatible wrapper class for WASM-style backend.- ``FutharkContext`` supports the same main interface as- ``FutharkModule``, including ``entry``, ``types``, array constructors,- and ``free``.+.. js:function:: FutharkContext.unpause_profiling() -New code should prefer ``FutharkModule``.+ Resume profiling. Values ------ Numeric types ``u8``, ``u16``, ``u32``, ``i8``, ``i16``, ``i32``, ``f16``, ``f32``, and ``f64`` are mapped to JavaScript's standard number type.-64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``. ``bool``-is mapped to JavaScript's ``boolean`` type. Arrays are represented by-``FutharkArray`` objects. Complex types (records, nested tuples, etc.)-are represented by ``FutharkOpaque`` objects.+64-bit integers ``u64`` and ``i64`` are mapped to ``BigInt``, although a+number is also accepted as an argument. An integer argument must be in+the range of its type. ``bool`` is mapped to JavaScript's ``boolean``+type. Arrays are represented by ``FutharkArray`` objects. Complex+types (records, nested tuples, etc.) are represented by+``FutharkOpaque`` objects. FutharkArray ------------ -Arrays are represented by ``FutharkArray`` objects. The common array API is-shared by the WASM backend and the WebGPU backend.+Arrays are represented by ``FutharkArray`` objects. .. js:function:: FutharkArray.shape() @@ -178,61 +153,69 @@ .. js:function:: FutharkArray.values() - Asynchronously returns a flat typed array containing the array data.--.. js:function:: FutharkArray.free()-- Frees the memory used by the ``FutharkArray``.--The WASM backend also provides a few older convenience methods.--.. js:function:: FutharkArray.toArray()-- Returns a nested JavaScript array.+ Asynchronously returns a flat typed array, in row-major order,+ containing a copy of the array elements. It remains valid after the+ ``FutharkArray`` is freed. The typed array corresponds to the+ element type, such as ``Int32Array`` for ``i32``, except that: -.. js:function:: FutharkArray.toTypedArray()+ * For ``f16``, it is a ``Float16Array`` if supported by the+ JavaScript implementation, and otherwise a ``Float32Array``. - Returns a flat typed array of the underlying data.+ * For ``bool``, it is a ``Uint8Array`` whose elements are 0 or 1. -.. js:function:: FutharkArray.shape()+.. js:function:: FutharkArray.free() - Returns the shape of the ``FutharkArray`` as an array of ``BigInt`` values.+ Frees the memory used by the ``FutharkArray``. Array construction ~~~~~~~~~~~~~~~~~~ -Each array type used by an entry point is exposed through a generated-constructor object. For example, for the type ``[]i32``:+Each array type used by an entry point, including as the field of a+record, is exposed through a generated constructor object. For+example, for the type ``[]i32``: .. js:function:: fut.i32_1d.from_data(data, dim0) Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from- a JavaScript ``Array`` or the corresponding typed array, with the- given shape.--For the WASM backend, the generated constructor object also-provides a convenience method for nested JavaScript arrays.--.. js:function:: fut.i32_1d.from_jsarray(jsarray)+ a JavaScript ``Array`` or typed array in row-major order, with the+ given shape. The dimensions can be numbers or ``BigInt``\ s, and+ their product must be the length of ``data``. The elements must be+ valid entry point arguments of the element type, except that for+ ``bool``, they may also be 0 or 1 (as in the result of+ :js:func:`FutharkArray.values`). - Creates and returns a one-dimensional ``i32`` ``FutharkArray`` from- a JavaScript array. For higher-dimensional arrays, the JavaScript- array is expected to be nested according to the array rank.+The constructor object for an array type with ``n`` dimensions has a+``from_data`` method that takes ``n`` dimensions. FutharkOpaque ------------- Complex types (records, nested tuples, etc) are represented by-``FutharkOpaque``. It has no use outside of being accepted and-returned by entry point functions. For this reason the method only has-one function for freeing the memory when ``FutharkOpaque`` is no-longer used.+``FutharkOpaque``. They can be passed to and returned by entry point+functions, and the fields of records (including tuples) can be read+by projection, as described below. .. js:function:: FutharkOpaque.free() - Frees memory used by FutharkOpaque. Should be called when Futhark- Opaque is no longer used.+ Frees the memory used by the ``FutharkOpaque``. +Records+~~~~~~~++Records, including tuples, are ``FutharkOpaque`` objects whose fields+can be *projected*. For each field of a record type, the top-level+wrapper object has a method named after the corresponding projection+function in the C API (see :ref:`records`), without the ``futhark_``+prefix. For example, the field ``x`` of a record with C type ``struct+futhark_opaque_foo`` is projected as follows:++.. js:function:: fut.project_opaque_foo_x(obj)++ Asynchronously returns the value of field ``x`` of the record+ ``obj``. An array or opaque field is returned as a new+ ``FutharkArray`` or ``FutharkOpaque``, which must be freed+ independently of ``obj``.+ Entry Points ------------ @@ -246,8 +229,13 @@ .. js:function:: fut.entry.<entry_point_name>(in1, ..., inN) - The entry point function takes the JavaScript representations of the- Futhark entry point arguments and returns the result.+ Asynchronously call the entry point with the JavaScript+ representations of its arguments, and return the result. - If the entry point has multiple outputs, the return value is an array- containing the outputs in order.+ An entry point that returns a tuple returns a single+ ``FutharkOpaque``, whose elements can be obtained by projection (see+ `Records`_).++ An argument that the entry point consumes (see+ :ref:`in-place-updates`) can afterwards only be freed, and must not+ also be passed as another argument.
docs/language-reference.rst view
@@ -717,11 +717,13 @@ Slicing of multiple dimensions can be done by separating with commas, and may be intermixed freely with indexing. -If ``s`` is elided it defaults to ``1``. If ``i`` or ``j`` is elided, their-value depends on the sign of ``s``. If ``s`` is positive, ``i`` become ``0``-and ``j`` become the length of the array. If ``s`` is negative, ``i`` becomes-the length of the array minus one, and ``j`` becomes minus one. This means that-``a[::-1]`` is the reverse of the array ``a``.+If ``s`` is elided it defaults to ``1``. If ``i`` or ``j`` is elided, their+value depends on the sign of ``s``. If ``s`` is positive, ``i`` becomes ``0`` and+``j`` becomes the length of the array. If ``s`` is negative, ``i`` becomes the+length of the array minus one, and ``j`` becomes minus one. This means that+``a[::-1]`` is the reverse of the array ``a``, and that ``a[:]`` retrieves the+entire dimension - this is only useful when slicing multiple dimensions. Note+that ``a[::]`` is not syntactically valid. In the general case, the size of the array produced by a slice is unknown (see `Size types`_). In a few cases, the size is known@@ -1767,12 +1769,13 @@ attribute, written as ``#[attr]``. This may affect how it is treated by the compiler or other tools. In no case will attributes affect or change the *semantics* of a program, but it may affect how well it compiles and runs (or in-some cases, whether it compiles or runs at all). Unknown attributes are silently-ignored. Most have no effect in the interpreter. An attribute can be either an-*atom*, written as an identifier or number, or *compound*, consisting of an-identifier and a comma-separated sequence of attributes. The latter is used for-grouping and encoding of more complex information. An identifier prefixed with a-``$`` must be the name of a term variable in scope.+some cases, whether it compiles or runs at all). Unknown attributes are usually+ignored silently, but may in some cases cause a warning. Most have no effect in+the interpreter. An attribute can be either an *atom*, written as an identifier+or number, or *compound*, consisting of an identifier and a comma-separated+sequence of attributes. The latter is used for grouping and encoding of more+complex information. An identifier prefixed with a ``$`` must be the name of a+term variable in scope. Expression attributes ~~~~~~~~~~~~~~~~~~~~~@@ -1838,29 +1841,35 @@ ``flattening(no_outer)`` ........................ -When using incremental flattening, do not generate the "only outer-parallelism" version for the attributed SOACs.+During incremental flattening, do not generate the version that exploits only+outermost parallelism. ``flattening(no_intra)`` ........................ -When using incremental flattening, do not generate the "intra-block-parallelism" version for the attributed SOACs.+During incremental flattening, do not generate the version that maps outermost+parallelism onto GPU thread blocks. +``flattening(only_outer)``+..........................++During incremental flattening, only generate the version that exploits outermost+parallelism and sequentialises inner parallelism.+ ``flattening(only_intra)`` .......................... -When using incremental flattening, *only* generate the "intra-block-parallelism" version of the attributed SOACs. **Beware**: the-resulting program will fail to run if the inner parallelism does not-fit on the device.+During incremental flattening, *only* generate the version that maps outermost+parallelism onto GPU thread blocks. -``flattening(only_inner)``+**Beware**: the resulting program will fail to run if the inner parallelism does+not fit on the device.++``flattening(only_full)`` .......................... -When using incremental flattening, do not generate multiple versions-for this SOAC, but do exploit inner parallelism (which may give rise-to multiple versions at deeper levels).+During incremental flattening, generate only the fully flattened version. Any+versioning attributes inside the SOAC are ignored. ``noinline`` ............@@ -1886,11 +1895,6 @@ Turn the outer parallelism in the attributed SOAC sequential, but preserve any inner parallelism.--``sequential_inner``-....................--Exploit only outer parallelism in the attributed SOAC. ``unroll`` ..........
docs/man/futhark-wasm-multicore.rst view
@@ -16,10 +16,8 @@ ``futhark wasm-multicore`` translates a Futhark program to multi-threaded WebAssembly code by first generating C as ``futhark-c``, and then using Emscripten (``emcc``). This produces a ``.js``-file that allows the compiled code to be invoked from JavaScript.-Executables implement the Futhark server protocol and can be run with-Node.js.+c``, and then using Emscripten (``emcc``). The files produced are+the same as for :ref:`futhark-wasm(1)`. OPTIONS =======@@ -35,7 +33,12 @@ ================== Generated executables accept the same options as those generated by-:ref:`futhark-wasm(1)`.+:ref:`futhark-wasm(1)`. The following additional options are+accepted.++--num-threads=INT++ Use this many threads. SEE ALSO ========
docs/man/futhark-wasm.rst view
@@ -16,9 +16,12 @@ ``futhark wasm`` translates a Futhark program to sequential WebAssembly code by first generating C as ``futhark c``, and then-using Emscripten (``emcc``). This produces a ``.js`` file that allows-the compiled code to be invoked from JavaScript. Executables-implement the Futhark server protocol and can be run with Node.js.+using Emscripten (``emcc``). For a program ``prog.fut``, this+produces an ES module ``prog.mjs``, along with the WebAssembly module+``prog.wasm``. With ``--library``, the ES module exports the API+described in :ref:`js-api`. Otherwise, ``prog`` is an executable+that runs the ES module with Node.js, and implements the Futhark+server protocol. OPTIONS =======@@ -31,7 +34,7 @@ ``CFLAGS`` Space-separated list of options passed to ``emcc``. Defaults- to ``-O3 -std=c99`` if unset.+ to ``-O3 -msimd128`` if unset. ``EMCFLAGS`` @@ -57,6 +60,27 @@ Print various low-overhead logging information to stderr while running. +-P, --profile++ Enable the collection of profiling information.++--param=ASSIGNMENT++ Set a tuning parameter to the given value. ``ASSIGNMENT`` must be of+ the form ``NAME=INT``.++--print-params++ Print all tuning parameters that can be set with ``--param`` or+ ``--tuning``.++--tuning=FILE++ Read size=value assignments from the given file.++--cache-file=FILE++ Accepted for compatibility with other backends, but has no effect. SEE ALSO ========
futhark.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name: futhark-version: 0.28.2+version: 0.29.1 synopsis: An optimising compiler for a functional, array-oriented language. description: Futhark is a small programming language designed to be compiled to@@ -70,7 +70,7 @@ rts/style.css rts/javascript/server.js rts/javascript/values.js- rts/javascript/wrapperclasses.js+ rts/javascript/wrappers.js rts/opencl/copy.cl rts/opencl/prelude.cl rts/opencl/transpose.cl
rts/c/backends/cuda.h view
@@ -215,6 +215,8 @@ CUdeviceptr global_failure_args; // True if a potentially failing kernel has been enqueued. int32_t failure_is_an_option;+ CUdeviceptr bookkeeping_counters;+ CUdeviceptr bookkeeping_locks; int total_runs; long int total_runtime; int64_t peak_mem_usage_device;@@ -724,6 +726,16 @@ return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+ CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_counters, 0,+ sizeof(int32_t)*bookkeeping_counters_size,+ ctx->stream));+ CUDA_SUCCEED_OR_RETURN(cuMemsetD8Async(ctx->bookkeeping_locks, 0,+ sizeof(int32_t)*bookkeeping_locks_size,+ ctx->stream));+ return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) { CUDA_SUCCEED_OR_RETURN(cuCtxPushCurrent(ctx->cu_ctx)); CUDA_SUCCEED_OR_RETURN(cuCtxSynchronize());@@ -745,6 +757,11 @@ &no_failure, sizeof(int32_t))); + // The failing kernel may have left the bookkeeping buffers nonzero.+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ int64_t args[max_failure_args+1]; CUDA_SUCCEED_OR_RETURN( cuMemcpyDtoH(&args,@@ -856,6 +873,12 @@ // The +1 is to avoid zero-byte allocations. CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1))); + CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));+ CUDA_SUCCEED_FATAL(cuMemAlloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) { return 1; }@@ -868,7 +891,11 @@ free_builtin_kernels(ctx, ctx->kernels); cuMemFree(ctx->global_failure); cuMemFree(ctx->global_failure_args);- CUDA_SUCCEED_FATAL(gpu_free_all(ctx));+ cuMemFree(ctx->bookkeeping_counters);+ cuMemFree(ctx->bookkeeping_locks);+ if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {+ futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);+ } CUDA_SUCCEED_FATAL(cuStreamDestroy(ctx->stream)); CUDA_SUCCEED_FATAL(cuModuleUnload(ctx->module)); if (ctx->cfg->use_primary_context) {
rts/c/backends/hip.h view
@@ -182,6 +182,8 @@ void* global_failure_args; // True if a potentially failing kernel has been enqueued. int32_t failure_is_an_option;+ void* bookkeeping_counters;+ void* bookkeeping_locks; int total_runs; long int total_runtime; int64_t peak_mem_usage_device;@@ -578,6 +580,16 @@ return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+ HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_counters, 0,+ sizeof(int32_t)*bookkeeping_counters_size,+ ctx->stream));+ HIP_SUCCEED_OR_RETURN(hipMemsetAsync(ctx->bookkeeping_locks, 0,+ sizeof(int32_t)*bookkeeping_locks_size,+ ctx->stream));+ return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) { HIP_SUCCEED_OR_RETURN(hipStreamSynchronize(ctx->stream)); if (ctx->failure_is_an_option) {@@ -596,6 +608,11 @@ &no_failure, sizeof(int32_t))); + // The failing kernel may have left the bookkeeping buffers nonzero.+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ int64_t args[max_failure_args+1]; HIP_SUCCEED_OR_RETURN(hipMemcpyDtoH(&args, ctx->global_failure_args,@@ -685,6 +702,12 @@ // The +1 is to avoid zero-byte allocations. HIP_SUCCEED_FATAL(hipMalloc(&ctx->global_failure_args, sizeof(int64_t)*(max_failure_args+1))); + HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_counters, sizeof(int32_t)*bookkeeping_counters_size));+ HIP_SUCCEED_FATAL(hipMalloc(&ctx->bookkeeping_locks, sizeof(int32_t)*bookkeeping_locks_size));+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) { return 1; }@@ -697,7 +720,11 @@ free_builtin_kernels(ctx, ctx->kernels); hipFree(ctx->global_failure); hipFree(ctx->global_failure_args);- HIP_SUCCEED_FATAL(gpu_free_all(ctx));+ hipFree(ctx->bookkeeping_counters);+ hipFree(ctx->bookkeeping_locks);+ if (gpu_free_all(ctx) != FUTHARK_SUCCESS) {+ futhark_panic(-1, "Failed to free GPU memory:\n%s", ctx->error);+ } HIP_SUCCEED_FATAL(hipStreamDestroy(ctx->stream)); HIP_SUCCEED_FATAL(hipModuleUnload(ctx->module)); }
rts/c/backends/opencl.h view
@@ -449,6 +449,8 @@ cl_mem global_failure_args; // True if a potentially failing kernel has been enqueued. cl_int failure_is_an_option;+ cl_mem bookkeeping_counters;+ cl_mem bookkeeping_locks; int total_runs; long int total_runtime; int64_t peak_mem_usage_device;@@ -672,6 +674,19 @@ return 0; } +static int zero_bookkeeping(struct futhark_context* ctx) {+ cl_int zero = 0;+ OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_counters,+ &zero, sizeof(zero),+ 0, sizeof(int32_t)*bookkeeping_counters_size,+ 0, NULL, NULL));+ OPENCL_SUCCEED_OR_RETURN(clEnqueueFillBuffer(ctx->queue, ctx->bookkeeping_locks,+ &zero, sizeof(zero),+ 0, sizeof(int32_t)*bookkeeping_locks_size,+ 0, NULL, NULL));+ return 0;+}+ int futhark_context_sync(struct futhark_context* ctx) { // Check for any delayed error. cl_int failure_idx = -1;@@ -696,6 +711,11 @@ 0, sizeof(cl_int), &no_failure, 0, NULL, NULL)); + // The failing kernel may have left the bookkeeping buffers nonzero.+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ int64_t args[max_failure_args+1]; OPENCL_SUCCEED_OR_RETURN( clEnqueueReadBuffer(ctx->queue,@@ -1154,6 +1174,18 @@ sizeof(int64_t)*(max_failure_args+1), NULL, &error); OPENCL_SUCCEED_OR_RETURN(error); + ctx->bookkeeping_counters =+ clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,+ sizeof(int32_t)*bookkeeping_counters_size, NULL, &error);+ OPENCL_SUCCEED_OR_RETURN(error);+ ctx->bookkeeping_locks =+ clCreateBuffer(ctx->ctx, CL_MEM_READ_WRITE,+ sizeof(int32_t)*bookkeeping_locks_size, NULL, &error);+ OPENCL_SUCCEED_OR_RETURN(error);+ if (zero_bookkeeping(ctx) != 0) {+ return 1;+ }+ if ((ctx->kernels = init_builtin_kernels(ctx)) == NULL) { return 1; }@@ -1168,6 +1200,8 @@ free_builtin_kernels(ctx, ctx->kernels); OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure)); OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->global_failure_args));+ OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_counters));+ OPENCL_SUCCEED_FATAL(clReleaseMemObject(ctx->bookkeeping_locks)); (void)gpu_free_all(ctx); (void)clReleaseProgram(ctx->clprogram); (void)clReleaseCommandQueue(ctx->queue);
rts/c/gpu.h view
@@ -172,6 +172,7 @@ struct builtin_kernels* init_builtin_kernels(struct futhark_context* ctx) { struct builtin_kernels *kernels = malloc(sizeof(struct builtin_kernels));+ gpu_create_kernel(ctx, &kernels->map_transpose_1b, "map_transpose_1b"); gpu_create_kernel(ctx, &kernels->map_transpose_1b_large, "map_transpose_1b_large"); gpu_create_kernel(ctx, &kernels->map_transpose_1b_low_height, "map_transpose_1b_low_height");@@ -375,23 +376,27 @@ void* args[15]; size_t args_sizes[15] = {- sizeof(gpu_mem), sizeof(int64_t),- sizeof(gpu_mem), sizeof(int64_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t),- sizeof(int32_t)+ sizeof(int64_t), // dst_offset+ sizeof(int64_t), // src_offset+ sizeof(int32_t), // num_arrays+ sizeof(int32_t), // x_elems+ sizeof(int32_t), // y_elems+ sizeof(int32_t), // dst_array_stride+ sizeof(int32_t), // dst_row_stride+ sizeof(int32_t), // src_array_stride+ sizeof(int32_t), // src_row_stride+ sizeof(int32_t), // mulx+ sizeof(int32_t), // muly+ sizeof(int32_t), // repeat_1+ sizeof(int32_t), // repeat_2+ sizeof(gpu_mem), // dst+ sizeof(gpu_mem) // src }; - args[0] = &dst;- args[1] = &dst_offset;- args[2] = &src;- args[3] = &src_offset;+ args[0] = &dst_offset;+ args[1] = &src_offset;+ args[13] = &dst;+ args[14] = &src; if (dst_extent <= 2147483647L && src_extent <= 2147483647L) { if (m <= TR_BLOCK_DIM/2 && n <= TR_BLOCK_DIM/2) {@@ -435,12 +440,12 @@ block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD; block[2] = 1; }- args[4] = &k32;- args[5] = &m32;- args[6] = &n32;- for (int i = 0; i < 4; i++) { args[7+i] = &strides32[i]; }- args[11] = &mulx32;- args[12] = &muly32;+ args[2] = &k32;+ args[3] = &m32;+ args[4] = &n32;+ for (int i = 0; i < 4; i++) { args[5+i] = &strides32[i]; }+ args[9] = &mulx32;+ args[10] = &muly32; } else { if (ctx->logging) { fprintf(ctx->log, "Using large kernel\n"); } kernel = kernel_large;@@ -450,13 +455,13 @@ block[0] = TR_TILE_DIM; block[1] = TR_TILE_DIM/TR_ELEMS_PER_THREAD; block[2] = 1;- args[4] = &k;- args[5] = &m;- args[6] = &n;- for (int i = 0; i < 4; i++) { args[7+i] = &strides[i]; }- args[11] = &mulx;- args[12] = &muly;- for (int i = 4; i < 13; i++) { args_sizes[i] = sizeof(int64_t); }+ args[2] = &k;+ args[3] = &m;+ args[4] = &n;+ for (int i = 0; i < 4; i++) { args[5+i] = &strides[i]; }+ args[9] = &mulx;+ args[10] = &muly;+ for (int i = 2; i < 11; i++) { args_sizes[i] = sizeof(int64_t); } } // Cap the number of thead blocks we launch and figure out how many@@ -465,10 +470,10 @@ int32_t repeat_2 = grid[2] / MAX_TR_THREAD_BLOCKS; grid[1] = repeat_1 > 0 ? MAX_TR_THREAD_BLOCKS : grid[1]; grid[2] = repeat_2 > 0 ? MAX_TR_THREAD_BLOCKS : grid[2];- args[13] = &repeat_1;- args[14] = &repeat_2;- args_sizes[13] = sizeof(repeat_1);- args_sizes[14] = sizeof(repeat_2);+ args[11] = &repeat_1;+ args[12] = &repeat_2;+ args_sizes[11] = sizeof(repeat_1);+ args_sizes[12] = sizeof(repeat_2); if (ctx->logging) { fprintf(ctx->log, "\n");@@ -515,19 +520,20 @@ void* args[6+(8*3)]; size_t args_sizes[6+(8*3)]; - args[0] = &dst;- args_sizes[0] = sizeof(gpu_mem);- args[1] = &dst_offset;- args_sizes[1] = sizeof(dst_offset);- args[2] = &src;- args_sizes[2] = sizeof(gpu_mem);- args[3] = &src_offset;- args_sizes[3] = sizeof(src_offset);- args[4] = &n;- args_sizes[4] = sizeof(n);- args[5] = &r;- args_sizes[5] = sizeof(r);+ args[28] = &dst;+ args_sizes[28] = sizeof(gpu_mem);+ args[29] = &src;+ args_sizes[29] = sizeof(gpu_mem); + args[0] = &dst_offset;+ args_sizes[0] = sizeof(dst_offset);+ args[1] = &src_offset;+ args_sizes[1] = sizeof(src_offset);+ args[2] = &n;+ args_sizes[2] = sizeof(n);+ args[3] = &r;+ args_sizes[3] = sizeof(r);+ int64_t zero = 0; // To improve the odds of coalesced memory accesses, we pass the dimensions in@@ -536,7 +542,10 @@ int perm[8]; for (int i = 0; i < r; i++) { int j = i;- while (j > 0 && llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])) {+ while (j > 0 &&+ (llabs(src_strides[perm[j-1]]) > llabs(src_strides[i])+ || (llabs(src_strides[perm[j-1]]) == llabs(src_strides[i])+ && llabs(dst_strides[perm[j-1]]) > llabs(dst_strides[i])))) { perm[j] = perm[j-1]; j--; }@@ -544,17 +553,17 @@ } for (int i = 0; i < 8; i++) {- args_sizes[6+i*3] = sizeof(int64_t);- args_sizes[6+i*3+1] = sizeof(int64_t);- args_sizes[6+i*3+2] = sizeof(int64_t);+ args_sizes[4+i*3] = sizeof(int64_t);+ args_sizes[4+i*3+1] = sizeof(int64_t);+ args_sizes[4+i*3+2] = sizeof(int64_t); if (i < r) {- args[6+i*3] = &shape[perm[i]];- args[6+i*3+1] = &dst_strides[perm[i]];- args[6+i*3+2] = &src_strides[perm[i]];+ args[4+i*3] = &shape[perm[i]];+ args[4+i*3+1] = &dst_strides[perm[i]];+ args[4+i*3+2] = &src_strides[perm[i]]; } else {- args[6+i*3] = &zero;- args[6+i*3+1] = &zero;- args[6+i*3+2] = &zero;+ args[4+i*3] = &zero;+ args[4+i*3+1] = &zero;+ args[4+i*3+2] = &zero; } } const size_t w = 256; // XXX: hardcoded thread block size.
rts/cuda/prelude.cu view
@@ -128,7 +128,9 @@ #define INFINITY (1.0/0.0) #endif -extern volatile __shared__ unsigned char shared_mem[];+// shared_mem itself must be 8-byte aligned no matter what, in case we access it+// with atomics.+extern volatile __shared__ __attribute__((aligned(8))) unsigned char shared_mem[]; #define SHARED_MEM_PARAM #define FUTHARK_KERNEL extern "C" __global__ __launch_bounds__(MAX_THREADS_PER_BLOCK)
rts/javascript/server.js view
@@ -1,8 +1,8 @@ // Start of server.js -// The server is implemented directly on top of the C API exported by the-// WebAssembly module, much like the C server. The manifest describes the-// types and entry points, and names the C functions that operate on them.+// The server is implemented in terms of the C API exported by the WebAssembly+// module. The manifest describes the types and entry points, and names the C+// functions that operate on them. // // A variable is a pair of a type name and a value. A value of primitive type // is a JavaScript number, or a BigInt for 64-bit integers. Booleans are 0 or@@ -38,8 +38,13 @@ this.wasm = ctx.wasm; this.manifest = manifest; this._vars = {};+ this._output = []; } + _print(line) {+ this._output.push(line);+ }+ _get_arg(args, i) { if (i < args.length) { return args[i];@@ -99,18 +104,33 @@ // Calling into the C API. - _c(fname, ...args) {- return this.wasm['_' + fname](this.ctx.ctx, ...args);+ async _c(fname, ...args) {+ return await this.wasm.ccall(fname, 'number',+ Array(1 + args.length).fill('number'),+ [this.ctx.ctx, ...args], {async: true}); } + // Read a NUL-terminated string allocated by C, and free it.+ _c_string(p) {+ var heap = this.wasm.HEAPU8;+ // Browsers' TextDecoder rejects shared memory.+ var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));+ this.wasm._free(p);+ return str;+ }++ _get_error() {+ return this._c_string(this.wasm._futhark_context_get_error(this.ctx.ctx));+ }+ _check(err) { if (err != 0) {- throw this.ctx.get_error();+ throw this._get_error(); } } - _sync() {- this._check(this.wasm._futhark_context_sync(this.ctx.ctx));+ async _sync() {+ this._check(await this._c('futhark_context_sync')); } _malloc(n) {@@ -141,10 +161,10 @@ // Call 'f' with a pointer to space for one value of the given type, and // return the value that it stores there.- _with_out(tname, f) {+ async _with_out(tname, f) { var p = this._malloc(this._sizeof(tname)); try {- f(p);+ await f(p); return this._load(tname, p); } finally { this.wasm._free(p);@@ -152,9 +172,19 @@ } _bytes(p, n) {- return Buffer.from(this.wasm.HEAPU8.slice(p, p + n));+ return this.wasm.HEAPU8.slice(p, p + n); } + async _read_file(fname) {+ const fs = await import('node:fs');+ return new Uint8Array(fs.readFileSync(fname));+ }++ async _write_file(fname, bytes) {+ const fs = await import('node:fs');+ fs.writeFileSync(fname, bytes);+ }+ // Types. _type(tname) {@@ -210,16 +240,16 @@ return this._type(tname).sum; } - _free_value(tname, value) {+ async _free_value(tname, value) { var t = this._type(tname); if (t !== null) {- this._check(this._c(t.ops.free, value));+ this._check(await this._c(t.ops.free, value)); } } - _shape(a, arr) {+ async _shape(a, arr) { var fshape = a.ops ? a.ops.shape : a.shape;- var p = this._c(fshape, arr);+ var p = await this._c(fshape, arr); var shape = []; for (var i = 0; i < a.rank; i++) { shape.push(this._load('i64', p + i * 8));@@ -237,7 +267,7 @@ // Values. - _restore_value(reader, tname) {+ async _restore_value(reader, tname) { var t = this._type(tname); if (t === null) { return read_value(tname, reader);@@ -247,11 +277,11 @@ var p = this._malloc(bytes.length); try { this.wasm.HEAPU8.set(bytes, p);- var arr = this._c(t.ops.new, p, ...shape);+ var arr = await this._c(t.ops.new, p, ...shape); if (arr == 0) {- throw this.ctx.get_error();+ throw this._get_error(); }- this._sync();+ await this._sync(); return arr; } finally { this.wasm._free(p);@@ -263,22 +293,22 @@ var p = this._malloc(buff.length); try { this.wasm.HEAPU8.set(buff, p);- var obj = this._c(t.ops.restore, p);+ var obj = await this._c(t.ops.restore, p); if (obj == 0) {- throw this.ctx.get_error();+ throw this._get_error(); } } finally { this.wasm._free(p); }- var n = this._with_out('size', (np) => {- this._check(this._c(t.ops.store, obj, 0, np));+ var n = await this._with_out('size', async (np) => {+ this._check(await this._c(t.ops.store, obj, 0, np)); }); reader.buff = buff.slice(n); return obj; } } - _store_value(tname, value) {+ async _store_value(tname, value) { var t = this._type(tname); if (t === null) { var p = this._malloc(this._sizeof(tname));@@ -289,12 +319,12 @@ this.wasm._free(p); } } else if (t.kind == "array") {- var shape = this._shape(this._array_type(tname), value);+ var shape = await this._shape(this._array_type(tname), value); var n = shape.reduce((x, y) => x * y, 1n) * BigInt(this._sizeof(t.elemtype)); var p = this._malloc(Number(n)); try {- this._check(this._c(t.ops.values, value, p));- this._sync();+ this._check(await this._c(t.ops.values, value, p));+ await this._sync(); return binary_value(t.elemtype, shape, this._bytes(p, Number(n))); } finally { this.wasm._free(p);@@ -304,8 +334,8 @@ var np = pp + serverPointer.size; try { this._poke('ptr', pp, 0);- this._check(this._c(t.ops.store, value, pp, np));- this._sync();+ this._check(await this._c(t.ops.store, value, pp, np));+ await this._sync(); var p = this._load('ptr', pp); var bytes = this._bytes(p, this._load('size', np)); this.wasm._free(p);@@ -321,23 +351,35 @@ _cmd_inputs(args) { var entry = this._get_entry_point(this._get_arg(args, 0)); for (var input of entry.inputs) {- console.log((input.consumed ? "*" : "") + input.type);+ this._print((input.consumed ? "*" : "") + input.type); } } _cmd_output(args) { var entry = this._get_entry_point(this._get_arg(args, 0));- console.log((entry.output.fresh ? "*" : "") + entry.output.type);+ this._print((entry.output.fresh ? "*" : "") + entry.output.type); } - _cmd_dummy(args) {- // pass+ async _cmd_clear(args) {+ this._check(await this._c('futhark_context_clear_caches')); } - _cmd_free(args) {+ async _cmd_pause_profiling(args) {+ await this._c('futhark_context_pause_profiling');+ }++ async _cmd_unpause_profiling(args) {+ await this._c('futhark_context_unpause_profiling');+ }++ async _cmd_report(args) {+ this._print(this._c_string(await this._c('futhark_context_report')));+ }++ async _cmd_free(args) { for (var vname of args) { var v = this._get_var(vname);- this._free_value(v.type, v.value);+ await this._free_value(v.type, v.value); delete this._vars[vname]; } }@@ -353,27 +395,27 @@ _cmd_types(args) { for (var t in serverPrimtypes) {- console.log(t);+ this._print(t); } for (var t in this.manifest.types) {- console.log(t);+ this._print(t); } } _cmd_entry_points(args) { for (var e in this.manifest.entry_points) {- console.log(e);+ this._print(e); } } _cmd_attributes(args) { var entry = this._get_entry_point(this._get_arg(args, 0)); for (var attr of entry.attributes) {- console.log(attr);+ this._print(attr); } } - _cmd_call(args) {+ async _cmd_call(args) { var entry = this._get_entry_point(this._get_arg(args, 0)); var out_vname = this._get_arg(args, 1); var in_vnames = args.slice(2);@@ -384,39 +426,39 @@ this._check_new_var(out_vname); var bef = performance.now() * 1000;- var out = this._with_out(entry.output.type, (p) => {- this._check(this._c(entry.cfun, p, ...ins));- this._sync();+ var out = await this._with_out(entry.output.type, async (p) => {+ this._check(await this._c(entry.cfun, p, ...ins));+ await this._sync(); }); var aft = performance.now() * 1000;- console.log("runtime: " + Math.round(aft - bef));+ this._print("runtime: " + Math.round(aft - bef)); this._set_var(out_vname, entry.output.type, out); } - _cmd_store(args) {+ async _cmd_store(args) { var fname = this._get_arg(args, 0); var bufs = []; for (var vname of args.slice(1)) { var v = this._get_var(vname);- bufs.push(this._store_value(v.type, v.value));+ bufs.push(await this._store_value(v.type, v.value)); }- require("fs").writeFileSync(fname, Buffer.concat(bufs));+ await this._write_file(fname, concat_bytes(bufs)); } - _cmd_restore(args) {+ async _cmd_restore(args) { var fname = this._get_arg(args, 0); if (args.length % 2 == 0) { throw "Invalid argument count"; } - var reader = new Reader(fname);+ var reader = new Reader(await this._read_file(fname)); for (var i = 1; i < args.length; i += 2) { var vname = args[i]; var tname = args[i + 1]; this._check_new_var(vname); this._type(tname); try {- this._set_var(vname, tname, this._restore_value(reader, tname));+ this._set_var(vname, tname, await this._restore_value(reader, tname)); } catch (err) { throw "Failed to restore variable " + vname + ".\nPossibly malformed data in " + fname + ".\n" + err.toString(); }@@ -428,29 +470,29 @@ } _cmd_kind(args) {- console.log(this._kind(this._get_arg(args, 0)));+ this._print(this._kind(this._get_arg(args, 0))); } _cmd_type(args) {- console.log(this._get_var(this._get_arg(args, 0)).type);+ this._print(this._get_var(this._get_arg(args, 0)).type); } _cmd_rank(args) {- console.log(this._array_type(this._get_arg(args, 0)).rank);+ this._print(this._array_type(this._get_arg(args, 0)).rank); } _cmd_elemtype(args) {- console.log(this._array_type(this._get_arg(args, 0)).elemtype);+ this._print(this._array_type(this._get_arg(args, 0)).elemtype); } - _cmd_shape(args) {+ async _cmd_shape(args) { var v = this._get_var(this._get_arg(args, 0));- for (var d of this._shape(this._array_type(v.type), v.value)) {- console.log(d.toString());+ for (var d of await this._shape(this._array_type(v.type), v.value)) {+ this._print(d.toString()); } } - _cmd_new_array(args) {+ async _cmd_new_array(args) { var dst = this._get_arg(args, 0); var tname = this._get_arg(args, 1); this._check_new_var(dst);@@ -469,49 +511,49 @@ vs.forEach((v, i) => this._poke(a.elemtype, p + i * size, v)); var arr; if (a.ops) {- arr = this._c(a.ops.new, p, ...dims);+ arr = await this._c(a.ops.new, p, ...dims); if (arr == 0) {- throw this.ctx.get_error();+ throw this._get_error(); } } else {- arr = this._with_out(tname, (out) => {- this._check(this._c(a.new, out, p, ...dims));+ arr = await this._with_out(tname, async (out) => {+ this._check(await this._c(a.new, out, p, ...dims)); }); }- this._sync();+ await this._sync(); this._set_var(dst, tname, arr); } finally { this.wasm._free(p); } } - _cmd_set(args) {+ async _cmd_set(args) { var arr = this._get_var(this._get_arg(args, 0)); var a = this._array_type(arr.type); var val = this._get_typed_var(this._get_arg(args, 1), a.elemtype); var is = this._parse_ints(args.slice(2), a.rank);- this._check_bounds(this._shape(a, arr.value), is);+ this._check_bounds(await this._shape(a, arr.value), is); var fset = a.ops ? a.ops.set : a.set;- this._check(this._c(fset, arr.value, val, ...is));- this._sync();+ this._check(await this._c(fset, arr.value, val, ...is));+ await this._sync(); } - _cmd_index(args) {+ async _cmd_index(args) { var dst = this._get_arg(args, 0); var arr = this._get_var(this._get_arg(args, 1)); this._check_new_var(dst); var a = this._array_type(arr.type); var is = this._parse_ints(args.slice(2), a.rank);- this._check_bounds(this._shape(a, arr.value), is);+ this._check_bounds(await this._shape(a, arr.value), is); var findex = a.ops ? a.ops.index : a.index;- var v = this._with_out(a.elemtype, (p) => {- this._check(this._c(findex, p, arr.value, ...is));- this._sync();+ var v = await this._with_out(a.elemtype, async (p) => {+ this._check(await this._c(findex, p, arr.value, ...is));+ await this._sync(); }); this._set_var(dst, a.elemtype, v); } - _cmd_zip(args) {+ async _cmd_zip(args) { var dst = this._get_arg(args, 0); var tname = this._get_arg(args, 1); this._check_new_var(dst);@@ -524,14 +566,14 @@ throw a.fields.length + " arrays expected but " + vnames.length + " values provided."; } var vs = vnames.map((v, i) => this._get_typed_var(v, a.fields[i].type));- var arr = this._with_out(tname, (p) => {- this._check(this._c(a.zip, p, ...vs));- this._sync();+ var arr = await this._with_out(tname, async (p) => {+ this._check(await this._c(a.zip, p, ...vs));+ await this._sync(); }); this._set_var(dst, tname, arr); } - _cmd_unzip(args) {+ async _cmd_unzip(args) { var arr = this._get_var(this._get_arg(args, 0)); var a = this._array_type(arr.type); if (!a.fields) {@@ -542,22 +584,23 @@ throw a.fields.length + " arrays expected but " + dsts.length + " values provided."; } dsts.forEach((dst) => this._check_new_var(dst));- a.fields.forEach((f, i) => {- var v = this._with_out(f.type, (p) => {- this._check(this._c(f.project, p, arr.value));+ for (var i = 0; i < a.fields.length; i++) {+ var f = a.fields[i];+ var v = await this._with_out(f.type, async (p) => {+ this._check(await this._c(f.project, p, arr.value)); }); this._set_var(dsts[i], f.type, v);- });- this._sync();+ }+ await this._sync(); } _cmd_fields(args) { for (var f of this._record_type(this._get_arg(args, 0)).fields) {- console.log(f.name + " " + f.type);+ this._print(f.name + " " + f.type); } } - _cmd_new(args) {+ async _cmd_new(args) { var dst = this._get_arg(args, 0); var tname = this._get_arg(args, 1); this._check_new_var(dst);@@ -567,13 +610,13 @@ throw r.fields.length + " fields expected but " + vnames.length + " values provided."; } var vs = vnames.map((v, i) => this._get_typed_var(v, r.fields[i].type));- var obj = this._with_out(tname, (p) => {- this._check(this._c(r.new, p, ...vs));+ var obj = await this._with_out(tname, async (p) => {+ this._check(await this._c(r.new, p, ...vs)); }); this._set_var(dst, tname, obj); } - _cmd_project(args) {+ async _cmd_project(args) { var dst = this._get_arg(args, 0); var from = this._get_var(this._get_arg(args, 1)); var field = this._get_arg(args, 2);@@ -582,31 +625,31 @@ if (f === undefined) { throw "No such field: " + field; }- var v = this._with_out(f.type, (p) => {- this._check(this._c(f.project, p, from.value));+ var v = await this._with_out(f.type, async (p) => {+ this._check(await this._c(f.project, p, from.value)); }); this._set_var(dst, f.type, v); } _cmd_variants(args) { for (var v of this._sum_type(this._get_arg(args, 0)).variants) {- console.log(v.name);+ this._print(v.name); for (var t of v.payload) {- console.log("- " + t);+ this._print("- " + t); } } } - _variant_of(v) {+ async _variant_of(v) { var s = this._sum_type(v.type);- return s.variants[this._c(s.variant, v.value)];+ return s.variants[await this._c(s.variant, v.value)]; } - _cmd_variant(args) {- console.log(this._variant_of(this._get_var(this._get_arg(args, 0))).name);+ async _cmd_variant(args) {+ this._print((await this._variant_of(this._get_var(this._get_arg(args, 0)))).name); } - _cmd_construct(args) {+ async _cmd_construct(args) { var dst = this._get_arg(args, 0); var tname = this._get_arg(args, 1); var vname = this._get_arg(args, 2);@@ -620,15 +663,15 @@ throw variant.payload.length + " values expected but " + vnames.length + " provided."; } var vs = vnames.map((v, i) => this._get_typed_var(v, variant.payload[i]));- var obj = this._with_out(tname, (p) => {- this._check(this._c(variant.construct, p, ...vs));+ var obj = await this._with_out(tname, async (p) => {+ this._check(await this._c(variant.construct, p, ...vs)); }); this._set_var(dst, tname, obj); } - _cmd_destruct(args) {+ async _cmd_destruct(args) { var v = this._get_var(this._get_arg(args, 0));- var variant = this._variant_of(v);+ var variant = await this._variant_of(v); var dsts = args.slice(1); if (dsts.length != variant.payload.length) { throw variant.payload.length + " variables expected but " + dsts.length + " provided.";@@ -636,75 +679,82 @@ dsts.forEach((dst) => this._check_new_var(dst)); var ps = variant.payload.map((t) => this._malloc(this._sizeof(t))); try {- this._check(this._c(variant.destruct, ...ps, v.value));+ this._check(await this._c(variant.destruct, ...ps, v.value)); variant.payload.forEach((t, i) => this._set_var(dsts[i], t, this._load(t, ps[i]))); } finally { ps.forEach((p) => this.wasm._free(p)); } } - _process_line(line) {+ async _process_command(cmd, args) {+ switch (cmd) {+ case 'inputs': return this._cmd_inputs(args);+ case 'output': return this._cmd_output(args);+ case 'call': return this._cmd_call(args);+ case 'restore': return this._cmd_restore(args);+ case 'store': return this._cmd_store(args);+ case 'free': return this._cmd_free(args);+ case 'clear': return this._cmd_clear(args);+ case 'pause_profiling': return this._cmd_pause_profiling(args);+ case 'unpause_profiling': return this._cmd_unpause_profiling(args);+ case 'report': return this._cmd_report(args);+ case 'rename': return this._cmd_rename(args);+ case 'types': return this._cmd_types(args);+ case 'entry_points': return this._cmd_entry_points(args);+ case 'attributes': return this._cmd_attributes(args);+ case 'kind': return this._cmd_kind(args);+ case 'type': return this._cmd_type(args);+ case 'rank': return this._cmd_rank(args);+ case 'elemtype': return this._cmd_elemtype(args);+ case 'shape': return this._cmd_shape(args);+ case 'new_array': return this._cmd_new_array(args);+ case 'set': return this._cmd_set(args);+ case 'index': return this._cmd_index(args);+ case 'zip': return this._cmd_zip(args);+ case 'unzip': return this._cmd_unzip(args);+ case 'fields': return this._cmd_fields(args);+ case 'new': return this._cmd_new(args);+ case 'project': return this._cmd_project(args);+ case 'variants': return this._cmd_variants(args);+ case 'construct': return this._cmd_construct(args);+ case 'destruct': return this._cmd_destruct(args);+ case 'variant': return this._cmd_variant(args);+ default: throw "Unknown command: " + cmd;+ }+ }++ async _process_line(line) { var words = split_words(line); if (words.length == 0) { throw "Empty line";- } else {- var cmd = words[0];- var args = words.slice(1);- switch (cmd) {- case 'inputs': this._cmd_inputs(args); break;- case 'output': this._cmd_output(args); break;- case 'call': this._cmd_call(args); break;- case 'restore': this._cmd_restore(args); break;- case 'store': this._cmd_store(args); break;- case 'free': this._cmd_free(args); break;- case 'clear': this._cmd_dummy(args); break;- case 'pause_profiling': this._cmd_dummy(args); break;- case 'unpause_profiling': this._cmd_dummy(args); break;- case 'report': this._cmd_dummy(args); break;- case 'rename': this._cmd_rename(args); break;- case 'types': this._cmd_types(args); break;- case 'entry_points': this._cmd_entry_points(args); break;- case 'attributes': this._cmd_attributes(args); break;- case 'kind': this._cmd_kind(args); break;- case 'type': this._cmd_type(args); break;- case 'rank': this._cmd_rank(args); break;- case 'elemtype': this._cmd_elemtype(args); break;- case 'shape': this._cmd_shape(args); break;- case 'new_array': this._cmd_new_array(args); break;- case 'set': this._cmd_set(args); break;- case 'index': this._cmd_index(args); break;- case 'zip': this._cmd_zip(args); break;- case 'unzip': this._cmd_unzip(args); break;- case 'fields': this._cmd_fields(args); break;- case 'new': this._cmd_new(args); break;- case 'project': this._cmd_project(args); break;- case 'variants': this._cmd_variants(args); break;- case 'construct': this._cmd_construct(args); break;- case 'destruct': this._cmd_destruct(args); break;- case 'variant': this._cmd_variant(args); break;- default: throw "Unknown command: " + cmd;- } }+ await this._process_command(words[0], words.slice(1)); } - run() {+ async run() { console.log('%%% OK'); // TODO figure out if flushing is neccesary for JS- const readline = require('readline');+ const readline = await import('node:readline'); const rl = readline.createInterface(process.stdin);- rl.on('line', (line) => {+ // Commands are asynchronous, so we must finish one before reading the+ // next.+ for await (const line of rl) { if (line == "") {- rl.close();- return;+ break; }+ this._output = []; try {- this._process_line(line);+ await this._process_line(line);+ this._output.forEach((l) => console.log(l)); console.log('%%% OK'); } catch (err) { console.log('%%% FAILURE');+ this._output.forEach((l) => console.log(l)); console.log(err); console.log('%%% OK'); }- }).on('close', () => { process.exit(0); });+ }+ this.ctx.free();+ process.exit(0); } } @@ -726,6 +776,142 @@ throw "Unterminated quote"; } return words;+}++// Run a server as the main program of a Node.js process.+async function futhark_server_main() {+ const fs = await import('node:fs');+ const progname = process.argv[1];+ const fail = (msg) => {+ console.error(progname + ": " + msg);+ process.exit(1);+ };+ const parse_int = (what, x) => {+ if (!/^[0-9]+$/.test(x)) {+ fail("Invalid " + what + ": " + x);+ }+ return Number(x);+ };++ const options = { tuning_params: {} };+ var print_params = false;+ // Long name, short name, argument, description, and action.+ const table = [+ ["debugging", "D", null,+ "Perform possibly expensive internal correctness checks and verbose logging.",+ () => { options.debugging = true; }],+ ["log", "L", null,+ "Print various low-overhead logging information while running.",+ () => { options.logging = true; }],+ ["profile", "P", null,+ "Enable the collection of profiling information.",+ () => { options.profiling = true; }],+ ["help", "h", null,+ "Print help information and exit.",+ () => {+ console.log("Usage: " + progname + " [OPTIONS]...\nOptions:\n");+ for (const [long, short, arg, desc] of table) {+ console.log(" " + (short ? "-" + short + "/" : "") + "--" + long ++ (arg ? " " + arg : "") + "\n " + desc);+ }+ console.log("\nFor more information, consult the Futhark User's Guide or the man pages.");+ process.exit(0);+ }],+ ["print-params", null, null,+ "Print all tuning parameters that can be set with --param or --tuning.",+ () => { print_params = true; }],+ ["param", null, "ASSIGNMENT",+ "Set a tuning parameter to the given value.",+ (x) => {+ const eq = x.indexOf("=");+ if (eq < 0) {+ fail("Invalid argument for --param option: " + x);+ }+ options.tuning_params[x.slice(0, eq)] = parse_int("tuning parameter value", x.slice(eq + 1));+ }],+ ["tuning", null, "FILE",+ "Read size=value assignments from the given file.",+ (fname) => {+ var lines;+ try {+ lines = fs.readFileSync(fname, "utf8").split("\n");+ } catch (e) {+ fail("When loading tuning file '" + fname + "': Cannot open file: " + e.message);+ }+ lines.forEach((line, i) => {+ if (/^[ \t]*($|--)/.test(line)) {+ return;+ }+ const m = /^([^=]*)=(-?[0-9]+)$/.exec(line);+ if (m === null) {+ fail("When loading tuning file '" + fname + "': Invalid line " + (i + 1) ++ " (must be of form 'name=int').");+ }+ options.tuning_params[m[1]] = Number(m[2]);+ });+ }],+ // There is nothing to cache.+ ["cache-file", null, "FILE",+ "Store program cache here (has no effect for this backend).",+ () => {}],+ ];+ if (futhark_manifest.backend == "wasm_multicore") {+ table.push(["num-threads", null, "INT",+ "Set number of threads used for execution.",+ (x) => { options.num_threads = parse_int("number of threads", x); }]);+ }++ const args = process.argv.slice(2);+ for (var i = 0; i < args.length; i++) {+ const a = args[i];+ if (a.startsWith("--")) {+ const eq = a.indexOf("=");+ const name = eq < 0 ? a.slice(2) : a.slice(2, eq);+ const opt = table.find((o) => o[0] == name);+ if (opt === undefined) {+ fail("unrecognized option '" + a + "'");+ }+ var arg;+ if (opt[2] !== null) {+ arg = eq < 0 ? args[++i] : a.slice(eq + 1);+ if (arg === undefined) {+ fail("option '--" + name + "' requires an argument");+ }+ } else if (eq >= 0) {+ fail("option '--" + name + "' doesn't allow an argument");+ }+ opt[4](arg);+ } else if (a.startsWith("-") && a.length > 1) {+ for (const c of a.slice(1)) {+ const opt = table.find((o) => o[1] == c);+ if (opt === undefined) {+ fail("invalid option -- '" + c + "'");+ }+ opt[4]();+ }+ } else {+ fail("unexpected argument: " + a);+ }+ }++ if (print_params) {+ const m = await loadWASM();+ const heap = () => m.HEAPU8;+ const str = (p) => new TextDecoder().decode(heap().slice(p, heap().indexOf(0, p)));+ for (var j = 0; j < m._futhark_get_tuning_param_count(); j++) {+ console.log(str(m._futhark_get_tuning_param_name(j)) + " (" ++ str(m._futhark_get_tuning_param_class(j)) + ")");+ }+ process.exit(0);+ }++ const context = new FutharkContext();+ try {+ await context.init(options);+ } catch (e) {+ fail(e.message);+ }+ await new Server(context, futhark_manifest).run(); } // End of server.js
rts/javascript/values.js view
@@ -1,4 +1,8 @@ // Start of values.js+//+// Reading and writing values in the binary data format. Values are read from+// and written to byte buffers (Uint8Arrays).+ var futharkPrimtypes = new Set([ 'i8',@@ -29,19 +33,6 @@ 'bool' : Uint8Array }; -function binToStringArray(buff, array) {- for (var i = 0; i < array.length; i++) {- array[i] = buff[i];- }-}--function fileToBuff(fname) {- var readline = require('readline');- var fs = require('fs');- var buff = fs.readFileSync(fname);- return buff;-}- var typToSize = { "bool" : 1, " u8" : 1,@@ -61,6 +52,16 @@ return new Uint8Array(ta.buffer, ta.byteOffset, ta.byteLength); } +function concat_bytes(arrs) {+ var out = new Uint8Array(arrs.reduce((n, a) => n + a.length, 0));+ var off = 0;+ for (var a of arrs) {+ out.set(a, off);+ off += a.length;+ }+ return out;+}+ // The binary encoding of a value with the given primitive type and shape (a // list of BigInts), where 'bytes' is the payload. function binary_value(typ, shape, bytes) {@@ -73,65 +74,59 @@ header[3+i] = ftype.charCodeAt(i); } var dims = new BigInt64Array(shape);- return Buffer.concat([header, toU8(dims), bytes]);+ return concat_bytes([header, toU8(dims), bytes]); } +function is_space(b) {+ return b == 32 || (b >= 9 && b <= 13);+}+ class Reader {- constructor(f) {- this.f = f;- this.buff = fileToBuff(f);+ constructor(buff) {+ this.buff = buff; } - read_bin_array(num_dim, typ) {- var u8_array = new Uint8Array(num_dim * 8);- binToStringArray(this.buff.slice(0, num_dim * 8), u8_array);- var shape = new BigInt64Array(u8_array.buffer);- var length = shape[0];- for (var i = 1; i < shape.length; i++) {- length = length * shape[i];+ // Remove the first n bytes from the buffer, returning a copy of them.+ take(n) {+ if (this.buff.length < n) {+ throw "Unexpected end of input"; }- length = Number(length);- var dbytes = typToSize[typ];- var u8_data = new Uint8Array(length * dbytes);- binToStringArray(this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length), u8_data);- var data = new (typToType[typ])(u8_data.buffer);- var tmp_buff = this.buff.slice(num_dim * 8, num_dim * 8 + dbytes * length);- this.buff = this.buff.slice(num_dim * 8 + dbytes * length);- return [shape, data];+ var bytes = this.buff.slice(0, n);+ this.buff = this.buff.subarray(n);+ return bytes; } + read_bin_array(num_dim, typ) {+ var shape = new BigInt64Array(this.take(num_dim * 8).buffer);+ var length = Number(shape.reduce((x, y) => x * y, 1n));+ var data = new (typToType[typ])(this.take(length * typToSize[typ]).buffer);+ return [shape, data];+ }+ read_bin_scalar(typ) {- var size = typToSize[typ];- var u8_array = new Uint8Array(size);- binToStringArray(this.buff, u8_array);- var array = new (typToType[typ])(u8_array.buffer);- this.buff = this.buff.slice(u8_array.length); // Update buff to be unread part of the string- return array[0];+ return new (typToType[typ])(this.take(typToSize[typ]).buffer)[0]; } skip_spaces() {- while (this.buff.length > 0 && this.buff.slice(0, 1).toString().trim() == "") {- this.buff = this.buff.slice(1);+ while (this.buff.length > 0 && is_space(this.buff[0])) {+ this.buff = this.buff.subarray(1); } } read_binary(typename, dim) {- // Skip leading white space- while (this.buff.slice(0, 1).toString().trim() == "") {- this.buff = this.buff.slice(1);- }- if (this.buff[0] != 'b'.charCodeAt(0)) {+ this.skip_spaces();+ var header = this.take(7);+ if (header[0] != 'b'.charCodeAt(0)) { throw "Not in binary format" }- var version = this.buff[1];+ var version = header[1]; if (version != 2) { throw "Not version 2"; }- var num_dim = this.buff[2];- var typ = this.buff.slice(3, 7);- this.buff = this.buff.slice(7);+ var num_dim = header[2];+ var typ = String.fromCharCode(...header.subarray(3, 7)); var exp_typ = "[]".repeat(dim) + typename;- var given_typ = "[]".repeat(num_dim) + typ.toString().trim();+ var given_typ = "[]".repeat(num_dim) + typ.trim(); if (exp_typ !== given_typ) { throw ("Expected type : " + exp_typ + ", Actual type : " + given_typ); }
− rts/javascript/wrapperclasses.js
@@ -1,91 +0,0 @@-// Start of wrapperclasses.js--class FutharkArray {- constructor(ctx, ptr, type_name, dim, heap, fshape, fvalues, ffree) {- this.ctx = ctx;- this.ptr = ptr;- this.type_name = type_name;- this.dim = dim;- this.heap = heap;- this.fshape = fshape;- this.fvalues = fvalues;- this.ffree = ffree;- this.valid = true;- }-- validCheck() {- if (!this.valid) {- throw "Using freed memory"- }- }-- futharkType() {- return this.type_name;- }-- free() {- this.validCheck();- this.ffree(this.ctx.ctx, this.ptr);- this.valid = false;- }-- shape() {- this.validCheck();- var s = this.fshape(this.ctx.ctx, this.ptr) >> 3;- return this.ctx.wasm.HEAP64.subarray(s, s + this.dim);- }-- // get_shape() {- // return BigInt64Array.from(this.shape());- // }-- toTypedArray(dims = this.shape()) {- this.validCheck();- console.assert(dims.length === this.dim, "dim=%s,dims=%s", this.dim, dims.toString());- var length = Number(dims.reduce((a, b) => a * b));- var v = this.fvalues(this.ctx.ctx, this.ptr) / this.heap.BYTES_PER_ELEMENT;- return this.heap.subarray(v, v + length);- }-- toArray() {- this.validCheck();- var dims = this.shape();- var ta = this.toTypedArray(dims);- return (function nest(offs, ds) {- var d0 = Number(ds[0]);- if (ds.length === 1) {- return Array.from(ta.subarray(offs, offs + d0));- } else {- var d1 = Number(ds[1]);- return Array.from(Array(d0), (x,i) => nest(offs + i * d1, ds.slice(1)));- }- })(0, dims);- }-- async values() {- return this.toTypedArray();- }-}--class FutharkOpaque {- constructor(ctx, ptr, ffree) {- this.ctx = ctx;- this.ptr = ptr;- this.ffree = ffree;- this.valid = true;- }-- validCheck() {- if (!this.valid) {- throw "Using freed memory"- }- }-- free() {- this.validCheck();- this.ffree(this.ctx.ctx, this.ptr);- this.valid = false;- }-}--// End of wrapperclasses.js
+ rts/javascript/wrappers.js view
@@ -0,0 +1,567 @@+// Start of wrappers.js+//+// The JavaScript API (docs/js-api.rst), driven by 'futhark_manifest'.+//+// Asynchronous operations run one at a time, as Asyncify (used by WebGPU)+// allows no other calls into the module while one is suspended.++// f16 values and their bit patterns, rounding to nearest even.+function futhark_f16_bits(x) {+ if (Number.isNaN(x)) {+ return 0x7e00;+ }+ var sign = (x < 0 || Object.is(x, -0)) ? 0x8000 : 0;+ x = Math.abs(x);+ if (x < 2 ** -14) {+ // Subnormal or zero.+ return sign | futhark_round_even(x * 2 ** 24);+ }+ var e = Math.floor(Math.log2(x));+ // Math.log2 may be inexact.+ if (2 ** e > x) {+ e--;+ } else if (2 ** (e + 1) <= x) {+ e++;+ }+ var m = futhark_round_even((x / 2 ** e - 1) * 1024);+ if (m == 1024) {+ m = 0;+ e++;+ }+ if (e > 15) {+ return sign | 0x7c00;+ }+ return sign | ((e + 15) << 10) | m;+}++function futhark_round_even(x) {+ var r = Math.floor(x);+ var d = x - r;+ return (d > 0.5 || (d == 0.5 && r % 2 == 1)) ? r + 1 : r;+}++function futhark_f16_value(b) {+ var sign = (b & 0x8000) ? -1 : 1;+ var e = (b >> 10) & 0x1f;+ var m = b & 0x3ff;+ if (e == 0) {+ return sign * m * 2 ** -24;+ } else if (e == 31) {+ return m == 0 ? sign * Infinity : NaN;+ } else {+ return sign * (1 + m / 1024) * 2 ** (e - 15);+ }+}++function futhark_check_int(x, bits, signed, what) {+ var ok;+ if (typeof x == "bigint") {+ ok = (signed ? BigInt.asIntN(bits, x) : BigInt.asUintN(bits, x)) == x;+ } else if (typeof x == "number") {+ var lo = signed ? -(2 ** (bits - 1)) : 0;+ var hi = signed ? 2 ** (bits - 1) : 2 ** bits;+ ok = Number.isInteger(x) && x >= lo && x < hi;+ } else {+ throw new TypeError(what + " must be a number or a BigInt");+ }+ if (!ok) {+ throw new RangeError(what + " is not a " + (signed ? "i" : "u") + bits + " value: " + x);+ }+}++// 'array' is used for values(), and 'c_array' for the C representation.+function futhark_int_prim(bits, signed, array) {+ var big = bits == 64;+ return {+ size: bits / 8,+ array: array,+ c_array: array,+ to_c: (x, what) => {+ futhark_check_int(x, bits, signed, what);+ return big ? BigInt(x) : Number(x);+ },+ get: (dv, p) => big+ ? (signed ? dv.getBigInt64(p, true) : dv.getBigUint64(p, true))+ : dv["get" + (signed ? "Int" : "Uint") + bits](p, true),+ to_values: (bytes) => new array(bytes.buffer),+ };+}++function futhark_float_prim(bits, array) {+ return {+ size: bits / 8,+ array: array,+ c_array: array,+ to_c: (x, what) => {+ if (typeof x != "number") {+ throw new TypeError(what + " must be a number");+ }+ return x;+ },+ get: (dv, p) => dv["getFloat" + bits](p, true),+ to_values: (bytes) => new array(bytes.buffer),+ };+}++// Float32Array represents all f16 values exactly.+var futhark_f16_array = globalThis.Float16Array ?? Float32Array;++var futhark_prims = {+ i8: futhark_int_prim(8, true, Int8Array),+ i16: futhark_int_prim(16, true, Int16Array),+ i32: futhark_int_prim(32, true, Int32Array),+ i64: futhark_int_prim(64, true, BigInt64Array),+ u8: futhark_int_prim(8, false, Uint8Array),+ u16: futhark_int_prim(16, false, Uint16Array),+ u32: futhark_int_prim(32, false, Uint32Array),+ u64: futhark_int_prim(64, false, BigUint64Array),+ f16: {+ size: 2,+ array: futhark_f16_array,+ c_array: Uint16Array,+ to_c: (x, what) => futhark_f16_bits(futhark_prims.f32.to_c(x, what)),+ get: (dv, p) => futhark_f16_value(dv.getUint16(p, true)),+ to_values: (bytes) =>+ futhark_f16_array.from(new Uint16Array(bytes.buffer), futhark_f16_value),+ },+ f32: futhark_float_prim(32, Float32Array),+ f64: futhark_float_prim(64, Float64Array),+ bool: {+ size: 1,+ array: Uint8Array,+ c_array: Uint8Array,+ to_c: (x, what) => {+ if (typeof x != "boolean") {+ throw new TypeError(what + " must be a boolean");+ }+ return x ? 1 : 0;+ },+ get: (dv, p) => dv.getUint8(p) != 0,+ to_values: (bytes) => bytes,+ },+};++// Booleans may also be 0 or 1, as returned by values().+function futhark_from_data(type, prim, data) {+ if (data instanceof prim.c_array && prim.c_array === prim.array && type != "bool") {+ return data;+ }+ var out = new prim.c_array(data.length);+ for (var i = 0; i < data.length; i++) {+ var x = data[i];+ if (type == "bool" && (x === 0 || x === 1)) {+ x = x === 1;+ }+ out[i] = prim.to_c(x, "Element " + i + " of the array data");+ }+ return out;+}++function futhark_check_value(obj, cls, ctx, type, what) {+ if (!(obj instanceof cls)) {+ throw new TypeError(what + " must be a " + cls.name);+ }+ if (obj.ctx !== ctx) {+ throw new Error(what + " belongs to a different FutharkContext");+ }+ if (obj.type !== type) {+ throw new TypeError(what + " has type " + obj.type + ", but must have type " + type);+ }+ obj._check_usable();+}++class FutharkValue {+ constructor(ctx, type, ptr) {+ this.ctx = ctx;+ this.type = type;+ this.ptr = ptr;+ this.consumed = false;+ }++ _check_usable() {+ this._check_not_freed();+ if (this.consumed) {+ throw new Error("Using a value that has been consumed by an entry point");+ }+ }++ _check_not_freed() {+ if (this.ptr === undefined) {+ throw new Error("Using a value that has been freed");+ }+ this.ctx._check_usable();+ }++ free() {+ this._check_not_freed();+ this.ctx._check_idle();+ this.ctx.wasm["_" + this.ctx.manifest.types[this.type].ops.free](this.ctx.ctx, this.ptr);+ this.ptr = undefined;+ }+}++class FutharkArray extends FutharkValue {+ shape() {+ this._check_usable();+ this.ctx._check_idle();+ return this._shape();+ }++ _shape() {+ var info = this.ctx.manifest.types[this.type];+ var p = this.ctx.wasm["_" + info.ops.shape](this.ctx.ctx, this.ptr);+ return new BigInt64Array(this.ctx.wasm.HEAPU8.slice(p, p + 8 * info.rank).buffer);+ }++ values() {+ var ctx = this.ctx;+ return ctx._exclusive(async () => {+ this._check_usable();+ var info = ctx.manifest.types[this.type];+ var prim = futhark_prims[info.elemtype];+ var bytes = Number(this._shape().reduce((a, b) => a * b, 1n)) * prim.size;+ var p = ctx._malloc(bytes);+ try {+ ctx._check(await ctx._call(info.ops.values, ctx.ctx, this.ptr, p));+ await ctx._sync();+ return prim.to_values(ctx.wasm.HEAPU8.slice(p, p + bytes));+ } finally {+ ctx.wasm._free(p);+ }+ });+ }+}++class FutharkOpaque extends FutharkValue {+}++class FutharkContext {+ constructor() {+ this.manifest = futhark_manifest;+ this.wasm = undefined;+ this.cfg = undefined;+ this.ctx = undefined;+ this.initialised = false;+ this.freed = false;+ this.queue = Promise.resolve();+ this.running = false;+ this.entry = Object.create(null);+ this.types = Object.create(null);+ }++ async init(options = {}) {+ if (this.initialised) {+ throw new Error("FutharkContext.init() may only be called once");+ }+ this.initialised = true;+ const flags = ["debugging", "logging", "profiling"];+ for (const option of Object.keys(options)) {+ if (option != "num_threads" && option != "tuning_params" && !flags.includes(option)) {+ throw new Error("Unknown FutharkContext.init() option: " + option);+ }+ }+ for (const flag of flags) {+ if (options[flag] !== undefined && typeof options[flag] != "boolean") {+ throw new TypeError("The " + flag + " option must be a boolean");+ }+ }+ if (options.num_threads !== undefined) {+ futhark_check_int(options.num_threads, 32, true, "The num_threads option");+ }+ const tuning_params = Object.entries(options.tuning_params ?? {});+ for (const [name, value] of tuning_params) {+ futhark_check_int(value, 32, false, "Tuning parameter " + name);+ }++ if (this.manifest.backend == "wasm_multicore" && globalThis.SharedArrayBuffer === undefined) {+ throw new Error("The wasm-multicore backend requires SharedArrayBuffer, which browsers " ++ "only provide to pages that are cross-origin isolated");+ }+ try {+ this.wasm = await loadWASM();+ } catch (e) {+ throw e instanceof Error ? e : new Error("Failed to load the WebAssembly module: " + e);+ }+ if (options.num_threads !== undefined &&+ this.wasm._futhark_context_config_set_num_threads === undefined) {+ throw new Error("The num_threads option is not supported by this backend");+ }++ this.cfg = this.wasm._futhark_context_config_new();+ for (const flag of flags) {+ if (options[flag]) {+ this.wasm["_futhark_context_config_set_" + flag](this.cfg, 1);+ }+ }+ if (options.num_threads !== undefined) {+ this.wasm._futhark_context_config_set_num_threads(this.cfg, options.num_threads);+ }+ for (const [name, value] of tuning_params) {+ if (this.wasm.ccall("futhark_context_config_set_tuning_param", "number",+ ["number", "string", "number"], [this.cfg, name, value]) != 0) {+ this.free();+ throw new Error("Unknown tuning parameter: " + name);+ }+ }+ var ctx = await this._call("futhark_context_new", this.cfg);+ var err = this._get_error(ctx);+ if (err !== null) {+ this.wasm._futhark_context_free(ctx);+ this.free();+ throw new Error(err);+ }+ this.ctx = ctx;++ for (const [name, info] of Object.entries(this.manifest.types)) {+ if (info.kind == "array") {+ const constructor = {+ from_data: (data, ...shape) => this._new_array(name, data, shape),+ };+ this.types[name] = constructor;+ this[info.elemtype + "_" + info.rank + "d"] = constructor;+ } else if (info.record !== undefined) {+ for (const field of info.record.fields) {+ this[field.project.replace(/^futhark_/, "")] =+ (obj) => this._exclusive(() => this._project(name, field, obj));+ }+ }+ }+ for (const [name, info] of Object.entries(this.manifest.entry_points)) {+ this.entry[name] = (...args) => this._exclusive(() => this._call_entry(name, info, args));+ }+ }++ free() {+ this._check_idle();+ if (this.ctx !== undefined) {+ this.wasm._futhark_context_free(this.ctx);+ this.ctx = undefined;+ }+ if (this.cfg !== undefined) {+ this.wasm._futhark_context_config_free(this.cfg);+ this.cfg = undefined;+ }+ this.freed = true;+ }++ context_sync() {+ return this._exclusive(async () => {+ this._check_usable();+ await this._sync();+ });+ }++ clear_caches() {+ return this._exclusive(async () => {+ this._check_usable();+ this._check(await this._call("futhark_context_clear_caches", this.ctx));+ });+ }++ report() {+ return this._exclusive(async () => {+ this._check_usable();+ return this._c_string(await this._call("futhark_context_report", this.ctx));+ });+ }++ pause_profiling() {+ return this._exclusive(async () => {+ this._check_usable();+ await this._call("futhark_context_pause_profiling", this.ctx);+ });+ }++ unpause_profiling() {+ return this._exclusive(async () => {+ this._check_usable();+ await this._call("futhark_context_unpause_profiling", this.ctx);+ });+ }++ // Everything below is internal.++ _check_usable() {+ if (this.freed) {+ throw new Error("The FutharkContext has been freed");+ }+ if (this.ctx === undefined) {+ throw new Error("The FutharkContext has not been initialised");+ }+ }++ // Run 'f' after all previously started asynchronous operations.+ _exclusive(f) {+ var run = this.queue.then(async () => {+ this.running = true;+ try {+ return await f();+ } finally {+ this.running = false;+ }+ });+ this.queue = run.catch(() => {});+ return run;+ }++ _check_idle() {+ if (this.running) {+ throw new Error("Cannot do this while an asynchronous operation on the FutharkContext is running");+ }+ }++ async _call(cfun, ...args) {+ return await this.wasm.ccall(cfun, "number", args.map(() => "number"), args, {async: true});+ }++ _malloc(n) {+ // malloc(0) may return NULL.+ var p = this.wasm._malloc(Math.max(n, 1));+ if (p == 0) {+ throw new RangeError("Failed to allocate " + n + " bytes");+ }+ return p;+ }++ _c_string(p) {+ var heap = this.wasm.HEAPU8;+ // Browsers' TextDecoder rejects shared memory.+ var str = new TextDecoder().decode(heap.slice(p, heap.indexOf(0, p)));+ this.wasm._free(p);+ return str;+ }++ _get_error(ctx = this.ctx) {+ var p = this.wasm._futhark_context_get_error(ctx);+ return p == 0 ? null : this._c_string(p);+ }++ _error() {+ return new Error(this._get_error() ?? "Unknown error");+ }++ _check(ret) {+ if (ret != 0) {+ throw this._error();+ }+ }++ async _sync() {+ this._check(await this._call("futhark_context_sync", this.ctx));+ }++ _to_c(type, x, what) {+ if (type in futhark_prims) {+ return futhark_prims[type].to_c(x, what);+ }+ var cls = this.manifest.types[type].kind == "array" ? FutharkArray : FutharkOpaque;+ futhark_check_value(x, cls, this, type, what);+ return x.ptr;+ }++ _from_c(type, p) {+ var dv = new DataView(this.wasm.HEAPU8.buffer);+ if (type in futhark_prims) {+ return futhark_prims[type].get(dv, p);+ }+ var ptr = dv.getUint32(p, true);+ if (this.manifest.types[type].kind == "array") {+ return new FutharkArray(this, type, ptr);+ } else {+ return new FutharkOpaque(this, type, ptr);+ }+ }++ _free_c(type, p) {+ if (!(type in futhark_prims)) {+ var ptr = new DataView(this.wasm.HEAPU8.buffer).getUint32(p, true);+ this.wasm["_" + this.manifest.types[type].ops.free](this.ctx, ptr);+ }+ }++ // For C functions with an output parameter after the context.+ async _call_with_output(cfun, type, args, after_call) {+ var out = this._malloc(8);+ try {+ var ret = await this._call(cfun, this.ctx, out, ...args);+ after_call();+ this._check(ret);+ try {+ await this._sync();+ } catch (e) {+ this._free_c(type, out);+ throw e;+ }+ return this._from_c(type, out);+ } finally {+ this.wasm._free(out);+ }+ }++ _new_array(type, data, shape) {+ this._check_usable();+ this._check_idle();+ var info = this.manifest.types[type];+ var prim = futhark_prims[info.elemtype];+ if (shape.length != info.rank) {+ throw new TypeError("Expected " + info.rank + " dimensions, but got " + shape.length);+ }+ shape = shape.map((d, i) => {+ futhark_check_int(d, 64, true, "dimension " + i);+ if (d < 0) {+ throw new RangeError("dimension " + i + " is negative: " + d);+ }+ return BigInt(d);+ });+ if (!(data instanceof Array || (ArrayBuffer.isView(data) && !(data instanceof DataView)))) {+ throw new TypeError("Array data must be an Array or a typed array");+ }+ var n = shape.reduce((a, b) => a * b, 1n);+ if (BigInt(data.length) != n) {+ throw new RangeError("Array data has " + data.length + " elements, but the shape requires " + n);+ }+ var elems = futhark_from_data(info.elemtype, prim, data);+ var bytes = new Uint8Array(elems.buffer, elems.byteOffset, elems.byteLength);+ var p = this._malloc(bytes.length);+ try {+ this.wasm.HEAPU8.set(bytes, p);+ var ptr = this.wasm["_" + info.ops.new](this.ctx, p, ...shape);+ if (ptr == 0) {+ throw this._error();+ }+ return new FutharkArray(this, type, ptr);+ } finally {+ this.wasm._free(p);+ }+ }++ async _project(type, field, obj) {+ this._check_usable();+ futhark_check_value(obj, FutharkOpaque, this, type, "The record");+ return await this._call_with_output(field.project, field.type, [obj.ptr], () => {});+ }++ async _call_entry(name, info, args) {+ this._check_usable();+ if (args.length != info.inputs.length) {+ throw new TypeError("Entry point " + name + " expects " + info.inputs.length ++ " arguments, but got " + args.length);+ }+ var c_args = info.inputs.map((input, i) =>+ this._to_c(input.type, args[i], "Argument " + i + " of entry point " + name));+ info.inputs.forEach((input, i) => {+ if (input.consumed && args.some((x, j) => j != i && x === args[i])) {+ throw new Error("Argument " + i + " of entry point " + name ++ " is consumed, so it must not also be passed as another argument");+ }+ });+ var mark_consumed = () => info.inputs.forEach((input, i) => {+ if (input.consumed) {+ args[i].consumed = true;+ }+ });+ return await this._call_with_output(info.cfun, info.output.type, c_args, mark_consumed);+ }+}++// End of wrappers.js
rts/opencl/copy.cl view
@@ -2,9 +2,7 @@ #define GEN_COPY_KERNEL(NAME, ELEM_TYPE) \ FUTHARK_KERNEL void lmad_copy_##NAME(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \ int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \ int64_t src_offset, \ int64_t n, \ int r, \@@ -15,7 +13,9 @@ int64_t shape4, int64_t dst_stride4, int64_t src_stride4, \ int64_t shape5, int64_t dst_stride5, int64_t src_stride5, \ int64_t shape6, int64_t dst_stride6, int64_t src_stride6, \- int64_t shape7, int64_t dst_stride7, int64_t src_stride7) { \+ int64_t shape7, int64_t dst_stride7, int64_t src_stride7, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ int64_t gtid = get_global_id(0); \ int64_t remainder = gtid; \ \
rts/opencl/transpose.cl view
@@ -3,9 +3,7 @@ #define GEN_TRANSPOSE_KERNELS(NAME, ELEM_TYPE) \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\ void map_transpose_##NAME(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \ int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \ int64_t src_offset, \ int32_t num_arrays, \ int32_t x_elems, \@@ -17,7 +15,9 @@ int32_t mulx, \ int32_t muly, \ int32_t repeat_1, \- int32_t repeat_2) { \+ int32_t repeat_2, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ (void)mulx; (void)muly; \ __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \ int tblock_id_0 = get_tblock_id(0); \@@ -29,6 +29,8 @@ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ if (tblock_id_2 >= num_arrays) { break; } \+ /* The previous iteration may still be reading shared memory. */ \+ if (i1 > 0 || i2 > 0) { barrier_local(); } \ int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \ int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \ int32_t x_index = global_id_0; \@@ -66,21 +68,21 @@ \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1) \ void map_transpose_##NAME##_low_height(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \- int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \- int64_t src_offset, \- int32_t num_arrays, \- int32_t x_elems, \- int32_t y_elems, \- int32_t dst_array_stride, \- int32_t dst_row_stride, \- int32_t src_array_stride, \- int32_t src_row_stride, \- int32_t mulx, \- int32_t muly, \- int32_t repeat_1, \- int32_t repeat_2) { \+ int64_t dst_offset, \+ int64_t src_offset, \+ int32_t num_arrays, \+ int32_t x_elems, \+ int32_t y_elems, \+ int32_t dst_array_stride, \+ int32_t dst_row_stride, \+ int32_t src_array_stride, \+ int32_t src_row_stride, \+ int32_t mulx, \+ int32_t muly, \+ int32_t repeat_1, \+ int32_t repeat_2, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \ int tblock_id_0 = get_tblock_id(0); \ int global_id_0 = get_global_id(0); \@@ -91,6 +93,8 @@ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ if (tblock_id_2 >= num_arrays) { break; } \+ /* The previous iteration may still be reading shared memory. */ \+ if (i1 > 0 || i2 > 0) { barrier_local(); } \ int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \ int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \ int32_t x_index = \@@ -124,9 +128,7 @@ \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM, TR_BLOCK_DIM, 1) \ void map_transpose_##NAME##_low_width(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \ int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \ int64_t src_offset, \ int32_t num_arrays, \ int32_t x_elems, \@@ -138,7 +140,9 @@ int32_t mulx, \ int32_t muly, \ int32_t repeat_1, \- int32_t repeat_2) { \+ int32_t repeat_2, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \ int tblock_id_0 = get_tblock_id(0); \ int global_id_0 = get_global_id(0); \@@ -149,6 +153,8 @@ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ if (tblock_id_2 >= num_arrays) { break; } \+ /* The previous iteration may still be reading shared memory. */ \+ if (i1 > 0 || i2 > 0) { barrier_local(); } \ int32_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \ int32_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \ int32_t x_index = tblock_id_0 * TR_BLOCK_DIM + get_local_id(0)/muly; \@@ -178,10 +184,8 @@ } \ \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*TR_BLOCK_DIM, 1, 1) \-void map_transpose_##NAME##_small(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \+void map_transpose_##NAME##_small(SHARED_MEM_PARAM \ int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \ int64_t src_offset, \ int32_t num_arrays, \ int32_t x_elems, \@@ -193,7 +197,9 @@ int32_t mulx, \ int32_t muly, \ int32_t repeat_1, \- int32_t repeat_2) { \+ int32_t repeat_2, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ (void)mulx; (void)muly; \ __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \ int tblock_id_0 = get_tblock_id(0); \@@ -224,9 +230,7 @@ \ FUTHARK_KERNEL_SIZED(TR_BLOCK_DIM*2, TR_TILE_DIM/TR_ELEMS_PER_THREAD, 1)\ void map_transpose_##NAME##_large(SHARED_MEM_PARAM \- __global ELEM_TYPE *dst_mem, \ int64_t dst_offset, \- __global ELEM_TYPE *src_mem, \ int64_t src_offset, \ int64_t num_arrays, \ int64_t x_elems, \@@ -238,9 +242,11 @@ int64_t mulx, \ int64_t muly, \ int32_t repeat_1, \- int32_t repeat_2) { \+ int32_t repeat_2, \+ __global ELEM_TYPE *dst_mem, \+ __global ELEM_TYPE *src_mem) { \ (void)mulx; (void)muly; \- __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \+ __local ELEM_TYPE* block = (__local ELEM_TYPE*)shared_mem; \ int tblock_id_0 = get_tblock_id(0); \ int global_id_0 = get_global_id(0); \ int tblock_id_1 = get_tblock_id(1); \@@ -250,6 +256,8 @@ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ if (tblock_id_2 >= num_arrays) { break; } \+ /* The previous iteration may still be reading shared memory. */ \+ if (i1 > 0 || i2 > 0) { barrier_local(); } \ int64_t odata_offset = dst_offset + tblock_id_2 * dst_array_stride; \ int64_t idata_offset = src_offset + tblock_id_2 * src_array_stride; \ int64_t x_index = global_id_0; \
rts/python/opencl.py view
@@ -183,6 +183,14 @@ ) self.failure_is_an_option = np.int32(0) + self.bookkeeping_counters = self.pool.allocate(+ np.int32().itemsize * self.bookkeeping_counters_size+ )+ self.bookkeeping_locks = self.pool.allocate(+ np.int32().itemsize * self.bookkeeping_locks_size+ )+ zero_bookkeeping(self)+ if "default_group_size" in user_sizes: default_group_size = user_sizes["default_group_size"] del user_sizes["default_group_size"]@@ -389,6 +397,23 @@ self.pool.free_held() +def zero_bookkeeping(self):+ cl.enqueue_fill_buffer(+ self.queue,+ self.bookkeeping_counters,+ np.int32(0),+ 0,+ np.int32().itemsize * self.bookkeeping_counters_size,+ )+ cl.enqueue_fill_buffer(+ self.queue,+ self.bookkeeping_locks,+ np.int32(0),+ 0,+ np.int32().itemsize * self.bookkeeping_locks_size,+ )++ def sync(self): failure = np.empty(1, dtype=np.int32) cl.enqueue_copy(self.queue, failure, self.global_failure, is_blocking=True)@@ -403,6 +428,9 @@ np.int32().itemsize, ) + # The failing kernel may have left the bookkeeping buffers nonzero.+ zero_bookkeeping(self)+ # Read failure args. failure_args = np.empty( self.global_failure_args_max + 1, dtype=np.int64@@ -441,9 +469,7 @@ kernel.set_args( cl.LocalMemory(TR_TILE_DIM * (TR_TILE_DIM + 1) * elem_size),- dst, dst_offset,- src, src_offset, k32, m32,@@ -453,6 +479,8 @@ muly32, np.int32(0), np.int32(0),+ dst,+ src, ) cl.enqueue_nd_range_kernel(self.queue, kernel, dims, group_dims) @@ -490,13 +518,13 @@ kernel = self.copy_kernels[elem_size] kernel.set_args( cl.LocalMemory(1),- dst, dst_offset,- src, src_offset, n, np.int32(r), *layout_args,+ dst,+ src, ) w = 256 dims = ((n + w - 1) // w * w,)
rts/python/scalar.py view
@@ -409,6 +409,13 @@ return ssignum(x) +def sitofp_T_f16(x):+ return np.float16(x)+++sitofp_i8_f16 = sitofp_i16_f16 = sitofp_i32_f16 = sitofp_i64_f16 = sitofp_T_f16++ def sitofp_T_f32(x): return np.float32(x) @@ -423,6 +430,13 @@ sitofp_i8_f64 = sitofp_i16_f64 = sitofp_i32_f64 = sitofp_i64_f64 = sitofp_T_f64 +def uitofp_T_f16(x):+ return np.float36(unsigned(x))+++uitofp_i8_f16 = uitofp_i16_f16 = uitofp_i32_f16 = uitofp_i64_f16 = uitofp_T_f16++ def uitofp_T_f32(x): return np.float32(unsigned(x)) @@ -481,7 +495,7 @@ if np.isnan(x) or x < 0 or x >= (1 << 8): return np.int8(0) else:- return np.int8(np.trunc(x))+ return np.uint8(np.trunc(x)).astype(np.int8) fptoui_f16_i8 = fptoui_f32_i8 = fptoui_f64_i8 = fptoui_T_i8@@ -491,7 +505,7 @@ if np.isnan(x) or x < 0 or x >= (1 << 16): return np.int16(0) else:- return np.int16(np.trunc(x))+ return np.uint16(np.trunc(x)).astype(np.int16) fptoui_f16_i16 = fptoui_f32_i16 = fptoui_f64_i16 = fptoui_T_i16@@ -501,7 +515,7 @@ if np.isnan(x) or x < 0 or x >= (1 << 32): return np.int32(0) else:- return np.int32(np.trunc(x))+ return np.uint32(np.trunc(x)).astype(np.int32) fptoui_f16_i32 = fptoui_f32_i32 = fptoui_f64_i32 = fptoui_T_i32@@ -511,7 +525,7 @@ if np.isnan(x) or x < 0 or x >= (1 << 64): return np.int64(0) else:- return np.int64(np.trunc(x))+ return np.uint64(np.trunc(x)).astype(np.int64) fptoui_f16_i64 = fptoui_f32_i64 = fptoui_f64_i64 = fptoui_T_i64
src/Futhark/Actions.hs view
@@ -33,12 +33,14 @@ import Control.Monad import Control.Monad.IO.Class import Control.Monad.State (get)+import Data.Aeson qualified as JSON import Data.Bifunctor import Data.ByteString.Lazy.Char8 qualified as BS import Data.List (intercalate) import Data.Map qualified as M import Data.Maybe (fromMaybe) import Data.Text qualified as T+import Data.Text.Encoding qualified as T import Data.Text.IO qualified as T import Futhark.Analysis.AccessPattern import Futhark.Analysis.Alias@@ -49,6 +51,8 @@ import Futhark.Analysis.Metrics import Futhark.CodeGen.Backends.CCUDA qualified as CCUDA import Futhark.CodeGen.Backends.COpenCL qualified as COpenCL+import Futhark.CodeGen.Backends.GenericC qualified as GC+import Futhark.CodeGen.Backends.GenericWASM qualified as GenericWASM import Futhark.CodeGen.Backends.HIP qualified as HIP import Futhark.CodeGen.Backends.MulticoreC qualified as MulticoreC import Futhark.CodeGen.Backends.MulticoreISPC qualified as MulticoreISPC@@ -532,30 +536,46 @@ cmdEMCFLAGS :: [String] -> [String] cmdEMCFLAGS def = maybe def words $ lookup "EMCFLAGS" unixEnvironment -runEMCC :: String -> String -> FilePath -> [String] -> [String] -> [String] -> Bool -> FutharkM ()-runEMCC cpath outpath classpath cflags_def ldflags expfuns lib = do+-- | The typed views of the WASM heap, which the generated JavaScript+-- uses to move array data in and out of the module.+heapExports :: [String]+heapExports =+ [ "'HEAP8'",+ "'HEAP16'",+ "'HEAP32'",+ "'HEAP64'",+ "'HEAPU8'",+ "'HEAPU16'",+ "'HEAPU32'",+ "'HEAPU64'",+ "'HEAPF32'",+ "'HEAPF64'"+ ]++-- | Options passed to @emcc@ by both WASM backends.+wasmFlags :: FilePath -> [String] -> [String]+wasmFlags classpath expfuns =+ [ "--extern-post-js",+ classpath,+ -- Called by the generated JavaScript.+ "-sEXPORT_NAME=loadWASM",+ "-sWASM_BIGINT",+ -- Up to 2GiB; more would need unsigned pointers in JavaScript.+ "-sALLOW_MEMORY_GROWTH",+ "-sEXPORTED_RUNTIME_METHODS=[" ++ intercalate "," ("'ccall'" : heapExports) ++ "]",+ "-sEXPORTED_FUNCTIONS=[" ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns) ++ "]"+ ]++runEMCC :: String -> String -> [String] -> [String] -> [String] -> FutharkM ()+runEMCC cpath outpath cflags_def ldflags extra_flags = do ret <- liftIO $ runProgramWithExitCode "emcc" ( [cpath, "-o", outpath]- ++ ["-lnodefs.js"]- ++ ["-s", "--extern-post-js", classpath]- ++ ( if lib- then ["-s", "EXPORT_NAME=loadWASM"]- else []- )- ++ ["-s", "WASM_BIGINT"]- ++ [ "-s",- "EXPORTED_RUNTIME_METHODS=['HEAP8','HEAP16','HEAP32','HEAP64','HEAPU8','HEAPU16','HEAPU32','HEAPF32','HEAPF64']"- ]+ ++ extra_flags ++ cmdCFLAGS cflags_def ++ cmdEMCFLAGS [""]- ++ [ "-s",- "EXPORTED_FUNCTIONS=["- ++ intercalate "," ("'_malloc'" : "'_free'" : expfuns)- ++ "]"- ] -- The default LDFLAGS are always added. ++ ldflags )@@ -572,73 +592,63 @@ Right (ExitSuccess, _, _) -> pure () +-- | Compile to the ES module @outpath.mjs@ with emcc. Unless compiling a+-- library, @outpath@ is an executable that runs it as a server.+compileWASM ::+ CompilerMode ->+ FilePath ->+ [String] ->+ (GC.CParts, T.Text, [String]) ->+ FutharkM ()+compileWASM mode outpath ldflags (cprog, jsprog, exps) = do+ let (_, impl, _) = GC.asLibrary cprog+ liftIO $ T.writeFile cpath impl+ liftIO $ T.writeFile classpath $ jsprog <> extra+ runEMCC cpath mjspath ["-O3", "-msimd128"] ldflags (wasmFlags classpath exps)+ when (mode /= ToLibrary) . liftIO $ do+ T.writeFile outpath $ wasmLauncher $ takeFileName mjspath+ perms <- getPermissions outpath+ setPermissions outpath $ setOwnerExecutable True perms+ where+ extra = case mode of+ ToLibrary -> GenericWASM.libraryExports+ _ -> GenericWASM.runServer+ cpath = outpath `addExtension` "c"+ mjspath = outpath `addExtension` "mjs"+ classpath = outpath `addExtension` "class.js"++-- | A Node.js script that runs the given ES module. Unlike the module+-- itself, it works whatever module type @package.json@ specifies.+wasmLauncher :: FilePath -> T.Text+wasmLauncher mjsname =+ T.unlines+ [ "#!/usr/bin/env node",+ "import(\"./\" + encodeURIComponent(" <> jsString <> "));"+ ]+ where+ jsString = T.decodeUtf8 $ BS.toStrict $ JSON.encode mjsname+ -- | The @futhark wasm@ action. compileCtoWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action SeqMem compileCtoWASMAction fcfg mode outpath = Action- { actionName = "Compile to sequential C",- actionDescription = "Compile to sequential C",- actionProcedure = helper+ { actionName = "Compile to WebAssembly",+ actionDescription = "Compile to sequential WebAssembly",+ actionProcedure =+ compileWASM mode outpath ["-lm"]+ <=< handleWarnings fcfg . SequentialWASM.compileProg versionString }- where- helper prog = do- (cprog, jsprog, exps) <-- handleWarnings fcfg $ SequentialWASM.compileProg versionString prog- case mode of- ToLibrary -> do- writeLibs cprog jsprog- liftIO $ T.appendFile classpath SequentialWASM.libraryExports- runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm"] exps True- _ -> do- -- Non-server executables are not supported.- writeLibs cprog jsprog- liftIO $ T.appendFile classpath $ SequentialWASM.runServer $ SequentialWASM.cManifest cprog- runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm"] exps False- writeLibs cprog jsprog = do- let (h, imp, _) = SequentialC.asLibrary cprog- liftIO $ T.writeFile hpath h- liftIO $ T.writeFile cpath imp- liftIO $ T.writeFile classpath jsprog - cpath = outpath `addExtension` "c"- hpath = outpath `addExtension` "h"- mjspath = outpath `addExtension` "mjs"- classpath = outpath `addExtension` ".class.js"- -- | The @futhark wasm-multicore@ action. compileMulticoreToWASMAction :: FutharkConfig -> CompilerMode -> FilePath -> Action MCMem compileMulticoreToWASMAction fcfg mode outpath = Action- { actionName = "Compile to sequential C",- actionDescription = "Compile to sequential C",- actionProcedure = helper+ { actionName = "Compile to WebAssembly",+ actionDescription = "Compile to multicore WebAssembly",+ actionProcedure =+ compileWASM mode outpath ["-lm", "-pthread"]+ <=< handleWarnings fcfg . MulticoreWASM.compileProg versionString }- where- helper prog = do- (cprog, jsprog, exps) <-- handleWarnings fcfg $ MulticoreWASM.compileProg versionString prog-- case mode of- ToLibrary -> do- writeLibs cprog jsprog- liftIO $ T.appendFile classpath MulticoreWASM.libraryExports- runEMCC cpath mjspath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps True- _ -> do- -- Non-server executables are not supported.- writeLibs cprog jsprog- liftIO $ T.appendFile classpath $ MulticoreWASM.runServer $ MulticoreWASM.cManifest cprog- runEMCC cpath outpath classpath ["-O3", "-msimd128"] ["-lm", "-pthread"] exps False-- writeLibs cprog jsprog = do- let (h, imp, _) = MulticoreC.asLibrary cprog- liftIO $ T.writeFile hpath h- liftIO $ T.writeFile cpath imp- liftIO $ T.writeFile classpath jsprog-- cpath = outpath `addExtension` "c"- hpath = outpath `addExtension` "h"- mjspath = outpath `addExtension` "mjs"- classpath = outpath `addExtension` ".class.js" runWith :: Name ->
src/Futhark/Analysis/Interference.hs view
@@ -117,6 +117,10 @@ defbody : map caseBody cases Loop merge _ body -> analyseLoopParams merge <$> analyseBody lumap inuse_outside body+ -- The memory of the accumulated arrays is that of the pattern, and is thus+ -- in use throughout the lambda.+ WithAcc _ lam ->+ analyseLambda lumap inuse_outside lam Op (Inner (SegOp segop)) -> do analyseSegOp lumap inuse_outside segop _ ->
src/Futhark/CodeGen/Backends/GPU.hs view
@@ -159,6 +159,16 @@ GC.stm [C.cstm|$id:v = $exp:e;|] callKernel (LaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size) = genLaunchKernel safety kernel_name shared_memory args num_tblocks tblock_size+callKernel (GetBookkeeping v b) =+ GC.stm+ [C.cstm|$id:v = (struct memblock_device) {.references = NULL,+ .mem = $exp:buf,+ .size = $int:(4 * bookkeepingSize b),+ .desc = $string:desc};|]+ where+ (buf, desc) = case b of+ BookkeepingCounters -> ([C.cexp|ctx->bookkeeping_counters|], "bookkeeping_counters")+ BookkeepingLocks -> ([C.cexp|ctx->bookkeeping_locks|], "bookkeeping_locks") copygpu2gpu :: GC.DoCopy op s copygpu2gpu _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do@@ -472,6 +482,8 @@ mapM_ GC.earlyDecl [C.cunit|static const int max_failure_args = $int:max_failure_args;+ static const typename int64_t bookkeeping_counters_size = $int:(bookkeepingSize BookkeepingCounters);+ static const typename int64_t bookkeeping_locks_size = $int:(bookkeepingSize BookkeepingLocks); static const int f64_required = $exp:f64_required; static const char *gpu_program[] = {$inits:program_fragments}; $esc:(T.unpack gpuPrototypesH)
src/Futhark/CodeGen/Backends/GenericC.hs view
@@ -50,16 +50,18 @@ defCall dests fname args = do let out_args = [[C.cexp|&$id:d|] | d <- dests] args' = [C.cexp|ctx|] : out_args ++ args- item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; goto cleanup; }|]+ failure <- failureJump+ item [C.citem|if ($id:(funName fname)($args:args') != 0) { err = 1; $stm:failure }|] defError :: ErrorCompiler op s defError msg stacktrace = do (formatstr, formatargs) <- errorMsgString msg let formatstr' = "Error: " <> formatstr <> "\n\nBacktrace:\n%s"+ failure <- failureJump items [C.citems|set_error(ctx, msgprintf($string:formatstr', $args:formatargs, $string:stacktrace)); err = FUTHARK_PROGRAM_ERROR;- goto cleanup;|]+ $stm:failure|] lmadcopyCPU :: DoCopy op s lmadcopyCPU _ t shape dst (dstoffset, dststride) src (srcoffset, srcstride) = do@@ -657,6 +659,7 @@ compileCode init_consts decl_mem <- declAllocatedMem free_mem <- freeAllocatedMem+ error_label <- errorLabel libDecl [C.cedecl|static int init_constants($ty:ctx_ty *ctx) { (void)ctx;@@ -666,6 +669,7 @@ $items:init_consts' $items:free_mem $items:undefs+ $items:error_label cleanup: return err; }|]@@ -700,7 +704,7 @@ freeConst ScalarParam {} = pure () freeConst (MemParam name space) =- unRefMem [C.cexp|ctx->constants->$id:name|] space [C.cstm|err = 1;|]+ unRefMem [C.cexp|ctx->constants->$id:name|] space $ pure [C.cstm|err = 1;|] getConst (ScalarParam name bt) = do let ctp = primTypeToCType bt
src/Futhark/CodeGen/Backends/GenericC/Code.hs view
@@ -286,6 +286,25 @@ stm [C.cstm|if (!$exp:e') { $items:err }|] where stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs+compileCode (OnFailure body handler) = do+ -- Failures in the body jump to its own error label, after which the handler+ -- runs and the failure continues to the enclosing error label.+ (body', error_label) <- withoutPendingFailures $ do+ body' <- collect $ compileCode body+ (body',) <$> errorLabel+ if null error_label+ then items [C.citems|{ $items:body' }|]+ else do+ handler' <- collect $ compileCode handler+ failure <- failureJump+ no_failure <- newVName "no_failure"+ items+ [C.citems|{ $items:body' }+ goto $id:no_failure;+ $items:error_label+ { $items:handler' }+ $stm:failure+ $id:no_failure: ;|] compileCode (Allocate _ _ ScalarSpace {}) = -- Handled by the declaration of the memory block, which is -- translated to an actual array.@@ -294,19 +313,20 @@ size <- compileExp e cached <- cacheMem name case cached of- Just cur_size ->+ Just cur_size -> do+ failure <- failureJump stm [C.cstm|if ($exp:cur_size < $exp:size) { err = lexical_realloc(ctx, &$exp:name, &$exp:cur_size, $exp:size); if (err != FUTHARK_SUCCESS) {- goto cleanup;+ $stm:failure } }|] _ ->- allocMem name size space [C.cstm|{err = 1; goto cleanup;}|]+ allocMem name size space memoryFailure compileCode (Free name space) = do cached <- isJust <$> cacheMem name- unless cached $ unRefMem name space [C.cstm|{err = 1; goto cleanup;}|]+ unless cached $ unRefMem name space memoryFailure compileCode (For i bound body) = do let i' = C.toIdent i t = primTypeToCType $ primExpType bound@@ -403,7 +423,7 @@ src' <- compileExp src stm [C.cstm|$id:dest = $exp:src';|] compileCode (SetMem dest src space) =- setMem dest src space [C.cstm|{err = 1; goto cleanup;}|]+ setMem dest src space compileCode (Call [dest] fname args) | isBuiltInFunction fname = do args' <- mapM compileArg args
src/Futhark/CodeGen/Backends/GenericC/Fun.hs view
@@ -32,7 +32,7 @@ setRetVal' p (MemParam name space) = -- It is required that the memory block is already initialised -- (although it may be NULL).- setMem [C.cexp|*$exp:p|] name space [C.cstm|{err = 1; goto cleanup;}|]+ setMem [C.cexp|*$exp:p|] name space setRetVal' p (ScalarParam name _) = stm [C.cstm|*$exp:p = $id:name;|] @@ -64,6 +64,7 @@ body' <- collect $ compileFunBody out_ptrs outputs body decl_mem <- declAllocatedMem free_mem <- freeAllocatedMem+ error_label <- errorLabel let futhark_function = C.DeclSpec [] [C.EscTypeQual "FUTHARK_FUN_ATTR" mempty] (C.Tint Nothing mempty) mempty @@ -76,6 +77,7 @@ $items:decl_mem $items:get_constants $items:body'+ $items:error_label cleanup: { $stms:free_cached
src/Futhark/CodeGen/Backends/GenericC/Monad.hs view
@@ -29,6 +29,10 @@ generateProgramStruct, runCompilerM, inNewFunction,+ failureJump,+ memoryFailure,+ errorLabel,+ withoutPendingFailures, cachingMemory, volQuals, rawMem,@@ -116,7 +120,10 @@ compCtxFields :: DL.DList (C.Id, C.Type, Maybe C.Exp, Maybe (C.Stm, C.Stm)), compClearItems :: DL.DList C.BlockItem, compDeclaredMem :: [(VName, Space)],- compItems :: DL.DList C.BlockItem+ compItems :: DL.DList C.BlockItem,+ -- | The label that the failures since the previous 'errorLabel' jump to, if+ -- there have been any.+ compErrorLabel :: Maybe VName } newCompilerState :: VNameSource -> s -> CompilerState s@@ -131,7 +138,8 @@ compCtxFields = mempty, compClearItems = mempty, compDeclaredMem = mempty,- compItems = mempty+ compItems = mempty,+ compErrorLabel = Nothing } -- | In which part of the header file we put the declaration. This is@@ -252,7 +260,7 @@ freeAllocatedMem :: CompilerM op s [C.BlockItem] freeAllocatedMem = collect $ mapM_ free =<< gets compDeclaredMem where- free (name, space) = unRefMem name space [C.cstm|err = 1;|]+ free (name, space) = unRefMem name space $ pure [C.cstm|err = 1;|] declAllocatedMem :: CompilerM op s [C.BlockItem] declAllocatedMem = collect $ mapM_ f =<< gets compDeclaredMem@@ -383,12 +391,50 @@ inNewFunction m = do old_mem <- gets compDeclaredMem modify $ \s -> s {compDeclaredMem = mempty}- x <- local noCached m+ x <- local noCached $ withoutPendingFailures m modify $ \s -> s {compDeclaredMem = old_mem} pure x where noCached env = env {envCachedMem = mempty} +-- | A statement that jumps to the next error label; see 'errorLabel'. Don't do+-- this without setting @err@ or similar first.+failureJump :: CompilerM op s C.Stm+failureJump = do+ label <- maybe (newVName "failure") pure =<< gets compErrorLabel+ modify $ \s -> s {compErrorLabel = Just label}+ pure [C.cstm|goto $id:label;|]++-- | What do do after failing a memory operation: set @err@ and jump as by+-- 'failureJump'.+memoryFailure :: CompilerM op s C.Stm+memoryFailure = do+ failure <- failureJump+ pure [C.cstm|{err = 1; $stm:failure}|]++-- | The label that the failures since the previous error label jump to, if+-- there have been any. A failure thus jumps to the first error label that+-- follows it in the program text. Every function that may fail must end with an+-- error label just before its @cleanup@ label.+errorLabel :: CompilerM op s [C.BlockItem]+errorLabel = do+ label <- gets compErrorLabel+ modify $ \s -> s {compErrorLabel = Nothing}+ pure $ case label of+ Nothing -> []+ Just label' -> [C.citems|$id:label': ;|]++-- | Run an action as if no failures had happened before it, and afterwards+-- restore the failures that had. The action must place the error labels for its+-- own failures.+withoutPendingFailures :: CompilerM op s a -> CompilerM op s a+withoutPendingFailures m = do+ old_label <- gets compErrorLabel+ modify $ \s -> s {compErrorLabel = Nothing}+ x <- m+ modify $ \s -> s {compErrorLabel = old_label}+ pure x+ -- | Insert a block item in the generated code at this point. item :: C.BlockItem -> CompilerM op s () item x = modify $ \s -> s {compItems = DL.snoc (compItems s) x}@@ -589,12 +635,14 @@ when refcount $ stm [C.cstm|$exp:mem.references = NULL;|] -setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> C.Stm -> CompilerM op s ()-setMem dest src space on_failure = do+-- | Set @dest@ to @src@. A failure is handled by 'memoryFailure'.+setMem :: (C.ToExp a, C.ToExp b) => a -> b -> Space -> CompilerM op s ()+setMem dest src space = do refcount <- fatMemory space let src_s = T.unpack $ expText $ C.toExp src noLoc if refcount- then+ then do+ on_failure <- memoryFailure stm [C.cstm|if ($id:(fatMemSet space)(ctx, &$exp:dest, &$exp:src, $string:src_s) != 0) {@@ -613,15 +661,17 @@ }|] _ -> stm [C.cstm|$exp:dest = $exp:src;|] -unRefMem :: (C.ToExp a) => a -> Space -> C.Stm -> CompilerM op s ()+-- | The failure code is only generated if it is needed.+unRefMem :: (C.ToExp a) => a -> Space -> CompilerM op s C.Stm -> CompilerM op s () unRefMem mem space on_failure = do refcount <- fatMemory space cached <- isJust <$> cacheMem mem let mem_s = T.unpack $ expText $ C.toExp mem noLoc- when (refcount && not cached) $+ when (refcount && not cached) $ do+ on_failure' <- on_failure stm [C.cstm|if ($id:(fatMemUnRef space)(ctx, &$exp:mem, $string:mem_s) != 0) {- $stm:on_failure+ $stm:on_failure' }|] allocMem ::@@ -629,22 +679,23 @@ a -> b -> Space ->- C.Stm ->+ CompilerM op s C.Stm -> CompilerM op s () allocMem mem size space on_failure = do refcount <- fatMemory space let mem_s = T.unpack $ expText $ C.toExp mem noLoc+ on_failure' <- on_failure if refcount then stm [C.cstm|if ($id:(fatMemAlloc space)(ctx, &$exp:mem, $exp:size, $string:mem_s)) {- $stm:on_failure+ $stm:on_failure' }|] else do freeRawMem mem size space mem_s alloc <- allocRawMem mem size space [C.cexp|desc|]- stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure }|]+ stm [C.cstm|if ($exp:alloc != FUTHARK_SUCCESS) { $stm:on_failure' }|] copyMemoryDefaultSpace :: C.Exp ->
src/Futhark/CodeGen/Backends/GenericC/Types.hs view
@@ -56,7 +56,7 @@ [C.cexp|$exp:arr->mem|] [C.cexp|$exp:arr_size * $int:(primByteSize pt::Int)|] space- [C.cstm|err = 1;|]+ (pure [C.cstm|err = 1;|]) arrayLibraryFunctions :: Publicness ->@@ -114,7 +114,7 @@ let dim_s = "dim" ++ show i in stm [C.cstm|arr->shape[$int:i] = $id:dim_s;|] - free_body <- collect $ unRefMem [C.cexp|arr->mem|] space [C.cstm|ret = 1;|]+ free_body <- collect $ unRefMem [C.cexp|arr->mem|] space $ pure [C.cstm|ret = 1;|] values_body <- collect $
src/Futhark/CodeGen/Backends/GenericPython.hs view
@@ -1355,6 +1355,10 @@ cond' <- compileExp $ Imp.untyped cond body' <- collect $ compileCode body stm $ While cond' body'+compileCode (Imp.OnFailure body handler) = do+ body' <- collect $ compileCode body+ handler' <- collect $ compileCode handler+ stm $ Try body' [Catch (Var "Exception") $ handler' ++ [Raise $ Var "e"]] compileCode (Imp.For i bound body) = do bound' <- compileExp bound let i' = compileName i
src/Futhark/CodeGen/Backends/GenericWASM.hs view
@@ -1,126 +1,22 @@ {-# LANGUAGE QuasiQuotes #-} +-- | Code generation shared by the WASM backends. The JavaScript API is+-- in @rts/javascript/wrappers.js@, driven by the manifest. module Futhark.CodeGen.Backends.GenericWASM- ( GC.CParts (..),- GC.asLibrary,- GC.asExecutable,- GC.asServer,- EntryPointType,- JSEntryPoint (..),- JSRecordField (..),- JSOpaqueType (..),- emccExportNames,- manifestFunctions,+ ( emccExportNames, javascriptWrapper,- extToString,- opaqueToJS, runServer, libraryExports, ) where -import Data.List (intercalate) import Data.Map qualified as M import Data.Text qualified as T-import Futhark.CodeGen.Backends.GenericC qualified as GC-import Futhark.CodeGen.Backends.SimpleRep (isValidCName, opaqueName)-import Futhark.CodeGen.ImpCode.Sequential qualified as Imp import Futhark.CodeGen.RTS.JavaScript import Futhark.Manifest-import Futhark.Util (nubOrd, showText, zEncodeText)-import Language.Futhark.Core (nameToText)-import Language.Futhark.Primitive+import Futhark.Util (nubOrd) import NeatInterpolation (text) -extToString :: Imp.ExternalValue -> String-extToString (Imp.TransparentValue (Imp.ArrayValue vn _ pt s dimSize)) =- concat (replicate (length dimSize) "[]") ++ extToString (Imp.TransparentValue (Imp.ScalarValue pt s vn))-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float16) _ _)) = "f16"-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float32) _ _)) = "f32"-extToString (Imp.TransparentValue (Imp.ScalarValue (FloatType Float64) _ _)) = "f64"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Signed _)) = "i8"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Signed _)) = "i16"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Signed _)) = "i32"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Signed _)) = "i64"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int8) Imp.Unsigned _)) = "u8"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int16) Imp.Unsigned _)) = "u16"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int32) Imp.Unsigned _)) = "u32"-extToString (Imp.TransparentValue (Imp.ScalarValue (IntType Int64) Imp.Unsigned _)) = "u64"-extToString (Imp.TransparentValue (Imp.ScalarValue Bool _ _)) = "bool"-extToString (Imp.TransparentValue (Imp.ScalarValue Unit _ _)) = error "extToString: Unit"-extToString (Imp.OpaqueValue oname _) = T.unpack $ opaqueName oname--type EntryPointType = String--data JSEntryPoint = JSEntryPoint- { name :: String,- parameters :: [EntryPointType],- ret :: EntryPointType- }---- | A field in a JavaScript record opaque type.-data JSRecordField = JSRecordField- { -- | Original field name (used in the server protocol).- jsrfName :: String,- -- | JavaScript type string (e.g. @"i32"@, @"[]f64"@, @"opaque_foo"@).- jsrfType :: EntryPointType,- -- | JavaScript method name on @FutharkContext@ (e.g. @"project_opaque_foo_x"@).- jsrfProjectFn :: String- }---- | How an opaque type is represented for the JavaScript server.-data JSOpaqueType- = -- | A record type with named, projectable fields.- JSOpaqueRecord [JSRecordField]- | -- | Any other opaque type (no project support).- JSOpaqueOther---- | Convert the opaque-type table from an ImpCode program into the--- JavaScript-oriented representation used by the code generator.-opaqueToJS :: Imp.OpaqueTypes -> [(String, JSOpaqueType)]-opaqueToJS (Imp.OpaqueTypes types) = map convertOne types- where- convertOne (desc, (Imp.OpaqueRecord fields, _)) =- (T.unpack (opaqueName desc), JSOpaqueRecord (map (convertField desc) fields))- convertOne (desc, _) =- (T.unpack (opaqueName desc), JSOpaqueOther)-- convertField desc (fname, etype) =- JSRecordField- { jsrfName = T.unpack (nameToText fname),- jsrfType = entryTypeToJSString etype,- jsrfProjectFn =- "project_"- ++ T.unpack (opaqueName desc)- ++ "_"- ++ T.unpack f'- }- where- f'- | isValidCName (opaqueName desc <> "_" <> nameToText fname) =- nameToText fname- | otherwise = zEncodeText (nameToText fname)---- | Convert an 'Imp.EntryPointType' to its JavaScript string representation.-entryTypeToJSString :: Imp.EntryPointType -> String-entryTypeToJSString (Imp.TypeOpaque desc) = T.unpack $ opaqueName desc-entryTypeToJSString (Imp.TypeTransparent (Imp.ValueType sign (Imp.Rank rank) pt)) =- concat (replicate rank "[]") ++ primToJSString sign pt- where- primToJSString _ (FloatType Float16) = "f16"- primToJSString _ (FloatType Float32) = "f32"- primToJSString _ (FloatType Float64) = "f64"- primToJSString Imp.Signed (IntType Int8) = "i8"- primToJSString Imp.Signed (IntType Int16) = "i16"- primToJSString Imp.Signed (IntType Int32) = "i32"- primToJSString Imp.Signed (IntType Int64) = "i64"- primToJSString Imp.Unsigned (IntType Int8) = "u8"- primToJSString Imp.Unsigned (IntType Int16) = "u16"- primToJSString Imp.Unsigned (IntType Int32) = "u32"- primToJSString Imp.Unsigned (IntType Int64) = "u64"- primToJSString _ Bool = "bool"- primToJSString _ Unit = error "entryTypeToJSString: Unit"- -- | The names of the C functions that must be exported from the WebAssembly -- module. This is every function named in the manifest, as the server calls -- these directly, as well as the context management functions.@@ -129,6 +25,13 @@ map (\f -> "'_" ++ T.unpack f ++ "'") (nubOrd $ manifestFunctions manifest) ++ [ "_futhark_context_config_new", "_futhark_context_config_free",+ "_futhark_context_config_set_debugging",+ "_futhark_context_config_set_logging",+ "_futhark_context_config_set_profiling",+ "_futhark_context_config_set_tuning_param",+ "_futhark_get_tuning_param_count",+ "_futhark_get_tuning_param_name",+ "_futhark_get_tuning_param_class", "_futhark_context_new", "_futhark_context_free", "_futhark_context_get_error",@@ -180,460 +83,29 @@ variantFunctions v = [sumVariantConstruct v, sumVariantDestruct v] -javascriptWrapper :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-javascriptWrapper entryPoints opaqueTypes =+-- | The JavaScript code passed to @emcc@ with @--extern-post-js@.+javascriptWrapper :: Manifest -> T.Text+javascriptWrapper manifest = T.unlines [ serverJs, valuesJs,- wrapperclassesJs,- classFutharkContext entryPoints opaqueTypes+ wrappersJs,+ "const futhark_manifest = " <> manifestToJSON manifest <> ";" ] --- Make FutharkModule the generated primary class, but keeps FutharkContext around for backwards compatibility.-classFutharkContext :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-classFutharkContext entryPoints opaqueTypes =- T.unlines- [ "class FutharkModule {",- moduleConstructor,- moduleInitFromWasm entryPoints opaqueTypes,- moduleInit,- getFreeFun,- getEntryPointsFun,- getTypesFun,- getErrorFun,- getUtilityFuns,- T.unlines $ map toFutharkArray arrays,- T.unlines $ concatMap (generateProjectMethods . snd) opaqueTypes,- T.unlines $ map jsWrapEntryPoint entryPoints,- "}",- classFutharkContextCompat,- [text|- async function newFutharkContext(num_threads) {- var wasm = await loadWASM();- return new FutharkContext(wasm, num_threads);- }- |]- ]- where- -- Collect array types from entry points AND from record fields so the- -- array constructors are always available when returning projected values.- arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)- entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) entryPoints- recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]--moduleConstructor :: T.Text-moduleConstructor =- [text|- constructor() {- this.wasm = undefined;- this.cfg = undefined;- this.ctx = undefined;- this.entry_points = {};- this.types = {};- this.entry = {};- }- |]--moduleInitFromWasm :: [JSEntryPoint] -> [(String, JSOpaqueType)] -> T.Text-moduleInitFromWasm jses opaqueTypes =- [text|- _init_from_wasm(wasm, num_threads) {- this.wasm = wasm;- this.cfg = this.wasm._futhark_context_config_new();- if (num_threads) this.wasm._futhark_context_config_set_num_threads(this.cfg, num_threads);- this.ctx = this.wasm._futhark_context_new(this.cfg);-- this.entry_points = {- ${entries}- };- this.types = {- ${type_entries}- };- this.entry = {};- ${entry_aliases}- ${array_aliases}- ${array_type_aliases}- }- |]- where- entries = T.intercalate "," $ map dicEntry jses- type_entries = T.intercalate "," $ map dicTypeEntry opaqueTypes- entry_aliases = T.unlines $ map entryAlias jses- array_aliases = T.unlines $ map arrayAlias arrays- array_type_aliases = T.unlines $ map arrayTypeAlias arrays-- arrays = nubOrd $ filter isArray (entryPointTypes ++ recordFieldTypes)- entryPointTypes = concatMap (\jse -> parameters jse ++ [ret jse]) jses- recordFieldTypes = [jsrfType rf | (_, JSOpaqueRecord fields) <- opaqueTypes, rf <- fields]--moduleInit :: T.Text-moduleInit =- [text|- async init(wasm, num_threads) {- if (wasm === undefined) {- throw new Error("FutharkModule.init() requires the generated backend runtime module");- }-- this._init_from_wasm(wasm, num_threads);- }- |]--classFutharkContextCompat :: T.Text-classFutharkContextCompat =- [text|- class FutharkContext extends FutharkModule {- constructor(wasm, num_threads) {- super();-- if (wasm !== undefined) {- this._init_from_wasm(wasm, num_threads);- }- }- }- |]--getFreeFun :: T.Text-getFreeFun =- [text|- free() {- if (this.ctx !== undefined) {- this.wasm._futhark_context_free(this.ctx);- this.ctx = undefined;- }-- if (this.cfg !== undefined) {- this.wasm._futhark_context_config_free(this.cfg);- this.cfg = undefined;- }- }- |]--getEntryPointsFun :: T.Text-getEntryPointsFun =- [text|- get_entry_points() {- return this.entry_points;- }- |]--getTypesFun :: T.Text-getTypesFun =- [text|- get_types() {- return this.types;- }- |]--getErrorFun :: T.Text-getErrorFun =- [text|- get_error() {- var ptr = this.wasm._futhark_context_get_error(this.ctx);- var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);- var str = String.fromCharCode(...this.wasm.HEAP8.subarray(ptr, ptr + len));- this.wasm._free(ptr);- return str;- }- |]--getUtilityFuns :: T.Text-getUtilityFuns =- [text|- async context_sync() {- return this.wasm._futhark_context_sync(this.ctx);- }-- async clear_caches() {- return this.wasm._futhark_context_clear_caches(this.ctx);- }-- async report() {- var ptr = this.wasm._futhark_context_report(this.ctx);- var len = this.wasm.HEAP8.subarray(ptr).indexOf(0);- var bytes = this.wasm.HEAPU8.subarray(ptr, ptr + len);- var str = new TextDecoder().decode(bytes);- this.wasm._free(ptr);- return str;- }-- async pause_profiling() {- return this.wasm._futhark_context_pause_profiling(this.ctx);- }-- async unpause_profiling() {- return this.wasm._futhark_context_unpause_profiling(this.ctx);- }- |]--entryAlias :: JSEntryPoint -> T.Text-entryAlias jse =- [text|this.entry["${ename}"] = this.${fname}.bind(this);|]- where- fname = GC.escapeName $ T.pack $ name jse- ename = T.pack $ name jse--arrayAlias :: String -> T.Text-arrayAlias typ =- [text|- this.${signature} = {- from_data: (data, ${dims}) => this.new_${signature}(data, ${dims}),- from_jsarray: (data) => this.new_${signature}_from_jsarray(data)- };- |]- where- d = dim typ- ftype = baseType typ- signature = T.pack $ ftype ++ "_" ++ show d ++ "d"- dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]--arrayTypeAlias :: String -> T.Text-arrayTypeAlias typ =- [text|this.types["${typ_text}"] = this.${signature};|]- where- d = dim typ- ftype = baseType typ- signature = T.pack $ ftype ++ "_" ++ show d ++ "d"- typ_text = T.pack typ--dicEntry :: JSEntryPoint -> T.Text-dicEntry jse =- [text|- "${ename}" : ["${fname}", ${params}, ${rets}]- |]- where- fname = GC.escapeName $ T.pack $ name jse- ename = T.pack $ name jse- params = showText $ parameters jse- rets = showText $ ret jse---- | Generate the @types@ dictionary entry for a single opaque type.-dicTypeEntry :: (String, JSOpaqueType) -> T.Text-dicTypeEntry (tname, JSOpaqueRecord fields) =- T.pack $- show tname- ++ ": [\"record\", ["- ++ intercalate ", " (map fieldEntry fields)- ++ "]]"- where- fieldEntry rf =- "["- ++ show (jsrfName rf)- ++ ", "- ++ show (jsrfType rf)- ++ ", \""- ++ jsrfProjectFn rf- ++ "\"]"-dicTypeEntry (tname, JSOpaqueOther) =- T.pack $ show tname ++ ": [\"opaque\"]"---- | Generate @project_*@ wrapper methods for a record opaque type.-generateProjectMethods :: JSOpaqueType -> [T.Text]-generateProjectMethods (JSOpaqueRecord fields) = map generateProjectMethod fields-generateProjectMethods JSOpaqueOther = []--generateProjectMethod :: JSRecordField -> T.Text-generateProjectMethod rf =- T.pack $- unlines- [ " " ++ jsrfProjectFn rf ++ "(obj) {",- " var out = this.wasm._malloc(" ++ show (typeSize ftype) ++ ");",- " this.wasm._futhark_" ++ jsrfProjectFn rf ++ "(this.ctx, out, obj.ptr);",- " var result = " ++ readResult ++ ";",- " this.wasm._free(out);",- " return result;",- " }"- ]- where- ftype = jsrfType rf- readout = typeHeap ftype ++ "[out >> " ++ show (typeShift ftype) ++ "]"- readResult- | isArray ftype =- "this.new_" ++ baseType ftype ++ "_" ++ show (dim ftype) ++ "d_from_ptr(" ++ readout ++ ")"- | isOpaque ftype =- "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ ftype ++ ")"- | ftype == "bool" = readout ++ "!==0"- | otherwise = readout--jsWrapEntryPoint :: JSEntryPoint -> T.Text-jsWrapEntryPoint jse =- [text|- ${func_name}(${inparams}) {- var out = [${outparams}].map(n => this.wasm._malloc(n));- var to_free = [];- var do_free = () => { out.forEach(this.wasm._free); to_free.forEach(f => f.free()); };- ${paramsToPtr}- if (this.wasm._futhark_entry_${func_name}(this.ctx, ...out, ${ins}) > 0) {- do_free();- throw this.get_error();- }- ${results}- do_free();- return ${res};- }- |]- where- func_name = GC.escapeName $ T.pack $ name jse-- alp = [0 .. length (parameters jse) - 1]- inparams = T.pack $ intercalate ", " ["in" ++ show i | i <- alp]- ins = T.pack $ intercalate ", " [maybeDerefence ("in" ++ show i) $ parameters jse !! i | i <- alp]- paramsToPtr = T.pack $ unlines $ filter ("" /=) [arrayPointer ("in" ++ show i) $ parameters jse !! i | i <- alp]-- outparams = showText $ typeSize $ ret jse- results = T.pack $ makeResult 0 $ ret jse- res = "result0"--maybeDerefence :: String -> String -> String-maybeDerefence arg typ =- if isScalar typ then arg else arg ++ ".ptr"--arrayPointer :: String -> String -> String-arrayPointer arg typ =- if isArray typ- then " if (" ++ arg ++ " instanceof Array) { " ++ reassign ++ "; to_free.push(" ++ arg ++ "); }"- else ""- where- reassign = arg ++ " = this.new_" ++ signature ++ "_from_jsarray(" ++ arg ++ ")"- signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"--makeResult :: Int -> String -> String-makeResult i typ =- " var result"- ++ show i- ++ " = "- ++ if isArray typ- then "this.new_" ++ signature ++ "_from_ptr(" ++ readout ++ ");"- else- if isOpaque typ- then "new FutharkOpaque(this, " ++ readout ++ ", this.wasm._futhark_free_" ++ typ ++ ");"- else readout ++ if typ == "bool" then "!==0;" else ";"- where- res = "out[" ++ show i ++ "]"- readout = typeHeap typ ++ "[" ++ res ++ " >> " ++ show (typeShift typ) ++ "]"- signature = baseType typ ++ "_" ++ show (dim typ) ++ "d"--baseType :: String -> String-baseType ('[' : ']' : end) = baseType end-baseType typ = typ--dim :: String -> Int-dim ('[' : ']' : end) = dim end + 1-dim _ = 0--isArray :: String -> Bool-isArray typ = take 2 typ == "[]"--isOpaque :: String -> Bool-isOpaque typ = take 6 typ == "opaque"--isScalar :: String -> Bool-isScalar typ = not (isArray typ || isOpaque typ)--typeSize :: String -> Integer-typeSize typ =- case typ of- "i8" -> 1- "i16" -> 2- "i32" -> 4- "i64" -> 8- "u8" -> 1- "u16" -> 2- "u32" -> 4- "u64" -> 8- "f16" -> 2- "f32" -> 4- "f64" -> 8- "bool" -> 1- _ -> 4--typeShift :: String -> Integer-typeShift typ =- case typ of- "i8" -> 0- "i16" -> 1- "i32" -> 2- "i64" -> 3- "u8" -> 0- "u16" -> 1- "u32" -> 2- "u64" -> 3- "f16" -> 1- "f32" -> 2- "f64" -> 3- "bool" -> 0- _ -> 2--typeHeap :: String -> String-typeHeap typ =- case typ of- "i8" -> "this.wasm.HEAP8"- "i16" -> "this.wasm.HEAP16"- "i32" -> "this.wasm.HEAP32"- "i64" -> "this.wasm.HEAP64"- "u8" -> "this.wasm.HEAPU8"- "u16" -> "this.wasm.HEAPU16"- "u32" -> "this.wasm.HEAPU32"- "u64" -> "(new BigUint64Array(this.wasm.HEAP64.buffer))"- "f16" -> "this.wasm.HEAPU16"- "f32" -> "this.wasm.HEAPF32"- "f64" -> "this.wasm.HEAPF64"- "bool" -> "this.wasm.HEAP8"- _ -> "this.wasm.HEAP32"--toFutharkArray :: String -> T.Text-toFutharkArray typ =- [text|- ${new}_from_jsarray(${arraynd_p}) {- return this.${new}(${arraynd_flat_p}, ${arraynd_dims_p});- }- ${new}(array, ${dims}) {- console.assert(array.length === ${dims_multiplied}, 'len=%s,dims=%s', array.length, [${dims}].toString());- var copy = this.wasm._malloc(array.length << ${shift});- ${heapType}.set(array, copy >> ${shift});- var ptr = ${fnew}(this.ctx, copy, ${bigint_dims});- this.wasm._free(copy);- return this.${new}_from_ptr(ptr);- }-- ${new}_from_ptr(ptr) {- return new FutharkArray(this, ptr, ${args});- }- |]- where- d = dim typ- ftype = baseType typ- heap = typeHeap ftype- signature = ftype ++ "_" ++ show d ++ "d"- new = T.pack $ "new_" ++ signature- fnew = T.pack $ "this.wasm._futhark_new_" ++ signature- fshape = "this.wasm._futhark_shape_" ++ signature- fvalues = "this.wasm._futhark_values_raw_" ++ signature- ffree = "this.wasm._futhark_free_" ++ signature- arraynd = "array" ++ show d ++ "d"- shift = showText (typeShift ftype)- heapType = T.pack heap- arraynd_flat = if d > 1 then arraynd ++ ".flat()" else arraynd- arraynd_dims = intercalate ", " [arraynd ++ mult i "[0]" ++ ".length" | i <- [0 .. d - 1]]- dims = T.pack $ intercalate ", " ["d" ++ show i | i <- [0 .. d - 1]]- dims_multiplied = T.pack $ intercalate "*" ["Number(d" ++ show i ++ ")" | i <- [0 .. d - 1]]- bigint_dims = T.pack $ intercalate ", " ["BigInt(d" ++ show i ++ ")" | i <- [0 .. d - 1]]- mult i s = concat $ replicate i s- (arraynd_p, arraynd_flat_p, arraynd_dims_p) = (T.pack arraynd, T.pack arraynd_flat, T.pack arraynd_dims)- args = T.pack $ intercalate ", " ["'" ++ ftype ++ "'", show d, heap, fshape, fvalues, ffree]- -- | Javascript code that can be appended to the generated module to--- run a Futhark server instance on startup.-runServer :: Manifest -> T.Text-runServer manifest =+-- run a Futhark server instance when it is loaded.+runServer :: T.Text+runServer = [text|- Module.onRuntimeInitialized = () => {- var context = new FutharkContext(Module);- var server = new Server(context, ${manifest_json});- server.run();+ // Worker threads (wasm-multicore) also load this module.+ const is_main_thread = globalThis.process?.versions?.node+ ? (await import("node:worker_threads")).isMainThread+ : globalThis.WorkerGlobalScope === undefined;+ if (is_main_thread) {+ await futhark_server_main(); }|]- where- manifest_json = manifestToJSON manifest -- | The names exported by the generated module. libraryExports :: T.Text-libraryExports = "export {newFutharkContext, FutharkContext, FutharkModule, FutharkArray, FutharkOpaque};"+libraryExports = "export {FutharkContext, FutharkArray, FutharkOpaque};"
src/Futhark/CodeGen/Backends/MulticoreC.hs view
@@ -110,8 +110,8 @@ compileRetvalStructFields :: [VName] -> [(C.Type, C.Type, ValueType)] -> [C.FieldGroup] compileRetvalStructFields = zipWith field where- field name (ty, _, Prim _) =- [C.csdecl|$ty:ty *$id:(closureRetvalStructField name);|]+ field name (_, storage_ty, Prim _) =+ [C.csdecl|$ty:storage_ty *$id:(closureRetvalStructField name);|] field name (ty, _, ScalarMem) = [C.csdecl|$ty:ty $id:(closureRetvalStructField name);|] field name (_, _, _) =@@ -134,32 +134,55 @@ field name (_, _, ScalarMem) = [C.cstm|$id:struct.$id:(closureFreeStructField name)=$id:name;|] +-- | Point the retval fields of a task struct at the variables that will receive+-- the results, and return the statements that must be executed after the task+-- has run. A scalar retval field points to the storage representation of its+-- type (see 'compileRetvalStructFields'). When that differs from the+-- representation of the variable itself, the field instead points to a+-- temporary in storage representation, which is converted back after the task+-- has run. This matters for ISPC, where the variable lives in ISPC code but is+-- written by C code, and the two may not agree on the representation of+-- 'Float16'. compileSetRetvalStructValues :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] ->- [C.Stm]-compileSetRetvalStructValues struct vnames we = concat $ zipWith field vnames we+ GC.CompilerM op s [C.Stm]+compileSetRetvalStructValues struct vnames we = concat <$> zipWithM field vnames we where- field name (ct, _, Prim _) =- [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ct*)&$id:name);- $escstm:("#if defined(ISPC)")- $id:struct.$id:(closureRetvalStructField name)+= programIndex;- $escstm:("#endif")|]- field name (_, _, MemBlock) =- [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]- field name (_, _, RawMem) =- [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]- field name (_, _, ScalarMem) =- [C.cstms|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+ field name (ct, storage_ty, Prim pt)+ | ct == storage_ty = do+ point name storage_ty+ pure []+ | otherwise = do+ tmp <- newVName $ baseName name <> "_storage"+ GC.decl [C.cdecl|$ty:storage_ty $id:tmp = $exp:(toStorage pt (C.toExp name noLoc));|]+ point tmp storage_ty+ pure [[C.cstm|$id:name = $exp:(fromStorage pt (C.toExp tmp noLoc));|]]+ where+ point v ty =+ GC.stms+ [C.cstms|$id:struct.$id:(closureRetvalStructField name)=(($ty:ty*)&$id:v);+ $escstm:("#if defined(ISPC)")+ $id:struct.$id:(closureRetvalStructField name)+= programIndex;+ $escstm:("#endif")|]+ field name (_, _, MemBlock) = do+ GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name.mem;|]+ pure []+ field name (_, _, RawMem) = do+ GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+ pure []+ field name (_, _, ScalarMem) = do+ GC.stm [C.cstm|$id:struct.$id:(closureRetvalStructField name)=$id:name;|]+ pure [] compileGetRetvalStructVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.InitGroup] compileGetRetvalStructVals struct = zipWith field where- field name (ty, _storage_ty, Prim _) =+ field name (ty, _storage_ty, Prim pt) = let inner = [C.cexp|*$id:struct->$id:(closureRetvalStructField name)|]- in [C.cdecl|$ty:ty $id:name = $exp:inner;|]+ in [C.cdecl|$ty:ty $id:name = $exp:(fromStorage pt inner);|] field name (ty, _, ScalarMem) = [C.cdecl|$ty:ty $id:name = $id:struct->$id:(closureRetvalStructField name);|] field name (ty, _, _) =@@ -190,8 +213,8 @@ compileWriteBackResVals :: (C.ToIdent a) => a -> [VName] -> [(C.Type, C.Type, ValueType)] -> [C.Stm] compileWriteBackResVals struct = zipWith field where- field name (_, _, Prim _) =- [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(C.toExp name noLoc);|]+ field name (_, _, Prim pt) =+ [C.cstm|*$id:struct->$id:(closureRetvalStructField name) = $exp:(toStorage pt (C.toExp name noLoc));|] field name (_, _, ScalarMem) = [C.cstm|$id:struct->$id:(closureRetvalStructField name) = $id:name;|] field name (_, _, _) =@@ -286,6 +309,7 @@ mapM_ GC.item =<< GC.declAllocatedMem mapM_ GC.item code' free_mem <- GC.freeAllocatedMem+ mapM_ GC.item =<< GC.errorLabel GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|] pure [C.cedecl|int $id:s(void *args, typename int64_t iterations, int tid, struct scheduler_info info) {@@ -307,7 +331,7 @@ [(C.Type, C.Type, ValueType)] -> [VName] -> [(C.Type, C.Type, ValueType)] ->- GC.CompilerM Multicore s Name+ GC.CompilerM Multicore s (Name, [C.Stm]) prepareTaskStruct def name free_args free_ctypes retval_args retval_ctypes = do let makeStruct s = pure@@ -321,8 +345,8 @@ GC.decl [C.cdecl|struct $id:fstruct $id:fstruct';|] GC.stm [C.cstm|$id:fstruct'.ctx = ctx;|] GC.stms [C.cstms|$stms:(compileSetStructValues fstruct' free_args free_ctypes)|]- GC.stms [C.cstms|$stms:(compileSetRetvalStructValues fstruct' retval_args retval_ctypes)|]- pure fstruct+ after <- compileSetRetvalStructValues fstruct' retval_args retval_ctypes+ pure (fstruct, after) -- Generate a segop function for top_level and potentially nested SegOp code compileOp :: GC.OpCompiler Multicore s@@ -346,7 +370,7 @@ let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] params mempty seq_code - fstruct <-+ (fstruct, read_retvals) <- prepareTaskStruct multicoreDef "task" free_args free_ctypes retval_args retval_ctypes fpar_task <- generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval@@ -375,22 +399,24 @@ Nothing -> GC.stm [C.cstm|$id:ftask_name.nested_fn=NULL;|] + failure <- GC.failureJump let ftask_err = fpar_task <> "_err" code = [C.citems|int $id:ftask_err = scheduler_prepare_task(&ctx->scheduler, &$id:ftask_name); if ($id:ftask_err != 0) { err = $id:ftask_err;- goto cleanup;+ $stm:failure }|] mapM_ GC.item code+ GC.stms read_retvals compileOp (ParLoop s' body free) = do free_ctypes <- mapM paramToCType free let free_args = map paramName free let lexical = lexicalMemoryUsageMC TraverseKernels $ Function Nothing [] free mempty body - fstruct <-+ (fstruct, _) <- prepareTaskStruct multicoreDef (s' <> "_parloop_struct") free_args free_ctypes mempty mempty ftask <- multicoreDef (s' <> "_parloop") $ \s -> do@@ -406,6 +432,7 @@ mapM_ GC.item =<< GC.declAllocatedMem free_mem <- GC.freeAllocatedMem mapM_ GC.item body'+ mapM_ GC.item =<< GC.errorLabel GC.stm [C.cstm|cleanup: {$stms:free_cached $items:free_mem}|] pure [C.cedecl|static int $id:s(void *args,@@ -432,6 +459,7 @@ let ftask_err = ftask <> "_err" ftask_total = ftask <> "_total"+ failure <- GC.failureJump code' <- benchmarkCode ftask_total@@ -439,7 +467,7 @@ &$id:ftask_name); if ($id:ftask_err != 0) { err = $id:ftask_err;- goto cleanup;+ $stm:failure }|] mapM_ GC.item code'
src/Futhark/CodeGen/Backends/MulticoreISPC.hs view
@@ -707,6 +707,9 @@ GC.stm [C.cstm|if (!$exp:e') { $items:err }|] where stacktrace = T.unpack $ prettyStacktrace 0 $ map locText $ loc : locs+compileCode OnFailure {} =+ -- ISPC does not allow the goto.+ error "compileCode: OnFailure in ISPC kernel." compileCode code = GC.compileCode code @@ -790,6 +793,18 @@ $id:name.size = 0; $id:name.references = NULL;|] +-- | Does this code contain an 'OnFailure', not counting those in the tasks of+-- nested 'SegOp's?+hasOnFailure :: MCCode -> Bool+hasOnFailure OnFailure {} = True+hasOnFailure (x :>>: y) = hasOnFailure x || hasOnFailure y+hasOnFailure (If _ x y) = hasOnFailure x || hasOnFailure y+hasOnFailure (For _ _ x) = hasOnFailure x+hasOnFailure (While _ x) = hasOnFailure x+hasOnFailure (Op (ForEach _ _ _ body)) = hasOnFailure body+hasOnFailure (Op (ForEachActive _ body)) = hasOnFailure body+hasOnFailure _ = False+ -- | Can the given code produce an error? If so, we can't use foreach -- loops, since they don't allow for early-outs in error handling. mayProduceError :: MCCode -> Bool@@ -800,6 +815,7 @@ mayProduceError (Op (ForEachActive _ body)) = mayProduceError body mayProduceError (Op (ForEach _ _ _ body)) = mayProduceError body mayProduceError (Op SegOp {}) = True+mayProduceError (OnFailure x y) = mayProduceError x || mayProduceError y mayProduceError Allocate {} = True mayProduceError Assert {} = True mayProduceError SetMem {} = True@@ -822,7 +838,7 @@ let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] params mempty seq_code - fstruct <-+ (fstruct, read_retvals) <- MC.prepareTaskStruct sharedDef "task" free_args free_ctypes retval_args retval_ctypes fpar_task <- MC.generateParLoopFn lexical (name <> "_task") seq_code fstruct free retval@@ -874,6 +890,7 @@ struct $id:fstruct $tyqual:uniform * $tyqual:uniform args, $tyqual:uniform int iterations);|] + failure <- GC.failureJump aos_name <- newVName "aos" GC.items [C.citems|@@ -892,16 +909,23 @@ $escstm:("#else") err = $id:schedn(ctx, &$id:(fstruct <> "_"), $exp:e'); if (err != 0) {- goto cleanup;+ $stm:failure } $escstm:("#endif")|]+ GC.stms read_retvals+compileOp (ISPCKernel body free)+ | -- ISPC is too weak for our failure handling machinery.+ hasOnFailure body =+ local (\env -> env {GC.envOperations = MC.operations}) $+ MC.compileOp $+ ISPCKernel body free compileOp (ISPCKernel body free) = do free_ctypes <- mapM MC.paramToCType free let free_args = map paramName free let lexical = lexicalMemoryUsageMC OpaqueKernels $ Function Nothing [] free mempty body -- Generate ISPC kernel- fstruct <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] []+ (fstruct, _) <- MC.prepareTaskStruct sharedDef "param_struct" free_args free_ctypes [] [] let fstruct' = fstruct <> "_" ispcShim <- ispcDef "loop_ispc" $ \s -> do@@ -911,7 +935,9 @@ GC.collect $ do GC.decl [C.cdecl|$tyqual:uniform struct futhark_context * $tyqual:uniform ctx = $id:fstruct'->ctx;|] GC.items =<< compileGetStructVals fstruct free_args free_ctypes- body' <- GC.collect $ compileCode body+ body' <- GC.collect $ do+ compileCode body+ GC.items =<< GC.errorLabel mapM_ GC.item decl_cached mapM_ GC.item =<< GC.declAllocatedMem @@ -962,11 +988,12 @@ }|] -- Generate C code to call into ISPC kernel+ failure <- GC.failureJump GC.items [C.citems| err = $id:ispcShim(start, end, & $id:fstruct'); if (err != 0) {- goto cleanup;+ $stm:failure }|] compileOp (ForEach i from bound body) = do from' <- compileExp from@@ -1089,18 +1116,13 @@ free = freeIn bound findDeps (While cond x) = do local (<> freeIn cond) $ findDeps x-findDeps (Op (SegOp _ free _ _ retvals _)) =- mapM_- ( \x ->- addDeps (paramName x) $- namesFromList $- map paramName free- )- retvals findDeps (Op (ForEach _ _ _ body)) = findDeps body findDeps (Op (ForEachActive _ body)) = findDeps body+findDeps (OnFailure body handler) = do+ findDeps body+ findDeps handler findDeps (SetScalar name e) = addDeps name $ freeIn e findDeps (Call tars _ args) =@@ -1130,15 +1152,20 @@ deps' = M.map grow deps -- | Find roots of variance. These are memory blocks declared in--- the current scope as well as loop indices of foreach loops.+-- the current scope, loop indices of foreach loops, and the results of+-- nested SegOps. A SegOp in a kernel is run once per program instance,+-- each of which receives its result through a pointer to its own lane+-- of the variable (see 'MC.compileSetRetvalStructValues'). findVarying :: MCCode -> [VName] findVarying (x :>>: y) = findVarying x ++ findVarying y findVarying (If _ x y) = findVarying x ++ findVarying y findVarying (For _ _ x) = findVarying x findVarying (While _ x) = findVarying x findVarying (Op (ForEachActive _ body)) = findVarying body+findVarying (OnFailure body handler) = findVarying body ++ findVarying handler findVarying (Op (ForEach idx _ _ body)) = idx : findVarying body findVarying (DeclareMem mem _) = [mem]+findVarying (Op (SegOp _ _ _ _ retvals _)) = map paramName retvals findVarying _ = [] -- | Analyze variability in a body of code and run an action with@@ -1147,7 +1174,7 @@ analyzeVariability code m = do let roots = findVarying code let deps = depsFixedPoint $ execVariabilityM $ findDeps code- let safelist = M.filter (\b -> all (`notNameIn` b) roots) deps+ let safelist = M.filterWithKey (\v b -> v `notElem` roots && all (`notNameIn` b) roots) deps let safe = namesFromList $ M.keys safelist pre_state <- GC.getUserState GC.modifyUserState (\s -> s {sUniform = safe})
src/Futhark/CodeGen/Backends/MulticoreWASM.hs view
@@ -4,22 +4,14 @@ -- JavaScript wrapper. module Futhark.CodeGen.Backends.MulticoreWASM ( compileProg,- runServer,- libraryExports,- GC.CParts (..),- GC.asLibrary,- GC.asExecutable,- GC.asServer, ) where -import Data.Maybe import Data.Text qualified as T import Futhark.CodeGen.Backends.GenericC qualified as GC import Futhark.CodeGen.Backends.GenericWASM import Futhark.CodeGen.Backends.MulticoreC qualified as MC import Futhark.CodeGen.Backends.MulticoreC.Boilerplate (generateBoilerplate)-import Futhark.CodeGen.ImpCode.Multicore qualified as Imp import Futhark.CodeGen.ImpGen.Multicore qualified as ImpGen import Futhark.IR.MCMem import Futhark.MonadFreshNames@@ -32,10 +24,11 @@ -- * Generated C code (to be passed to Emscripten). -- -- * JavaScript wrapper code that presents a nicer interface to the--- Emscripten-produced code (this should be put in a @.class.js@--- file by itself).+-- Emscripten-produced code (passed to @emcc@ with+-- @--extern-post-js@). ----- * Options that should be passed to @emcc@.+-- * The names of the C functions that must be exported from the+-- WebAssembly module. compileProg :: (MonadFreshNames m) => T.Text ->@@ -58,20 +51,7 @@ pure ( ws, ( prog'',- javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')),+ javascriptWrapper (GC.cManifest prog''), "_futhark_context_config_set_num_threads" : emccExportNames (GC.cManifest prog'') ) )--fRepMyRep :: Imp.Definitions Imp.Multicore -> [JSEntryPoint]-fRepMyRep prog =- let Imp.Functions fs = Imp.defFuns prog- function fun = do- Imp.EntryPoint n res args _ <- Imp.functionEntry fun- Just $- JSEntryPoint- { name = nameToString n,- parameters = map (extToString . snd) args,- ret = extToString $ snd res- }- in mapMaybe (function . snd) fs
src/Futhark/CodeGen/Backends/PyOpenCL.hs view
@@ -233,6 +233,11 @@ callKernel (Imp.GetSizeMax v size_class) = do v' <- compileVar v stm $ Assign v' $ kernelConstToExp $ Imp.SizeMaxConst size_class+callKernel (Imp.GetBookkeeping v b) = do+ v' <- compileVar v+ stm . Assign v' . Var $ case b of+ Imp.BookkeepingCounters -> "self.bookkeeping_counters"+ Imp.BookkeepingLocks -> "self.bookkeeping_locks" callKernel (Imp.LaunchKernel safety name shared_memory args num_threadblocks workgroup_size) = do num_threadblocks' <- mapM (fmap asLong . compileExp) num_threadblocks workgroup_size' <- mapM compileBlockDim workgroup_size
src/Futhark/CodeGen/Backends/PyOpenCL/Boilerplate.hs view
@@ -11,13 +11,15 @@ import Futhark.CodeGen.Backends.GenericPython qualified as Py import Futhark.CodeGen.Backends.GenericPython.AST import Futhark.CodeGen.ImpCode.OpenCL- ( ErrorMsg (..),+ ( Bookkeeping (..),+ ErrorMsg (..), ErrorMsgPart (..), FailureMsg (..), KernelConst (..), KernelConstExp, ParamMap, PrimType (..),+ bookkeepingSize, errorMsgArgTypes, untyped, )@@ -57,6 +59,8 @@ [text| size_heuristics=$size_heuristics self.global_failure_args_max = $max_num_args+self.bookkeeping_counters_size = $num_counters+self.bookkeeping_locks_size = $num_locks self.failure_msgs=$failure_msgs constants = $constants' program = initialise_opencl_object(self,@@ -84,6 +88,8 @@ types' = prettyText $ map (show . prettyString) types -- Looks enough like Python. sizes' = prettyText $ Py.sizeClassesToPython sizes max_num_args = prettyText $ foldl max 0 $ map (errorMsgNumArgs . failureError) failures+ num_counters = prettyText $ bookkeepingSize BookkeepingCounters+ num_locks = prettyText $ bookkeepingSize BookkeepingLocks failure_msgs = prettyText $ List $ map formatFailure failures onConstant (name, e) = Tuple
src/Futhark/CodeGen/Backends/SequentialWASM.hs view
@@ -4,16 +4,9 @@ -- JavaScript wrapper. module Futhark.CodeGen.Backends.SequentialWASM ( compileProg,- runServer,- libraryExports,- GC.CParts (..),- GC.asLibrary,- GC.asExecutable,- GC.asServer, ) where -import Data.Maybe import Data.Text qualified as T import Futhark.CodeGen.Backends.GenericC qualified as GC import Futhark.CodeGen.Backends.GenericWASM@@ -31,10 +24,11 @@ -- * Generated C code (to be passed to Emscripten). -- -- * JavaScript wrapper code that presents a nicer interface to the--- Emscripten-produced code (this should be put in a @.class.js@--- file by itself).+-- Emscripten-produced code (passed to @emcc@ with+-- @--extern-post-js@). ----- * Options that should be passed to @emcc@.+-- * The names of the C functions that must be exported from the+-- WebAssembly module. compileProg :: (MonadFreshNames m) => T.Text -> Prog SeqMem -> m (ImpGen.Warnings, (GC.CParts, T.Text, [String])) compileProg version prog = do (ws, prog') <- ImpGen.compileProg prog@@ -49,23 +43,10 @@ (DefaultSpace, [DefaultSpace]) [] prog'- pure (ws, (prog'', javascriptWrapper (fRepMyRep prog') (opaqueToJS (Imp.defTypes prog')), emccExportNames (GC.cManifest prog'')))+ pure (ws, (prog'', javascriptWrapper (GC.cManifest prog''), emccExportNames (GC.cManifest prog''))) where operations :: GC.Operations Imp.Sequential () operations = GC.defaultOperations { GC.opsCompiler = const $ pure () }--fRepMyRep :: Imp.Program -> [JSEntryPoint]-fRepMyRep prog =- let Imp.Functions fs = Imp.defFuns prog- function (Imp.Function entry _ _ _ _) = do- Imp.EntryPoint n res args _doc <- entry- Just $- JSEntryPoint- { name = nameToString n,- parameters = map (extToString . snd) args,- ret = extToString $ snd res- }- in mapMaybe (function . snd) fs
src/Futhark/CodeGen/ImpCode.hs view
@@ -146,7 +146,7 @@ data Param = MemParam VName Space | ScalarParam VName PrimType- deriving (Eq, Show)+ deriving (Eq, Ord, Show) -- | The name of a parameter. paramName :: Param -> VName@@ -337,6 +337,12 @@ -- out not to be true, then report a failure along with -- the given error message. Assert Exp (ErrorMsg Exp) (Loc, [Loc])+ | -- | @OnFailure body handler@ executes @body@, and if it fails, executes+ -- @handler@ before the failure propagates as usual. This is for code that+ -- must still be executed after a failure, such as releasing a lock. The+ -- handler must not fail. In a GPU kernel, only the failing thread executes+ -- the handler, so @body@ must not contain barriers.+ OnFailure (Code a) (Code a) | -- | Print the given value to the screen, somehow -- annotated with the given string as a description. If -- no type/value pair, just print the string. This has@@ -390,6 +396,7 @@ go f (If _ x y) = f x <> f y go f (For _ _ x) = f x go f (While _ x) = f x+ go f (OnFailure x y) = f x <> f y go _ _ = mempty declared (DeclareMem mem space) =@@ -415,6 +422,7 @@ listify (If cond c1 c2) = [If cond (declsFirst c1) (declsFirst c2)] listify (For i e c) = [For i e (declsFirst c)] listify (While cond c) = [While cond (declsFirst c)]+ listify (OnFailure c1 c2) = [OnFailure (declsFirst c1) (declsFirst c2)] listify c = [c] isDecl (DeclareScalar {}) = True isDecl (DeclareMem {}) = True@@ -429,6 +437,7 @@ calledFuncs f (If _ x y) = calledFuncs f x <> calledFuncs f y calledFuncs f (For _ _ x) = calledFuncs f x calledFuncs f (While _ x) = calledFuncs f x+calledFuncs f (OnFailure x y) = calledFuncs f x <> calledFuncs f y calledFuncs _ _ = mempty -- | Compute call graph, as per 'calledFuncs', but also include@@ -449,6 +458,7 @@ foldProvenances f (For _ _ c) = foldProvenances f c foldProvenances f (While _ c) = foldProvenances f c foldProvenances f (If _ x y) = foldProvenances f x <> foldProvenances f y+foldProvenances f (OnFailure x y) = foldProvenances f x <> foldProvenances f y foldProvenances f (Op x) = f x foldProvenances _ _ = mempty @@ -643,6 +653,13 @@ pretty dest <+> "<-" <+> pretty from <+> "@" <> pretty space pretty (Assert e msg _) = "assert" <> parens (commasep [pretty msg, pretty e])+ pretty (OnFailure body handler) =+ "on_failure"+ <+> "{"+ </> indent 2 (pretty body)+ </> "} handler {"+ </> indent 2 (pretty handler)+ </> "}" pretty (Copy t shape (dst, dstspace) (dstoffset, dststrides) (src, srcspace) (srcoffset, srcstrides)) = ("lmadcopy_" <> pretty (length shape) <> "d_" <> pretty t) <> (parens . align)@@ -732,6 +749,8 @@ While cond <$> traverse f code traverse f (If cond x y) = If cond <$> traverse f x <*> traverse f y+ traverse f (OnFailure x y) =+ OnFailure <$> traverse f x <*> traverse f y traverse f (Op kernel) = Op <$> f kernel traverse _ Skip =@@ -779,6 +798,7 @@ declaredIn (x :>>: y) = declaredIn x <> declaredIn y declaredIn (For i _ body) = oneName i <> declaredIn body declaredIn (While _ body) = declaredIn body+declaredIn (OnFailure body handler) = declaredIn body <> declaredIn handler declaredIn _ = mempty instance FreeIn EntryPoint where@@ -837,6 +857,8 @@ freeIn' cond <> freeIn' t <> freeIn' f freeIn' (Assert e msg _) = freeIn' e <> foldMap freeIn' msg+ freeIn' (OnFailure body handler) =+ freeIn' body <> freeIn' handler freeIn' (Op op) = freeIn' op freeIn' (Meta {}) =
src/Futhark/CodeGen/ImpCode/GPU.hs view
@@ -7,6 +7,8 @@ KernelConst (..), KernelConstExp, HostOp (..),+ Bookkeeping (..),+ bookkeepingSize, KernelOp (..), Fence (..), AtomicOp (..),@@ -48,8 +50,29 @@ | GetSize VName Name SizeClass | CmpSizeLe VName Name SizeClass Exp | GetSizeMax VName SizeClass+ | -- | Bind the given device memory to one of the bookkeeping buffers of the+ -- context.+ GetBookkeeping VName Bookkeeping deriving (Show) +-- | The bookkeeping buffers are device buffers of 32-bit integers that belong+-- to the context and are shared by all kernels. They are zero whenever no+-- kernel is running, and a kernel that uses them must leave them zeroed when it+-- succeeds, and the context zeroes them again after a failure. As kernels run+-- serially, they never compete for them. If we ever add support for multiple+-- streams, we must have a set of buffers per stream.+data Bookkeeping+ = -- | Counters, e.g. for detecting the last block to finish.+ BookkeepingCounters+ | -- | Locks for operations that cannot be done atomically.+ BookkeepingLocks+ deriving (Eq, Ord, Show)++-- | The number of 32-bit integers in a bookkeeping buffer.+bookkeepingSize :: Bookkeeping -> Integer+bookkeepingSize BookkeepingCounters = 1024+bookkeepingSize BookkeepingLocks = 100151+ -- | The size of one dimension of a block. type BlockDim = Either Exp KernelConstExp @@ -114,6 +137,10 @@ <> parens (commasep [pretty key, pretty size_class]) pretty (GetSizeMax dest size_class) = pretty dest <+> "<-" <+> "get_size_max" <> parens (pretty size_class)+ pretty (GetBookkeeping dest BookkeepingCounters) =+ pretty dest <+> "<-" <+> "bookkeeping_counters()"+ pretty (GetBookkeeping dest BookkeepingLocks) =+ pretty dest <+> "<-" <+> "bookkeeping_locks()" pretty (CmpSizeLe dest name size_class x) = pretty dest <+> "<-"@@ -133,6 +160,8 @@ freeIn' dest freeIn' (GetSize dest _ _) = freeIn' dest+ freeIn' (GetBookkeeping dest _) =+ freeIn' dest instance FreeIn Kernel where freeIn' kernel =@@ -181,13 +210,14 @@ -- warp. Only supported for CUDA and HIP. WarpSync | SharedAlloc VName (Count Bytes (TExp Int64))- | -- | Perform a barrier and also check whether any- -- threads have failed an assertion. Make sure all- -- threads would reach all 'ErrorSync's if any of them- -- do. A failing assertion will jump to the next- -- following 'ErrorSync', so make sure it's not inside- -- control flow or similar.- ErrorSync Fence+ | -- | Perform a barrier and also check whether any threads have failed an+ -- assertion. If so, all threads of the block execute the given code and+ -- then stop. The code must not fail, and is usually empty. Make sure all+ -- threads would reach all 'ErrorSync's if any of them do. A failing+ -- assertion will jump to the next following 'ErrorSync' (unless it is in+ -- the body of an 'OnFailure'), so make sure it's not inside control flow or+ -- similar.+ ErrorSync Fence (Code KernelOp) deriving (Show) -- | Atomic operations return the value stored before the update. This@@ -257,10 +287,14 @@ "warp_sync()" pretty (SharedAlloc name size) = pretty name <+> equals <+> "shared_alloc" <> parens (pretty size)- pretty (ErrorSync FenceLocal) =- "error_sync_local()"- pretty (ErrorSync FenceGlobal) =- "error_sync_global()"+ pretty (ErrorSync f handler) =+ sync f <> handler'+ where+ sync FenceLocal = "error_sync_local()"+ sync FenceGlobal = "error_sync_global()"+ handler' = case handler of+ Skip -> mempty+ _ -> " on_failure" <> brace (pretty handler) pretty (Atomic _ (AtomicAdd t old arr ind x)) = pretty old <+> "<-"@@ -335,6 +369,7 @@ instance FreeIn KernelOp where freeIn' (Atomic _ op) = freeIn' op freeIn' (SharedAlloc _ size) = freeIn' size+ freeIn' (ErrorSync _ handler) = freeIn' handler freeIn' _ = mempty brace :: Doc a -> Doc a
src/Futhark/CodeGen/ImpCode/Kernels.hs view
@@ -5,6 +5,8 @@ KernelArg (..), HostCode, HostOp (..),+ Bookkeeping (..),+ bookkeepingSize, KernelSafety (..), numFailureParams, FailureMsg (..),@@ -17,7 +19,13 @@ where import Futhark.CodeGen.ImpCode-import Futhark.CodeGen.ImpCode.GPU (BlockDim, KernelConst (..), KernelConstExp)+import Futhark.CodeGen.ImpCode.GPU+ ( BlockDim,+ Bookkeeping (..),+ KernelConst (..),+ KernelConstExp,+ bookkeepingSize,+ ) import Futhark.IR.GPU.Sizes import Futhark.Util.Pretty @@ -73,6 +81,7 @@ | GetSize VName Name | CmpSizeLe VName Name Exp | GetSizeMax VName SizeClass+ | GetBookkeeping VName Bookkeeping deriving (Show) instance Pretty HostOp where
src/Futhark/CodeGen/ImpCode/Multicore.hs view
@@ -206,6 +206,7 @@ go f (If _ x y) = f x <> f y go f (For _ _ x) = f x go f (While _ x) = f x+ go f (OnFailure x y) = f x <> f y go f (Op op) = goOp f op go _ _ = mempty
src/Futhark/CodeGen/ImpGen.hs view
@@ -50,6 +50,7 @@ lookupAcc, askAttrs, askProvenance,+ bodyMayFail, -- * Building Blocks TV,@@ -126,6 +127,7 @@ import Control.Monad import Control.Monad.Reader import Control.Monad.State+import Control.Monad.Writer (execWriter, tell) import Control.Parallel.Strategies import Data.Bifunctor (first) import Data.DList qualified as DL@@ -134,6 +136,7 @@ import Data.List.NonEmpty (NonEmpty (..)) import Data.Map.Strict qualified as M import Data.Maybe+import Data.Monoid (Any (..)) import Data.Set qualified as S import Data.String import Data.Text qualified as T@@ -1354,6 +1357,17 @@ -- | The provenance of whatever we are currently generating code for. askProvenance :: ImpM rep r op Provenance askProvenance = asks envProvenance++-- | Can executing this body fail? This is the case if it contains an+-- assertion, or calls a non-builtin function, which we conservatively assume+-- may fail. Assertions inside 'Op's are not considered.+bodyMayFail :: (ASTRep rep) => Body rep -> Bool+bodyMayFail = any (expMayFail . stmExp) . bodyStms+ where+ expMayFail (BasicOp Assert {}) = True+ expMayFail (Apply fname _ _ _) = not $ isBuiltInFunction fname+ expMayFail e = getAny $ execWriter $ walkExpM walker e+ walker = identityWalker {walkOnBody = \_ -> tell . Any . bodyMayFail} -- | Wrap any code emitted in the enclosed section with the current provenance, -- if any.
src/Futhark/CodeGen/ImpGen/GPU.hs view
@@ -239,9 +239,8 @@ locksForInputs atomics ((c, (_, _, op)) : inputs') | Just (op_lam, _) <- op, AtomicLocking _ <- atomicUpdateLocking atomics op_lam = do- let num_locks = 100151- locks_arr <- genZeroes "withacc_locks" num_locks- let locks = Locks locks_arr num_locks+ locks_arr <- bookkeepingArray Imp.BookkeepingLocks+ let locks = Locks locks_arr $ fromInteger $ Imp.bookkeepingSize Imp.BookkeepingLocks extend env = env {hostLocks = M.insert c locks $ hostLocks env} localEnv extend $ locksForInputs atomics inputs' | otherwise =
src/Futhark/CodeGen/ImpGen/GPU/Base.hs view
@@ -18,6 +18,7 @@ warpLocal, blockScan, blockLoop,+ peeledKernelLoop, isActive, sKernel, sKernelThread,@@ -31,7 +32,7 @@ blockCoverSpace, fenceForArrays, updateAcc,- genZeroes,+ bookkeepingArray, isPrimParam, kernelConstToExp, getSize,@@ -107,9 +108,10 @@ -- | A mapping from dimensions of nested SegOps to already -- computed local thread IDs. Only valid in non-virtualised case. kernelLocalIdMap :: M.Map [SubExp] [Imp.TExp Int32],- -- | Mapping from dimensions of nested SegOps to how many- -- iterations the virtualisation loop needs.- kernelChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)+ -- | Mapping from dimensions of nested SegOps to how many full block-sized+ -- chunks the virtualisation loop has; see 'peeledKernelLoop'. Constant dimensions+ -- are not included.+ kernelFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32) } kernelGlobalThreadId, kernelLocalThreadId, kernelBlockId :: KernelConstants -> Imp.TExp Int32@@ -180,15 +182,16 @@ Nothing -> error $ "Missing locks for " ++ prettyString acc --- | Generate a constant device array of 32-bit integer zeroes with--- the given number of elements. Initialised with a replicate.-genZeroes :: Name -> Int -> CallKernelGen VName-genZeroes desc n = genConstants $ do- counters_mem <- sAlloc (desc <> "_mem") (4 * fromIntegral n) (Space "device")- let shape = Shape [intConst Int64 (fromIntegral n)]- counters <- sArrayInMem desc int32 shape counters_mem- sReplicate counters $ intConst Int32 0- pure (namesFromList [counters_mem], counters)+-- | Retrieve a bookkeeping array.+bookkeepingArray :: Imp.Bookkeeping -> CallKernelGen VName+bookkeepingArray b = do+ mem <- sDeclareMem (desc <> "_mem") (Space "device")+ sOp $ Imp.GetBookkeeping mem b+ sArrayInMem desc int32 (Shape [intConst Int64 $ Imp.bookkeepingSize b]) mem+ where+ desc = case b of+ Imp.BookkeepingCounters -> "counters"+ Imp.BookkeepingLocks -> "locks" compileThreadExp :: ExpCompiler GPUMem KernelEnv Imp.KernelOp compileThreadExp (Pat [pe]) (BasicOp (Opaque _ se)) =@@ -226,6 +229,28 @@ i <- dPrimVE "i" $ chunk_i * num_threads + tid sWhen (i .<. n) $ f i +-- | @peeledKernelLoop tid num_threads num_full_chunks n f@ invokes @f@ on the+-- iterations @tid@, @tid+num_threads@, and so on, below @n@, where+-- @num_full_chunks@ must be @n `quot` num_threads@. Only the final partial+-- chunk is bounds checked, so the body is generated twice; see Note [Peeled+-- block loops]. The body must contain thread-level code.+peeledKernelLoop ::+ (IntExp t) =>+ Imp.TExp t ->+ Imp.TExp t ->+ Imp.TExp t ->+ Imp.TExp t ->+ (Imp.TExp t -> InKernelGen ()) ->+ InKernelGen ()+peeledKernelLoop tid num_threads num_full_chunks n f =+ localOps threadOperations $ do+ sFor "chunk_i" num_full_chunks $ \chunk_i ->+ f =<< dPrimVE "i" (sMinAs (chunk_i * num_threads + tid) (n - 1))+ i <- dPrimVE "i" $ num_full_chunks * num_threads + tid+ sWhen (i .<. n) $ f i+ where+ sMinAs x y = TPrimExp $ BinOpExp (SMin $ expIntType x) (untyped x) (untyped y)+ -- | Assign iterations of a for-loop to threads in the threadblock. The -- passed-in function is invoked with the (symbolic) iteration. For -- multidimensional loops, use 'blockCoverSpace'.@@ -236,11 +261,13 @@ InKernelGen () blockLoop n f = do constants <- kernelConstants <$> askEnv- kernelLoop- (kernelLocalThreadId constants `sExtAs` n)- (kernelBlockSize constants `sExtAs` n)- n- f+ let tid = kernelLocalThreadId constants `sExtAs` n+ num_threads = kernelBlockSize constants `sExtAs` n+ if n == num_threads+ then localOps threadOperations $ f tid+ else do+ num_full_chunks <- dPrimVE "num_full_chunks" $ n `quot` num_threads+ peeledKernelLoop tid num_threads num_full_chunks n f -- | Iterate collectively though a multidimensional space, such that -- all threads in the block participate. The passed-in function is@@ -322,18 +349,17 @@ Imp.TExp Int32 -> Imp.TExp Bool -> [VName] ->- InKernelGen () ->+ Imp.Fence -> Lambda GPUMem -> InKernelGen ()-inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs barrier scan_lam = do+inChunkScan constants seg_flag arrs_full_size lockstep_width block_size active arrs fence scan_lam = do skip_threads <- dPrim "skip_threads" let actual_params = lambdaParams scan_lam (x_params, y_params) = splitAt (length actual_params `div` 2) actual_params y_to_x = forM_ (zip x_params y_params) $ \(x, y) ->- when (isPrimParam x) $- copyDWIM (paramName x) [] (Var (paramName y)) []+ copyDWIM (paramName x) [] (Var (paramName y)) [] -- Set initial y values sComment "read input for in-block scan" $@@ -366,25 +392,32 @@ $ compileBody' x_params $ lambdaBody scan_lam - maybeBarrier =- sWhen- (lockstep_width .<=. tvExp skip_threads)- barrier+ -- See Note [Failing block-level operators].+ maybeBarrier+ | may_fail = sOp $ Imp.ErrorSync fence mempty+ | otherwise =+ sWhen+ (lockstep_width .<=. tvExp skip_threads)+ barrier + warpLocal'+ | may_fail = id+ | otherwise = warpLocal+ sComment "in-block scan (hopefully no barriers needed)" $ do skip_threads <-- 1 sWhile (tvExp skip_threads .<. block_size) $ do thread_active <- dPrimVE "thread_active" $ tvExp skip_threads .<=. in_block_id .&&. active - warpLocal $ do+ warpLocal' $ do sWhen thread_active . sComment "read operands" $ zipWithM_ (readParam (sExt64 $ tvExp skip_threads)) x_params arrs sComment "perform operation" $ op_to_x thread_active maybeBarrier - warpLocal . sWhen thread_active . sComment "write result" $+ warpLocal' . sWhen thread_active . sComment "write result" $ sequence_ $ zipWith3 writeResult x_params y_params arrs @@ -397,9 +430,15 @@ ltid32 = kernelLocalThreadId constants ltid = sExt64 ltid32 array_scan = not $ all primType $ lambdaReturnType scan_lam+ barrier = sOp $ Imp.Barrier fence+ may_fail = bodyMayFail $ lambdaBody scan_lam - readInitial p arr =+ readInitial p arr = do copyDWIMFix (paramName p) [] (Var arr) [ltid]+ -- An array-typed element is also read @arrs_full_size@ elements along,+ -- by 'readParam', so put it there too.+ unless (isPrimParam p) $+ copyDWIMFix arr [ltid + arrs_full_size] (Var arr) [ltid] readParam behind p arr | isPrimParam p =@@ -408,10 +447,38 @@ copyDWIMFix (paramName p) [] (Var arr) [ltid - behind + arrs_full_size] writeResult x y arr = do- when (isPrimParam x) $- copyDWIMFix arr [ltid] (Var $ paramName x) []+ copyDWIMFix arr [ltid] (Var $ paramName x) []+ unless (isPrimParam x) $+ copyDWIMFix arr [ltid + arrs_full_size] (Var $ paramName x) [] copyDWIM (paramName y) [] (Var $ paramName x) [] +-- | 'blockScan' splits the block into chunks of this size, which are scanned+-- separately. The only requirement is that it should not be less than the+-- square root of the block size. With 32, we will work on blocks of size 1024+-- or smaller, which fits every device Troels has seen. Still, it would be nicer+-- if it were a runtime parameter. Some day.+blockScanChunkSize :: (Num a) => a+blockScanChunkSize = 32++-- | Scan the @w@ first elements of the given arrays, in place. @w@ must be at+-- most the thread block size; you'll need to wrap this function in a loop if+-- you want to scan larger arrays.+--+-- An array-typed element is kept twice: at index @i@, where the result is+-- expected, and again at index @i + arrs_full_size@. It is the latter copy that+-- a thread reads as its x operand, because the operator writes its result+-- through its x parameter, which may well be the former. With @k@ chunks, the+-- first @k@ elements are overwritten while computing the per-chunk carry-ins,+-- so they are saved at indexes @arrs_full_size + k@ to @arrs_full_size + 2k@,+-- whose copies are not needed by then. Only the @w@ first elements need be+-- within the /shape/ of the arrays, but the indexes from @arrs_full_size@ to+-- @arrs_full_size + w@ must also be backed by memory; see+-- 'Futhark.CodeGen.ImpGen.GPU.Block.blockScanElems'.+--+-- The callers in "Futhark.CodeGen.ImpGen.GPU.SegScan.TwoPass" and+-- "Futhark.CodeGen.ImpGen.GPU.SegRed" pass arrays whose elements /are/ the x+-- and y parameters of the operator, so for them the copies that keep the two in+-- step are redundant, but harmless. blockScan :: Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) -> Imp.TExp Int64 ->@@ -419,7 +486,7 @@ Lambda GPUMem -> [VName] -> InKernelGen ()-blockScan seg_flag arrs_full_size w lam arrs = do+blockScan seg_flag arrs_full_size w lam arrs = localOps threadOperations $ do constants <- kernelConstants <$> askEnv renamed_lam <- renameLambda lam @@ -437,12 +504,7 @@ -- scanned separately. Typically, these chunks are at most the -- lockstep width, which enables barrier-free execution inside them. --- -- We hardcode the chunk size here. The only requirement is that it- -- should not be less than the square root of the block size. With- -- 32, we will work on blocks of size 1024 or smaller, which fits- -- every device Troels has seen. Still, it would be nicer if it were- -- a runtime parameter. Some day.- let chunk_size = 32+ let chunk_size = blockScanChunkSize simd_width = kernelWaveSize constants chunk_id = ltid32 `quot` chunk_size in_chunk_id = ltid32 - chunk_id * chunk_size@@ -455,19 +517,18 @@ chunk_size active arrs- barrier+ barrier_fence array_scan = not $ all primType $ lambdaReturnType lam- barrier- | array_scan =- sOp $ Imp.Barrier Imp.FenceGlobal- | otherwise =- sOp $ Imp.Barrier fence+ barrier_fence+ | array_scan = Imp.FenceGlobal+ | otherwise = fence+ barrier = sOp $ Imp.Barrier barrier_fence errorsync | array_scan =- sOp $ Imp.ErrorSync Imp.FenceGlobal+ sOp $ Imp.ErrorSync Imp.FenceGlobal mempty | otherwise =- sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty writeBlockResult p arr = copyDWIMFix arr [sExt64 chunk_id] (Var $ paramName p) []@@ -478,19 +539,24 @@ doInChunkScan seg_flag ltid_in_bounds lam barrier + -- With more than one chunk, the first 'num_chunks' elements are overwritten+ -- by the chunk results, which are then scanned to produce the carry-ins.+ num_chunks <- dPrimVE "num_chunks" $ sExt32 $ w `ceilDiv` blockScanChunkSize let is_first_block = chunk_id .==. 0+ several_chunks = num_chunks .>. 1+ scans_carries = several_chunks .&&. ltid32 .<. num_chunks when array_scan $ do sComment "save correct values for first block" $- sWhen is_first_block $+ sWhen scans_carries $ forM_ (zip x_params arrs) $ \(x, arr) -> unless (isPrimParam x) $- copyDWIMFix arr [arrs_full_size + sExt64 chunk_size + ltid] (Var $ paramName x) []+ copyDWIMFix arr [arrs_full_size + sExt64 num_chunks + ltid] (Var $ paramName x) [] barrier let last_in_block = in_chunk_id .==. chunk_size - 1 sComment "last thread of block 'i' writes its result to offset 'i'" $- sWhen (last_in_block .&&. ltid_in_bounds) $+ sWhen (several_chunks .&&. last_in_block .&&. ltid_in_bounds) $ everythingVolatile $ zipWithM_ writeBlockResult x_params arrs @@ -502,20 +568,20 @@ flag_true (from * chunk_size + chunk_size - 1) (to * chunk_size + chunk_size - 1) sComment "scan the first block, after which offset 'i' contains carry-in for block 'i+1'"- $ doInChunkScan first_block_seg_flag (is_first_block .&&. ltid_in_bounds) renamed_lam+ $ doInChunkScan first_block_seg_flag scans_carries renamed_lam errorsync when array_scan $ do sComment "move correct values for first block back a block" $- sWhen is_first_block $+ sWhen scans_carries $ forM_ (zip x_params arrs) $ \(x, arr) -> unless (isPrimParam x) $ copyDWIMFix arr [arrs_full_size + ltid] (Var arr)- [arrs_full_size + sExt64 chunk_size + ltid]+ [arrs_full_size + sExt64 num_chunks + ltid] barrier @@ -541,8 +607,7 @@ write_final_result = forM_ (zip x_params arrs) $ \(p, arr) ->- when (isPrimParam p) $- copyDWIMFix arr [ltid] (Var $ paramName p) []+ copyDWIMFix arr [ltid] (Var $ paramName p) [] sComment "carry-in for every block except the first" $ localOps threadOperations $ do@@ -550,17 +615,20 @@ sComment "perform operation" op_to_x sComment "write final result" $ sUnless no_carry_in write_final_result - barrier+ -- See Note [Failing block-level operators].+ sOp $ Imp.ErrorSync barrier_fence mempty sComment "restore correct values for first block" $- sWhen (is_first_block .&&. ltid_in_bounds) $- forM_ (zip3 x_params y_params arrs) $ \(x, y, arr) ->+ sWhen scans_carries $+ forM_ (zip3 x_params y_params arrs) $ \(_x, y, arr) -> if isPrimParam y then copyDWIMFix arr [ltid] (Var $ paramName y) []- else copyDWIMFix (paramName x) [] (Var arr) [arrs_full_size + ltid]+ else copyDWIMFix arr [ltid] (Var arr) [arrs_full_size + ltid] barrier +-- | Reduce the @w@ first elements of the given arrays, leaving the result in+-- element zero of each. The others are clobbered. blockReduce :: Imp.TExp Int32 -> Lambda GPUMem ->@@ -580,26 +648,18 @@ constants <- kernelConstants <$> askEnv let local_tid = kernelLocalThreadId constants-- barrier- | all primType $ lambdaReturnType lam = sOp $ Imp.Barrier Imp.FenceLocal- | otherwise = sOp $ Imp.Barrier Imp.FenceGlobal+ may_fail = bodyMayFail $ lambdaBody lam errorsync- | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal- | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal+ | all primType $ lambdaReturnType lam = sOp $ Imp.ErrorSync Imp.FenceLocal mempty+ | otherwise = sOp $ Imp.ErrorSync Imp.FenceGlobal mempty readReduceArgument param arr = do let i = local_tid + tvExp offset copyDWIMFix (paramName param) [] (Var arr) [sExt64 i] writeReduceOpResult param arr =- when (isPrimParam param) $- copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []-- writeArrayOpResult param arr =- unless (isPrimParam param) $- copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) []+ copyDWIMFix arr [sExt64 local_tid] (Var $ paramName param) [] let (reduce_acc_params, reduce_arr_params) = splitAt (length arrs) $ lambdaParams lam@@ -622,7 +682,10 @@ sComment "write result of operation" $ zipWithM_ writeReduceOpResult reduce_acc_params arrs - wave_size = kernelWaveSize constants+ -- See Note [Failing block-level operators].+ wave_size+ | may_fail = 1+ | otherwise = kernelWaveSize constants tblock_size = kernelBlockSize constants wave_id = local_tid `quot` wave_size in_wave_id = local_tid - wave_id * wave_size@@ -652,20 +715,14 @@ arg_in_bounds .&&. is_first_thread_in_wave .&&. wave_not_skipped cross_wave_reductions = sWhile doing_cross_wave_reductions $ do- barrier offset <-- tvExp skip_waves * wave_size sWhen apply_in_cross_wave_iteration do_reduce+ errorsync skip_waves <-- tvExp skip_waves * 2 - in_wave_reductions- cross_wave_reductions+ unless may_fail in_wave_reductions errorsync-- unless (all isPrimParam reduce_acc_params) $- sComment "Copy array-typed operands to result array" $- sWhen (local_tid .==. 0) $- localOps threadOperations $- zipWithM_ writeArrayOpResult reduce_acc_params arrs+ cross_wave_reductions compileThreadOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp compileThreadOp pat (Alloc size space) =@@ -864,8 +921,11 @@ acquire_fence dLParams acc_params bind_acc_params- op_body- do_hist+ -- A thread whose operator fails must still release the lock, or other+ -- threads would wait for it forever.+ update <- collect $ op_body >> do_hist+ on_failure <- collect release_lock+ emit $ Imp.OnFailure update on_failure release_lock break_loop where@@ -1005,7 +1065,7 @@ kernelNumThreads = sExt32 (tblock_size' * num_tblocks'), kernelWaveSize = Imp.le32 wave_size, kernelLocalIdMap = mempty,- kernelChunkItersMap = mempty+ kernelFullChunksMap = mempty } let set_constants = do@@ -1095,7 +1155,7 @@ kernelNumThreads = sExt32 (tblock_size' * num_tblocks'), kernelWaveSize = 0, kernelLocalIdMap = mempty,- kernelChunkItersMap = mempty+ kernelFullChunksMap = mempty } wrapKernel m = do@@ -1142,7 +1202,7 @@ (tvExp phys_tblock_id + i * sExt32 (kernelNumBlocks constants)) -- Make sure the virtual block is actually done before we let -- another virtual block have its way with it.- sOp $ Imp.ErrorSync Imp.FenceGlobal+ sOp $ Imp.ErrorSync Imp.FenceGlobal mempty virtualiseBlocks _ _ m = m . tvExp . kernelBlockIdVar . kernelConstants =<< askEnv @@ -1477,3 +1537,41 @@ copyDWIMFix (patElemName pe) is what [] compileThreadResult _ _ TileReturns {} = compilerBugS "compileThreadResult: TileReturns unhandled."++-- Note [Peeled block loops]+--+-- When the threads of a block cover an iteration space larger than the block+-- (virtualised SegOps), each thread handles one iteration per block-sized+-- chunk, and only the final chunk can be partial. A bounds check in every chunk+-- keeps the kernel compiler from overlapping the memory accesses of consecutive+-- chunks, which makes memory-bound loops considerably slower.+-- 'peeledKernelLoop' therefore runs the full chunks without a check, and+-- generates the body again for the final chunk, with a bounds check.+--+-- A bounds check also tells the kernel compiler that the iteration is in+-- bounds, which lets it remove checks in the body that follow from it, such as+-- those of accumulator updates, which likewise keep memory accesses from+-- overlapping. To retain that information, the iteration is clamped to @n-1@ in+-- the full chunks, which does not change its value. This is a bit of a hack,+-- but it seems robust.+--+-- For the same reason, the number of full chunks should be computed in the+-- kernel when it is a constant, rather than be passed in as a kernel argument,+-- so that the kernel compiler can unroll the loop. This is why+-- 'kernelFullChunksMap' does not contain constant dimensions.++-- Note [Failing block-level operators]+--+-- Nothing prevents reduction/scan operators from having arbitrary assertions+-- inside them, which is a bit complicated to handle due to the intricate+-- synchronisation we perform during collective block-level reductions and+-- scans.+--+-- The problem is that we implement failures by jumping to the next+-- 'Imp.ErrorSync' in the program text, so no other barrier, nor the warp-level+-- synchronisation of 'warpLocal', must occur in between (see "Bounds Checking+-- on GPU", HLPP '20). In 'blockReduce' and 'blockScan', we handle operators+-- that may fail ('bodyMayFail') by skipping the warp-level phase (which is just+-- an optimisation anyway) and following each application with an+-- 'Imp.ErrorSync' in the same iteration. For operators that cannot fail, an+-- 'Imp.ErrorSync' is just a barrier.
src/Futhark/CodeGen/ImpGen/GPU/Block.hs view
@@ -29,7 +29,7 @@ import Futhark.IR.Mem.LMAD qualified as LMAD import Futhark.Transform.Rename import Futhark.Util (chunks, mapAccumLM, takeLast)-import Futhark.Util.IntegralExp (ceilDiv, rem)+import Futhark.Util.IntegralExp (ceilDiv, quot, rem) import Prelude hiding (quot, rem) -- | @flattenArray k flat arr@ flattens the outer @k@ dimensions of@@ -103,6 +103,8 @@ (Imp.TExp Int32 -> TV Int64 -> InKernelGen ()) -> InKernelGen () blockChunkLoop w m = do+ -- The loop body contains error syncs, and failures must not jump into it.+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty constants <- kernelConstants <$> askEnv let max_chunk_size = sExt32 $ kernelBlockSize constants num_chunks <- dPrimVE "num_chunks" $ w `ceilDiv` max_chunk_size@@ -123,6 +125,27 @@ crossesSegment segment_size from to = (to - from) .>. (to `rem` sExt32 segment_size) +-- | Given a chunk offset, adjust a segment predicate ('crossesSegment') defined+-- in terms of whole-array indexes to instead be defined in terms of chunk-local+-- indexes. Also returns whether the chunk begins a new segment. This is used to+-- process segmented reductions and scans that are chunked.+chunkSegFlags ::+ Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) ->+ Imp.TExp Int32 ->+ ( Imp.TExp Bool,+ Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool)+ )+chunkSegFlags seg_flag chunk_start =+ ( maybe false (\flag_true -> flag_true (chunk_start - 1) chunk_start) seg_flag,+ fmap shifted seg_flag+ )+ where+ shifted flag_true from to =+ flag_true (chunk_start + from) (chunk_start + to)++-- | Scan the @w@ first elements of the given arrays in-place. The scan is done+-- one block-sized chunk at a time, but @seg_flag@ is passed indexes relative to+-- the whole array. virtualisedBlockScan :: Maybe (Imp.TExp Int32 -> Imp.TExp Int32 -> Imp.TExp Bool) -> Imp.TExp Int32 ->@@ -131,14 +154,9 @@ InKernelGen () virtualisedBlockScan seg_flag w lam arrs = do blockChunkLoop w $ \chunk_start chunk_size -> do- constants <- kernelConstants <$> askEnv- let ltid = kernelLocalThreadId constants- crosses_segment =- case seg_flag of- Nothing -> false- Just flag_true ->- flag_true (sExt32 (chunk_start - 1)) (sExt32 chunk_start)- sComment "possibly incorporate carry" $+ ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+ let (crosses_segment, shifted) = chunkSegFlags seg_flag chunk_start+ sComment "possibly incorporate carry" . localOps threadOperations $ sWhen (chunk_start .>. 0 .&&. ltid .==. 0 .&&. bNot crosses_segment) $ do carry_idx <- dPrimVE "carry_idx" $ sExt64 chunk_start - 1 applyRenamedLambda@@ -150,10 +168,21 @@ arrs_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) arrs - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - blockScan seg_flag (sExt64 w) (tvExp chunk_size) lam arrs_chunks+ -- The chunks past this one are still live, so the second copy that+ -- 'blockScan' keeps of an array-typed element must go beyond the whole+ -- space, not merely beyond the chunk.+ blockScan shifted (sExt64 w) (tvExp chunk_size) lam arrs_chunks +-- | Semantically, a block-level statement finishes before the next one. This+-- means that any statement that writes memory must be enclosed in barriers, as+-- sibling statements may reuse the same memory. Our current approach to this is+-- a bit naive and may result in duplicate barriers. If this ever turns out to+-- be a performance problem, we'll do something more sophisticated.+beforeBlockWrite :: InKernelGen ()+beforeBlockWrite = sOp $ Imp.Barrier Imp.FenceLocal+ copyInBlock :: CopyCompiler GPUMem KernelEnv Imp.KernelOp copyInBlock pt destloc srcloc = do dest_space <- entryMemSpace <$> lookupMemory (memLocName destloc)@@ -179,6 +208,7 @@ (sliceMemLoc destloc destslice') (sliceMemLoc srcloc srcslice') _ -> do+ beforeBlockWrite blockCoverSpace (map sExt32 dims) $ \is -> lmadCopy pt@@ -244,6 +274,19 @@ pure (Just l', f l' (Space "shared") local_subhistos) +-- | The virtualisation that 'blockCoverSegSpace' will actually use for this+-- index space. Maybe we can statically detect that the requested one is really+-- a 'SegNoVirtFull', and generate ever-so-slightly simpler code. Covering an+-- index space twice has to know this, as it decides whether the cover loops -+-- and so introduces a scope of its own for the indexes.+blockCoverVirt :: SegVirt -> SegSpace -> InKernelGen SegVirt+blockCoverVirt virt space = do+ tblock_size <- kernelBlockSize . kernelConstants <$> askEnv+ pure $+ if map pe64 (segSpaceDims space) == [tblock_size]+ then SegNoVirtFull (SegSeqDims [])+ else virt+ blockCoverSegSpace :: SegVirt -> SegSpace -> InKernelGen () -> InKernelGen () blockCoverSegSpace virt space m = do let (ltids, dims) = unzip $ unSegSpace space@@ -251,24 +294,23 @@ constants <- kernelConstants <$> askEnv let tblock_size = kernelBlockSize constants- -- Maybe we can statically detect that this is actually a- -- SegNoVirtFull and generate ever-so-slightly simpler code.- let virt' = if dims' == [tblock_size] then SegNoVirtFull (SegSeqDims []) else virt+ virt' <- blockCoverVirt virt space case virt' of SegVirt -> do- iters <- M.lookup dims . kernelChunkItersMap . kernelConstants <$> askEnv- case iters of- Nothing -> do- iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'- blockLoop iterations $ \i -> do+ iterations <- dPrimVE "iterations" $ product $ map sExt32 dims'+ let onIteration i = do dIndexSpace (zip ltids dims') $ sExt64 i m- Just num_chunks -> localOps threadOperations $ do- let ltid = kernelLocalThreadId constants- sFor "chunk_i" num_chunks $ \chunk_i -> do- i <- dPrimVE "i" $ chunk_i * sExt32 tblock_size + ltid- dIndexSpace (zip ltids dims') $ sExt64 i- sWhen (inBounds (Slice (map (DimFix . le64) ltids)) dims') m+ full_chunks <- M.lookup dims . kernelFullChunksMap . kernelConstants <$> askEnv+ case full_chunks of+ Nothing -> blockLoop iterations onIteration+ Just num_full_chunks ->+ peeledKernelLoop+ (kernelLocalThreadId constants)+ (sExt32 tblock_size)+ num_full_chunks+ iterations+ onIteration SegNoVirt -> localOps threadOperations $ do zipWithM_ dPrimV_ ltids =<< localThreadIDs dims sWhen (isActive $ zip ltids dims) m@@ -288,7 +330,8 @@ -- The static arrays stuff does not work inside kernels. compileBlockExp (Pat [dest]) (BasicOp (ArrayVal vs t)) = compileBlockExp (Pat [dest]) (BasicOp (ArrayLit (map Constant vs) (Prim t)))-compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) =+compileBlockExp (Pat [dest]) (BasicOp (ArrayLit es _)) = do+ beforeBlockWrite forM_ (zip [0 ..] es) $ \(i, e) -> copyDWIMFix (patElemName dest) [fromIntegral (i :: Int64)] e [] compileBlockExp _ (BasicOp (UpdateAcc safety acc is vs)) = do@@ -296,6 +339,7 @@ sWhen (ltid .==. 0) $ updateAcc safety acc is vs sOp $ Imp.Barrier Imp.FenceLocal compileBlockExp (Pat [dest]) (BasicOp (Replicate ds se)) | ds /= mempty = do+ beforeBlockWrite flat <- newVName "rep_flat" is <- replicateM (arrayRank dest_t) (newVName "rep_i") let is' = map le64 is@@ -305,6 +349,7 @@ where dest_t = patElemType dest compileBlockExp (Pat [dest]) (BasicOp (Iota n e s it)) = do+ beforeBlockWrite n' <- toExp n e' <- toExp e s' <- toExp s@@ -339,7 +384,7 @@ -- Avoid that by always doing an error sync here. Potential -- improvement: only do this if any errors are pending (this could -- also be handled in later codegen).- when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal+ when (doSync e) $ sOp $ Imp.ErrorSync Imp.FenceLocal mempty defCompileExp dest e where doSync Loop {} = True@@ -362,192 +407,401 @@ blockAlloc dest _ _ = error $ "Invalid target for in-kernel allocation: " ++ show dest -compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp-compileBlockOp pat (Alloc size space) =- blockAlloc pat size space-compileBlockOp pat (Inner (SegOp (SegMap lvl space _ body))) = do- compileFlatId space+-- | How many elements an operand array for 'blockScan' must be backed by, given+-- the number of elements actually scanned and the return types of the operator.+-- Array-typed elements are kept twice over; see 'blockScan'.+blockScanElems :: Imp.TExp Int64 -> [Type] -> Imp.TExp Int64+blockScanElems n ts+ | all primType ts = n+ | otherwise = 2 * n - blockCoverSegSpace (segVirt lvl) space $- compileStms mempty (bodyStms body) $- zipWithM_ (compileThreadResult space) (patElems pat) $- bodyResult body- sOp $ Imp.ErrorSync Imp.FenceLocal-compileBlockOp pat (Inner (SegOp (SegScan lvl space _ body scans post_op))) = do- compileFlatId space+-- | A shared memory array of @elems@ values of type @elem_t@, staging the+-- operands of a 'blockScan' or 'blockReduce'. Note that 'blockScan' needs+-- memory past the index space it scans, and that excess is deliberately left+-- outside the shape (but is given by @elems@), as nothing but 'blockScan' may+-- touch it.+sharedStagingArray ::+ Name -> Type -> Imp.TExp Int64 -> Shape -> InKernelGen VName+sharedStagingArray desc elem_t elems shape = do+ mem <-+ sAlloc (desc <> "_mem") (Imp.bytes $ elems * unCount (typeSize elem_t)) $+ Space "shared"+ sArrayInMem desc (elemType elem_t) shape mem - let (ltids, dims) = unzip $ unSegSpace space- dims' = map pe64 dims- [scan] = scans- shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)- scan_ts = concatMap shpT scans- shpOfT t s =- arrayShape $- foldr (flip arrayOfRow) (arrayOfShape t s) $- segSpaceDims space- (scan_pars, map_pars) =- splitAt (length $ segBinOpNeutral scan) $ lambdaParams $ segPostOpLambda post_op- (body_scan_res, body_map_res) =- splitAt (length $ segBinOpNeutral scan) $ bodyResult body+-- | Will a single thread be responsible for more than one element of this index+-- space? If so, a value computed for one element in one phase of a multi-phase+-- operation cannot be kept in a scalar variable until a later phase, as the+-- thread will by then have overwritten it with the value belonging to some+-- other element.+severalElemsPerThread :: SegVirt -> Bool+severalElemsPerThread SegVirt = True+severalElemsPerThread SegNoVirt = False+severalElemsPerThread (SegNoVirtFull (SegSeqDims seq_dims)) = not $ null seq_dims - scan_out <- forM scan_ts $ \(s, t) ->- sAllocArray "scan_out" (elemType t) (shpOfT t s) $ Space "shared"+-- | Compile a block-level 'SegScan'. This proceeds in three phases, separated+-- by barriers:+--+-- 1. The kernel body is run to produce the scan inputs as distinct arrays.+--+-- 2. The scan itself is performed.+--+-- 3. Finally the post operator is applied to the scan results.+--+-- Everything that has to survive a barrier lives in shared memory, and because+-- the index space may be virtualised (such that a thread handles more than one+-- element), that includes the map results of the kernel body.+compileBlockScan ::+ Pat LetDecMem ->+ SegLevel ->+ SegSpace ->+ KernelBody GPUMem ->+ [SegBinOp GPUMem] ->+ SegPostOp GPUMem ->+ InKernelGen ()+compileBlockScan pat lvl space body scans post_op = do+ compileFlatId space - dScope Nothing $ scopeOfLParams $ lambdaParams $ segPostOpLambda post_op+ -- Whether a thread ends up handling several elements of the index space+ -- decides both where the map results have to live, and how the post operator+ -- gets at the indexes again.+ virt <- blockCoverVirt (segVirt lvl) space - blockCoverSegSpace (segVirt lvl) space $- compileStms mempty (bodyStms body) $ do- forM_ (zip scan_out body_scan_res) $ \(dest, res) ->- copyDWIMFix- dest- (map Imp.le64 ltids)- (kernelResultSubExp res)- []+ scan_arrs <- mapM scanArray scan_ts+ map_dests <- mapM (mapResDest virt) map_ts - sComment "bind map results to post lambda params" $- forM_ (zip map_pars body_map_res) $ \(par, res) ->- copyDWIMFix- (paramName par)- []- (kernelResultSubExp res)- []+ dScope Nothing $ scopeOfLParams $ lambdaParams post_lam - sOp $ Imp.ErrorSync Imp.FenceLocal+ computeScanInputs virt scan_arrs map_dests+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty+ doScan virt scan_arrs+ applyPostOp virt scan_arrs map_dests - let seg_flag = case dims' of- [_] -> Nothing -- Nonsegmented.- _ -> Just $ crossesSegment $ last dims'+ sOp $ Imp.Barrier Imp.FenceGlobal+ where+ (ltids, dims) = unzip $ unSegSpace space+ dims' = map pe64 dims+ [scan] = scans+ post_lam = segPostOpLambda post_op+ num_scan_res = length $ segBinOpNeutral scan+ (scan_pars, map_pars) = splitAt num_scan_res $ lambdaParams post_lam+ (body_scan_res, body_map_res) = splitAt num_scan_res $ bodyResult body+ scan_ts = concatMap shpT scans+ shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op)+ map_ts = map ((mempty,) . paramType) map_pars - -- blockScan needs to treat the scan output as a one-dimensional- -- array of scan elements, so we invent some new flattened arrays- -- here.- dims_flat <- dPrimV "dims_flat" $ product dims'- arrs_flat <-- mapM (flattenArray (length dims') dims_flat) scan_out+ shapeFor s t =+ arrayShape $ foldr (flip arrayOfRow) (arrayOfShape t s) dims - case segVirt lvl of- SegVirt ->- virtualisedBlockScan- seg_flag- (sExt32 $ tvExp dims_flat)- (segBinOpLambda scan)- arrs_flat- _ ->- blockScan- seg_flag- (product dims')+ -- One value per element of the index space, plus whatever 'blockScan'+ -- needs past that.+ scanArray (s, t) =+ sharedStagingArray+ "scan_out"+ (t `arrayOfShape` s)+ (blockScanElems (product dims') (lambdaReturnType (segBinOpLambda scan)))+ (shapeFor s t)++ -- Map results are not operands of the scan, so one value per element of+ -- the index space is all they need.+ mapArray (s, t) =+ sharedStagingArray+ "map_out"+ (t `arrayOfShape` s) (product dims')- (segBinOpLambda scan)- arrs_flat+ (shapeFor s t) - -- FIXME: we actually need something like blockCoverSegSpacee here, although in- -- practice we currently do not generate virtualised scans.- sWhen (isActive $ zip ltids dims) . localOps threadOperations $ do- sComment "bind scan results to post lambda params" $- forM_ (zip scan_pars scan_out) $ \(par, acc) ->- copyDWIMFix (paramName par) [] (Var acc) (map Imp.le64 ltids)+ -- Where does a map result produced by the kernel body live until the post+ -- operator needs it? A shared memory array when a thread handles several+ -- elements, as a scalar would by then have been overwritten; otherwise the+ -- post operator parameter itself. Accumulators are phantoms in ImpGen, so+ -- they need no storage at all.+ mapResDest virt (s, t)+ | isAcc t = pure Nothing+ | severalElemsPerThread virt = Just <$> mapArray (s, t)+ | otherwise = pure Nothing - let res = fmap resSubExp $ bodyResult $ lambdaBody $ segPostOpLambda post_op- sComment "compute post op." $- compileStms mempty (bodyStms $ lambdaBody $ segPostOpLambda post_op) $ do- sComment "write values" $- forM_ (zip (patElems pat) res) $ \(pe, subexp) ->- copyDWIMFix (patElemName pe) (map le64 ltids) subexp []+ seg_flag = case dims' of+ [_] -> Nothing -- Nonsegmented.+ _ -> Just $ crossesSegment $ last dims' - sOp $ Imp.Barrier Imp.FenceGlobal-compileBlockOp pat (Inner (SegOp (SegRed lvl space _ body ops))) = do- compileFlatId space+ -- Run the kernel body for every element of the index space, stashing the+ -- scan inputs - and the map results, if they cannot be kept in scalars -+ -- in shared memory.+ computeScanInputs virt scan_arrs map_dests =+ blockCoverSegSpace virt space $+ compileStms mempty (bodyStms body) $ do+ let ltids' = map Imp.le64 ltids+ forM_ (zip scan_arrs body_scan_res) $ \(arr, res) ->+ copyDWIMFix arr ltids' (kernelResultSubExp res) [] - let dims' = map pe64 dims- mkTempArr t =- sAllocArray "red_arr" (elemType t) (Shape dims <> arrayShape t) $ Space "shared"+ sComment "stash map results for the post operator" $+ forM_ (zip3 map_pars map_dests body_map_res) $ \(par, dest, res) ->+ case dest of+ Just arr -> copyDWIMFix arr ltids' (kernelResultSubExp res) []+ Nothing -> copyDWIMFix (paramName par) [] (kernelResultSubExp res) [] - tmp_arrs <- mapM mkTempArr $ concatMap (lambdaReturnType . segBinOpLambda) ops- blockCoverSegSpace (segVirt lvl) space $- compileStms mempty (bodyStms body) $ do- let (red_res, map_res) =- splitAt (segBinOpResults ops) $ bodyResult body- forM_ (zip tmp_arrs red_res) $ \(dest, res) ->- copyDWIMFix dest (map Imp.le64 ltids) (kernelResultSubExp res) []- zipWithM_ (compileThreadResult space) map_pes map_res+ -- blockScan needs to treat the scan input as a one-dimensional array of+ -- scan elements, so we invent some new flattened arrays here.+ doScan virt scan_arrs = do+ dims_flat <- dPrimV "dims_flat" $ product dims'+ arrs_flat <- mapM (flattenArray (length dims') dims_flat) scan_arrs+ case virt of+ SegVirt ->+ virtualisedBlockScan+ seg_flag+ (sExt32 $ tvExp dims_flat)+ (segBinOpLambda scan)+ arrs_flat+ _ ->+ blockScan+ seg_flag+ (product dims')+ (product dims')+ (segBinOpLambda scan)+ arrs_flat - sOp $ Imp.ErrorSync Imp.FenceLocal+ applyPostOp virt scan_arrs map_dests+ | severalElemsPerThread virt =+ blockCoverSegSpace virt space postOp+ | otherwise =+ -- In this case the ltids from computeScanInputs are still in scope.+ sWhen (isActive $ zip ltids dims) $ localOps threadOperations postOp+ where+ ltids' = map Imp.le64 ltids+ res = fmap resSubExp $ bodyResult $ lambdaBody post_lam+ postOp = do+ sComment "bind scan results to post lambda params" $+ forM_ (zip scan_pars scan_arrs) $ \(par, arr) ->+ copyDWIMFix (paramName par) [] (Var arr) ltids' - let tmps_for_ops = chunks (map (length . segBinOpNeutral) ops) tmp_arrs- case segVirt lvl of- SegVirt -> virtCase dims' tmps_for_ops- _ -> nonvirtCase dims' tmps_for_ops+ sComment "bind map results to post lambda params" $+ forM_ (zip map_pars map_dests) $ \(par, dest) ->+ forM_ dest $ \arr ->+ copyDWIMFix (paramName par) [] (Var arr) ltids'++ sComment "compute post op." $+ compileStms mempty (bodyStms $ lambdaBody post_lam) $+ sComment "write values" $+ forM_ (zip (patElems pat) res) $ \(pe, subexp) ->+ copyDWIMFix (patElemName pe) ltids' subexp []++isEmptySpace :: SegSpace -> Imp.TExp Bool+isEmptySpace = (.==. 0) . product . map pe64 . segSpaceDims++-- | Compile a block-level 'SegRed'. Unless the index space is virtualised, the+-- kernel body is first run for the entire index space, with the results stashed+-- in shared memory, which is then reduced. A virtualised index space is instead+-- handled one block-sized chunk at a time - chunks of the /flattened/ space,+-- when there are several segments - so that we need no more shared memory than+-- there are threads in the block.+compileBlockRed ::+ Pat LetDecMem ->+ SegLevel ->+ SegSpace ->+ KernelBody GPUMem ->+ [SegBinOp GPUMem] ->+ InKernelGen ()+compileBlockRed pat lvl space body ops = do+ compileFlatId space++ sIf (isEmptySpace space) emptyCase $ do+ virt <- blockCoverVirt (segVirt lvl) space+ case (virt, dims') of+ (SegVirt, [dim']) -> chunkedCase dim'+ (SegVirt, _) -> segmentedChunkedCase+ (_, [dim']) -> stageWholeSpace (product dims') $ singleSegmentCase dim'+ _ -> stageWholeSpace (blockScanElems (product dims') red_ts) segmentedCase where (ltids, dims) = unzip $ unSegSpace space+ dims' = map pe64 dims (red_pes, map_pes) = splitAt (segBinOpResults ops) $ patElems pat+ (body_red_res, body_map_res) =+ splitAt (segBinOpResults ops) $ bodyResult body+ red_ts = concatMap (lambdaReturnType . segBinOpLambda) ops+ forOps :: [a] -> [[a]]+ forOps = chunks $ map (length . segBinOpNeutral) ops - virtCase [dim'] tmps_for_ops = do+ -- Shared memory for @elems@ values per reduction operand, declared with+ -- the given shape.+ sharedArrays elems shape =+ forM red_ts $ \t ->+ sharedStagingArray "red_arr" t elems (shape <> arrayShape t)++ -- Write the reduction results, read at @src@ in the staging arrays, to+ -- @dest@ in the destinations.+ saveResults dest tmps_for_ops src =+ sComment "Save result of reduction." $+ forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->+ copyDWIM (patElemName pe) dest (Var arr) src++ -- The result of the chunks so far accumulates at @dest_is@ in the+ -- destination, so the carry is incorporated by folding the destination into+ -- the chunk's first element; the reduction or scan that follows propagates+ -- it through the rest of the chunk.+ incorporateCarry due dest_is tmps_for_ops = do ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+ sComment "possibly incorporate carry" . localOps threadOperations $+ sWhen (due .&&. ltid .==. 0) $+ forM_ (zip3 ops tmps_for_ops (forOps red_pes)) $ \(op, tmps, pes) ->+ applyRenamedLambda+ (segBinOpLambda op)+ (map (,[DimFix 0]) tmps)+ ( map ((,dest_is) . Var . patElemName) pes+ ++ map ((,[DimFix 0]) . Var) tmps+ )++ -- Run the segop body for the point of the index space that the thread's+ -- index variables are currently bound to, saving the reduction inputs at+ -- @is@ in the given @tmp_arrs@. Map results go straight to their+ -- destinations.+ runBody tmp_arrs is =+ compileStms mempty (bodyStms body) $ do+ forM_ (zip tmp_arrs body_red_res) $ \(dest, res) ->+ copyDWIMFix dest is (kernelResultSubExp res) []+ zipWithM_ (compileThreadResult space) map_pes body_map_res++ emptyCase = do+ -- This is the result if there are no chunks at all.+ sComment "set result to neutral element" $+ blockCoverSpace (map pe64 $ init dims) $ \is ->+ forM_ (zip red_pes $ concatMap segBinOpNeutral ops) $ \(pe, ne) ->+ copyDWIMFix (patElemName pe) is ne []++ -- Run the kernel body for the entire index space, staging the results in+ -- shared memory, then reduce them with the passed-in continuation.+ stageWholeSpace elems k = do+ tmp_arrs <- sharedArrays elems $ Shape dims+ blockCoverSegSpace (segVirt lvl) space $+ runBody tmp_arrs (map Imp.le64 ltids)+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty+ k $ forOps tmp_arrs++ -- A single segment, virtualised. We run the kernel body for one block-sized+ -- chunk at a time, reduce that chunk, and carry the result into the next+ -- chunk.+ chunkedCase dim' = do+ tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv+ ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+ -- A chunk is never larger than the block, nor than the index space+ -- itself, so that is all the shared memory we need.+ tmp_arrs <-+ sharedArrays (sMin64 dim' (pe64 (unCount tblock_size))) $+ Shape [unCount tblock_size]+ let tmps_for_ops = forOps tmp_arrs+ [ltid_name] = ltids+ blockChunkLoop (sExt32 dim') $ \chunk_start chunk_size -> do- sComment "possibly incorporate carry" $- sWhen (chunk_start .>. 0 .&&. ltid .==. 0) $- forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->- applyRenamedLambda- (segBinOpLambda op)- (map (,[DimFix $ sExt64 chunk_start]) tmps)- ( map ((,[]) . Var . patElemName) red_pes- ++ map ((,[DimFix $ sExt64 chunk_start]) . Var) tmps- )+ sComment "run the kernel body for this chunk" . localOps threadOperations $ do+ dPrimV_ ltid_name $ sExt64 chunk_start + sExt64 ltid+ sWhen (Imp.le64 ltid_name .<. dim') $+ runBody tmp_arrs [sExt64 ltid] - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do- tmps_chunks <- mapM (sliceArray (sExt64 chunk_start) chunk_size) tmps- blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps_chunks+ incorporateCarry (chunk_start .>. 0) [] tmps_for_ops - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - sComment "Save result of reduction." $- forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->- copyDWIMFix (patElemName pe) [] (Var arr) [sExt64 chunk_start]+ forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->+ blockReduce (sExt32 (tvExp chunk_size)) (segBinOpLambda op) tmps + sOp $ Imp.ErrorSync Imp.FenceLocal mempty++ saveResults [] tmps_for_ops [DimFix 0]++ -- Everyone must be done reading the result before the next chunk+ -- overwrites it.+ sOp $ Imp.Barrier Imp.FenceLocal++ -- Multiple segments, virtualised. We go through the flattened index space+ -- one block-sized chunk at a time: run the segop body for the chunk, scan+ -- the chunk with the segment flags, then let every thread whose element is+ -- the last of its segment write that segment's result. The running value of+ -- a segment that continues past the end of the chunk is carried into the+ -- next chunk. --- virtCase dims' tmps_for_ops = do- dims_flat <- dPrimV "dims_flat" $ product dims'+ -- Chunking the flattened space rather than taking whole segments at a time+ -- keeps every thread busy no matter how the segment size relates to the+ -- block size, and needs no separate strategy for segments larger than the+ -- block.+ segmentedChunkedCase = do+ tblock_size <- kernelBlockSizeCount . kernelConstants <$> askEnv+ ltid <- kernelLocalThreadId . kernelConstants <$> askEnv+ let w = product dims'+ -- A chunk is never larger than the block, nor than the index space+ -- itself, so that is all the shared memory we need.+ chunk_elems = sMin64 w $ pe64 $ unCount tblock_size+ seg_flag = crossesSegment $ last dims'+ tmp_arrs <-+ sharedArrays (blockScanElems chunk_elems red_ts) $+ Shape [unCount tblock_size]+ let tmps_for_ops = forOps tmp_arrs - forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do- tmps_flat <- mapM (flattenArray (length dims') dims_flat) tmps- virtualisedBlockScan- (Just $ crossesSegment $ last dims')- (sExt32 $ tvExp dims_flat)- (segBinOpLambda op)- tmps_flat+ blockChunkLoop (sExt32 w) $ \chunk_start chunk_size -> do+ flat_idx <- dPrimVE "flat_idx" $ sExt64 chunk_start + sExt64 ltid+ in_bounds <- dPrimVE "in_bounds" $ flat_idx .<. w+ dIndexSpace (zip ltids dims') flat_idx - sOp $ Imp.ErrorSync Imp.FenceLocal+ sComment "run segred body for this chunk" . localOps threadOperations $+ sWhen in_bounds $+ runBody tmp_arrs [sExt64 ltid] - sComment "Save result of reduction." $- forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->- copyDWIM- (patElemName pe)- []- (Var arr)- (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - sOp $ Imp.Barrier Imp.FenceLocal+ let (crosses_segment, shifted) =+ chunkSegFlags (Just seg_flag) chunk_start - -- Nonsegmented case (or rather, a single segment) - this we can- -- handle directly with a block-level reduction.- nonvirtCase [dim'] tmps_for_ops = do+ -- Thread zero is at @chunk_start@, so its segment index is that of the+ -- segment the carry belongs to - the one the preceding chunk ended in.+ incorporateCarry+ (chunk_start .>. 0 .&&. bNot crosses_segment)+ (map (DimFix . Imp.le64) (init ltids))+ tmps_for_ops++ sOp $ Imp.ErrorSync Imp.FenceLocal mempty++ -- Scan the currente chunk.+ forM_ (zip ops tmps_for_ops) $ \(op, tmps) ->+ blockScan shifted chunk_elems (tvExp chunk_size) (segBinOpLambda op) tmps++ sOp $ Imp.ErrorSync Imp.FenceLocal mempty++ -- A thread whose element is the last of its segment holds that+ -- segment's final result. The thread holding the last element of the+ -- chunk holds the running result of a segment that continues into the+ -- next chunk, which that chunk then reads back as its carry.+ ends_segment <-+ dPrimVE "ends_segment" $ Imp.le64 (last ltids) .==. last dims' - 1+ ends_chunk <-+ dPrimVE "ends_chunk" $ sExt64 ltid .==. tvExp chunk_size - 1+ localOps threadOperations $+ sWhen (in_bounds .&&. (ends_segment .||. ends_chunk)) $+ saveResults+ (map (DimFix . Imp.le64) (init ltids))+ tmps_for_ops+ [DimFix $ sExt64 ltid]++ -- Everyone must be done reading the chunk before the next one+ -- overwrites it, and the carry must be visible to the next chunk.+ fence <- fenceForArrays $ map patElemName red_pes+ sOp $ Imp.Barrier fence++ -- A single segment - this we can handle directly with a block-level+ -- reduction.+ singleSegmentCase dim' tmps_for_ops = do forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> blockReduce (sExt32 dim') (segBinOpLambda op) tmps- sOp $ Imp.ErrorSync Imp.FenceLocal- sComment "Save result of reduction." $- forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->- copyDWIMFix (patElemName pe) [] (Var arr) [0]- sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty+ saveResults [] tmps_for_ops [DimFix 0]+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - -- Segmented intra-block reductions are turned into (regular)- -- segmented scans. It is possible that this can be done+ -- Multiple segments, with the whole index space staged in shared memory.+ -- These are turned into (regular) segmented scans, of which we then pick+ -- the last element of each segment. It is possible that this can be done -- better, but at least this approach is simple.- nonvirtCase dims' tmps_for_ops = do- -- blockScan operates on flattened arrays. This does not- -- involve copying anything; merely playing with the index- -- function.+ segmentedCase tmps_for_ops = do+ -- The scans operate on flattened arrays. This does not involve copying+ -- anything; merely playing with the index function. dims_flat <- dPrimV "dims_flat" $ product dims' forM_ (zip ops tmps_for_ops) $ \(op, tmps) -> do@@ -559,19 +813,39 @@ (segBinOpLambda op) tmps_flat - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty - sComment "Save result of reduction." $- forM_ (zip red_pes $ concat tmps_for_ops) $ \(pe, arr) ->- copyDWIM- (patElemName pe)- []- (Var arr)- (map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1])+ saveResults [] tmps_for_ops lastInSegment sOp $ Imp.Barrier Imp.FenceLocal-compileBlockOp pat (Inner (SegOp (SegHist lvl space _ kbody ops))) = do++ lastInSegment =+ map (unitSlice 0) (init dims') ++ [DimFix $ last dims' - 1]++compileBlockOp :: OpCompiler GPUMem KernelEnv Imp.KernelOp+compileBlockOp pat (Alloc size space) =+ blockAlloc pat size space+compileBlockOp pat (Inner (SegOp op)) = do+ beforeBlockWrite+ compileBlockSegOp pat op+compileBlockOp pat _ =+ compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat++compileBlockSegOp :: Pat LetDecMem -> SegOp SegLevel GPUMem -> InKernelGen ()+compileBlockSegOp pat (SegMap lvl space _ body) = do compileFlatId space++ blockCoverSegSpace (segVirt lvl) space $+ compileStms mempty (bodyStms body) $+ zipWithM_ (compileThreadResult space) (patElems pat) $+ bodyResult body+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty+compileBlockSegOp pat (SegScan lvl space _ body scans post_op) =+ compileBlockScan pat lvl space body scans post_op+compileBlockSegOp pat (SegRed lvl space _ body ops) =+ compileBlockRed pat lvl space body ops+compileBlockSegOp pat (SegHist lvl space _ kbody ops) = do+ compileFlatId space let (ltids, dims) = unzip $ unSegSpace space -- We don't need the red_pes, because it is guaranteed by our type@@ -613,9 +887,7 @@ copyDWIMFix (paramName p) [] v is do_op (bin_is ++ is) - sOp $ Imp.ErrorSync Imp.FenceLocal-compileBlockOp pat _ =- compilerBugS $ "compileBlockOp: cannot compile rhs of binding " ++ prettyString pat+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- | The operations for block-level kernels. blockOperations :: Operations GPUMem KernelEnv Imp.KernelOp@@ -744,7 +1016,7 @@ -- | Various useful precomputed information for block-level SegOps. data Precomputed = Precomputed { pcSegOpSizes :: SegOpSizes,- pcChunkItersMap :: M.Map [SubExp] (Imp.TExp Int32)+ pcFullChunksMap :: M.Map [SubExp] (Imp.TExp Int32) } -- | Find the sizes of nested parallelism in a t'SegOp' body.@@ -773,13 +1045,16 @@ precomputeConstants :: Count BlockSize (Imp.TExp Int64) -> Stms GPUMem -> CallKernelGen Precomputed precomputeConstants tblock_size stms = do let sizes = segOpSizes stms- iters_map <- M.fromList <$> mapM mkMap (S.toList sizes)- pure $ Precomputed sizes iters_map+ -- Constant sizes are left to the kernel; see Note [Peeled block loops].+ full_chunks_map <-+ M.fromList <$> mapM mkMap (filter (any (isJust . subExpVar)) (S.toList sizes))+ pure $ Precomputed sizes full_chunks_map where mkMap dims = do let n = product $ map Imp.pe64 dims- num_chunks <- dPrimVE "num_chunks" $ sExt32 $ n `ceilDiv` unCount tblock_size- pure (dims, num_chunks)+ num_full_chunks <-+ dPrimVE "num_full_chunks" $ sExt32 $ n `quot` unCount tblock_size+ pure (dims, num_full_chunks) -- | Make use of various precomputed constants. precomputedConstants :: Precomputed -> InKernelGen a -> InKernelGen a@@ -791,7 +1066,7 @@ { kernelConstants = (kernelConstants env) { kernelLocalIdMap = new_ids,- kernelChunkItersMap = pcChunkItersMap pre+ kernelFullChunksMap = pcFullChunksMap pre } } localEnv f m
src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs view
@@ -196,21 +196,19 @@ [Imp.TExp Int64] -> InKernelGen () ) prepareAtomicUpdateGlobal l segments dests slug =- -- We need a separate lock array if the operators are not all of a- -- particularly simple form that permits pure atomic operations.+ -- We need locks if the operators are not all of a particularly+ -- simple form that permits pure atomic operations. case (l, slugAtomicUpdate slug) of (_, AtomicPrim f) -> pure (l, f (Space "global") dests) (_, AtomicCAS f) -> pure (l, f (Space "global") dests) (Just l', AtomicLocking f) -> pure (l, f l' (Space "global") dests) (Nothing, AtomicLocking f) -> do- -- The number of locks used here is too low, but since we are- -- currently forced to inline a huge list, I'm keeping it down- -- for now. Some quick experiments suggested that it has little- -- impact anyway (maybe the locking case is just too slow).- --- -- A fun solution would also be to use a simple hashing- -- algorithm to ensure good distribution of locks.- let num_locks = 100151+ -- The lock for a bucket is chosen by its flat index modulo the+ -- number of locks. More locks, or a simple hashing algorithm+ -- that distributes the buckets better, might reduce contention,+ -- but quick experiments suggested that it has little impact+ -- (maybe the locking case is just too slow).+ let num_locks = Imp.bookkeepingSize Imp.BookkeepingLocks dims = map pe64 $ shapeDims segments@@ -218,7 +216,7 @@ ++ [tvSize (slugNumSubhistos slug)] ++ shapeDims (histShape (slugOp slug)) - locks <- genZeroes "hist_locks" num_locks+ locks <- bookkeepingArray Imp.BookkeepingLocks let l' = Locking locks 0 1 0 (pure . (`rem` fromIntegral num_locks) . flattenIndex dims) pure (Just l', f l' (Space "global") dests) @@ -781,7 +779,7 @@ copyDWIMFix (paramName p) [] v is do_op (bucket_is ++ is) - sOp $ Imp.ErrorSync Imp.FenceGlobal+ sOp $ Imp.ErrorSync Imp.FenceGlobal mempty sComment "Compact the multiple shared memory subhistograms to result in global memory" $ onSlugs $ \slug dests hist_H_chk histo_dims _histo_size bins_per_thread -> do
src/Futhark/CodeGen/ImpGen/GPU/SegRed.hs view
@@ -9,13 +9,8 @@ -- kernel, with the final threadblock to write a result (tracked via -- an atomic counter) performing the final reduction as well. ----- * Instead of depending on storage layout transformations to handle--- non-commutative reductions efficiently, we slide a--- @tblocksize@-sized window over the input, and perform a parallel--- reduction for each window. This sacrifices the notion of--- efficient sequentialisation, but is sometimes faster and--- definitely simpler and more predictable (and uses less auxiliary--- storage).+-- * We have various strategies for handling non-commutative reductions. See+-- Note [Stage one strategies]. -- -- For segmented reductions we use the approach from "Strategies for -- Regular Segmented Reductions on GPU" (FHPC '17). This involves@@ -30,10 +25,8 @@ -- value per segment. -- -- Each thread /can/ read multiple elements, which will greatly--- increase performance; however, if the reduction is--- non-commutative we will have to use a less efficient traversal--- (with interim block-wide reductions) to enable coalesced memory--- accesses, just as in the non-segmented case.+-- increase performance. Non-commutative reductions use the same+-- strategies as in the non-segmented case. -- -- * Small: is used to let each block process *multiple* segments -- within a block. We will only use this approach when we can@@ -41,13 +34,6 @@ -- cases, we would allocate a /whole/ block per segment with the -- large strategy, but at most 50% of the threads in the block would -- have any element to read, which becomes highly inefficient.------ An optimization specfically targeted at non-segmented and large-segments--- segmented reductions with non-commutative is made: The stage one main loop is--- essentially stripmined by a factor *chunk*, inserting collective copies via--- shared memory of each reduction parameter going into the intra-block (partial)--- reductions. This saves a factor *chunk* number of intra-block reductions at--- the cost of some overhead in collective copies. module Futhark.CodeGen.ImpGen.GPU.SegRed ( compileSegRed, compileSegRed',@@ -56,7 +42,6 @@ where import Control.Monad-import Data.List (zip4) import Data.Map qualified as M import Data.Maybe import Futhark.CodeGen.ImpCode.GPU qualified as Imp@@ -72,6 +57,11 @@ forM2_ :: (Monad m) => [a] -> [b] -> (a -> b -> m c) -> m () forM2_ xs ys f = forM_ (zip xs ys) (uncurry f) +-- | The total size of values of the given types in 32-bit words, where+-- smaller types occupy a full word.+primWords :: [PrimType] -> Int+primWords = sum . map ((`div` 4) . max 4 . primByteSize)+ -- | Given available register and a list of parameter types, compute -- the largest available chunk size given the parameters for which we -- want chunking and the available resources.@@ -91,7 +81,7 @@ sizes = map primByteSize types' sum_sizes = sum sizes- sum_sizes' = sum (map (sMax64 4 . primByteSize) types') `quot` 4+ sum_sizes' = fromIntegral $ primWords types' max_size = maximum sizes mem_constraint = max k_mem sum_sizes `quot` max_size@@ -153,6 +143,45 @@ all primType (lambdaReturnType $ segBinOpLambda segbinop) && shapeRank (segBinOpShape segbinop) == 0 +-- | How the first stage of a reduction distributes the input among the threads.+-- For elaboration on the strategies, see Note [Stage one strategies].+data StageOne+ = -- | Thread @t@ of @T@ handles elements @t@, @t+T@, @t+2T@, and so on, not+ -- combining the per-thread results until all input has been processed.+ -- Requires a commutative operator.+ StageOneStrided+ | -- | Each block handles a sequence of contiguous tiles, using collective+ -- copies through shared memory and a block-wide reduction per tile.+ -- Requires primitive (non-array) operands.+ StageOneTiled+ | -- | Each thread handles a contiguous range of the input.+ StageOneContiguous+ | -- | Each block slides a window of @tblock_size@ elements across a+ -- contiguous range of the input, with a block-wide reduction per window.+ StageOneWindow+ deriving (Eq)++-- | The largest total size of the reduction operands, in 32-bit words, for+-- which we use 'StageOneTiled'. Heuristically chosen. It is difficult to make+-- this a tuning parameter, as the strategy is a compile-time choice.+maxTiledOperandWords :: Int+maxTiledOperandWords = 16++stageOneStrategy :: [SegBinOp GPUMem] -> StageOne+stageOneStrategy segbinops+ | all ((== Commutative) . segBinOpComm) segbinops =+ StageOneStrided+ | not $ all isPrimSegBinOp segbinops =+ StageOneWindow+ | operand_words <= maxTiledOperandWords =+ StageOneTiled+ | otherwise =+ StageOneContiguous+ where+ operand_words =+ primWords . map elemType $+ concatMap (lambdaReturnType . segBinOpLambda) segbinops+ -- | Like 'compileSegRed', but where the body is a monadic action. compileSegRed' :: Pat LetDecMem ->@@ -175,19 +204,19 @@ (compileReduction (chunk_v, chunk_const) largeSegmentsReduction) where compileReduction chunk f =- f pat num_tblocks tblock_size chunk space segbinops map_body_cont+ f pat num_tblocks tblock_size strategy chunk space segbinops map_body_cont param_types = map paramType $ concatMap paramOf segbinops num_tblocks = gridNumBlocks grid tblock_size = gridBlockSize grid - chunk_const =- if Noncommutative `elem` map segBinOpComm segbinops- && all isPrimSegBinOp segbinops- then getRedChunkSize param_types- else Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)+ strategy = stageOneStrategy segbinops + chunk_const+ | strategy == StageOneTiled = getRedChunkSize param_types+ | otherwise = Imp.ValueExp $ IntValue $ intValue Int64 (1 :: Int64)+ -- | Prepare intermediate arrays for the reduction. Prim-typed -- arguments go in shared memory (so we need to do the allocation of -- those arrays inside the kernel), while array-typed arguments go in@@ -198,14 +227,14 @@ -- For more info about the intermediate arrays used for the different reduction -- kernels, see note [IntermArrays]. makeIntermArrays ::+ StageOne -> Imp.TExp Int64 -> SubExp -> SubExp -> [SegBinOp GPUMem] -> InKernelGen [SegRedIntermediateArrays]-makeIntermArrays tblock_id tblock_size chunk segbinops- | Noncommutative <- mconcat (map segBinOpComm segbinops),- all isPrimSegBinOp segbinops =+makeIntermArrays strategy tblock_id tblock_size chunk segbinops+ | strategy == StageOneTiled = noncommPrimSegRedInterms | otherwise = generalSegRedInterms tblock_id tblock_size segbinops@@ -282,10 +311,10 @@ -- | Arrays for storing block results. -- -- The block-result arrays have an extra dimension because they are--- also used for keeping vectorised accumulators for first-stage--- reduction, if necessary. If necessary, this dimension has size--- tblock_size, and otherwise 1. When actually storing block results,--- the first index is set to 0.+-- also used for keeping the per-thread accumulators for first-stage+-- reduction of non-primitive operands. In that case, this dimension+-- has size tblock_size, and otherwise 1. When actually storing block+-- results, the first index is set to 0. groupResultArrays :: SubExp -> SubExp ->@@ -308,6 +337,7 @@ Pat LetDecMem -> Count NumBlocks SubExp -> Count BlockSize SubExp ->+ StageOne -> (TV Int64, Imp.KernelConstExp) -> SegSpace -> [SegBinOp GPUMem] ->@@ -315,7 +345,7 @@ CallKernelGen () nonsegmentedReduction :: DoCompileSegRed-nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do+nonsegmentedReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do let (gtids, dims) = unzip $ unSegSpace space chunk = tvExp chunk_v num_tblocks_se = unCount num_tblocks@@ -324,7 +354,7 @@ global_tid = Imp.le64 $ segFlat space n = pe64 $ last dims - counters <- genZeroes "counters" $ length segbinops+ counters <- bookkeepingArray Imp.BookkeepingCounters reds_block_res_arrs <- groupResultArrays num_tblocks_se tblock_size_se segbinops @@ -341,7 +371,7 @@ let ltid = kernelLocalThreadId constants let tblock_id = kernelBlockId constants - interms <- makeIntermArrays (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops+ interms <- makeIntermArrays strategy (sExt64 tblock_id) tblock_size_se (tvSize chunk_v) segbinops sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared" -- Since this is the nonsegmented case, all outer segment IDs must@@ -355,6 +385,7 @@ zip3 segbinops interms reds_block_res_arrs new_lambdas <- reductionStageOne+ strategy gtids n global_tid@@ -369,22 +400,18 @@ (map (length . segBinOpNeutral) segbinops) segred_pes - forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $- \(pes, slug, new_lambda, i) ->- reductionStageTwo- pes- tblock_id- [0]- 0- (sExt64 $ kernelNumBlocks constants)- slug- new_lambda- counters- sync_arr- (fromInteger i)+ reductionStageTwo+ tblock_id+ [0]+ 0+ (sExt64 $ kernelNumBlocks constants)+ (zip3 segred_pess slugs new_lambdas)+ counters+ sync_arr+ 0 smallSegmentsReduction :: DoCompileSegRed-smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ space segbinops map_body_cont = do+smallSegmentsReduction (Pat segred_pes) num_tblocks tblock_size _ _ space segbinops map_body_cont = do let (gtids, dims) = unzip $ unSegSpace space dims' = map pe64 dims segment_size = last dims'@@ -453,7 +480,7 @@ in_bounds out_of_bounds - sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier. let crossesSegment from to = (sExt64 to - sExt64 from) .>. (sExt64 to `rem` segment_size) sWhen (segment_size .>. 0) $@@ -495,7 +522,7 @@ sOp $ Imp.Barrier Imp.FenceLocal largeSegmentsReduction :: DoCompileSegRed-largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size (chunk_v, chunk_const) space segbinops map_body_cont = do+largeSegmentsReduction (Pat segred_pes) num_tblocks tblock_size strategy (chunk_v, chunk_const) space segbinops map_body_cont = do let (gtids, dims) = unzip $ unSegSpace space dims' = map pe64 dims num_segments = product $ init dims'@@ -537,13 +564,13 @@ -- However, we exploit the fact that the number of segments being -- reduced at any point in time is limited by the number of -- threadblocks. If we bound the number of threadblocks, we can get away- -- with using that many counters. FIXME: Is this limit checked- -- anywhere? There are other places in the compiler that will fail- -- if the block count exceeds the maximum block size, which is at- -- most 1024 anyway.- let counters_per_op = 1024- counters <-- genZeroes "counters" $ fromIntegral $ length segbinops * counters_per_op+ -- with using that many counters, so we use the bookkeeping counters (see+ -- 'Imp.Bookkeeping'), indexed by the segment modulo their number. FIXME:+ -- Is this limit checked anywhere? There are other places in the compiler+ -- that will fail if the block count exceeds the maximum block size, which+ -- is at most 1024 anyway.+ counters <- bookkeepingArray Imp.BookkeepingCounters+ let num_counters = Imp.bookkeepingSize Imp.BookkeepingCounters let attrs = (defKernelAttrs num_tblocks tblock_size)@@ -555,7 +582,7 @@ let tblock_id = sExt64 $ kernelBlockId constants ltid = kernelLocalThreadId constants - interms <- makeIntermArrays tblock_id tblock_size_se (tvSize chunk_v) segbinops+ interms <- makeIntermArrays strategy tblock_id tblock_size_se (tvSize chunk_v) segbinops sync_arr <- sAllocArray "sync_arr" Bool (Shape [intConst Int32 1]) $ Space "shared" -- We probably do not have enough actual threadblocks to cover the@@ -583,6 +610,7 @@ zip3 segbinops interms reds_block_res_arrs new_lambdas <- reductionStageOne+ strategy gtids n global_tid@@ -598,23 +626,15 @@ segred_pes multiple_blocks_per_segment =- forM_ (zip4 segred_pess slugs new_lambdas [0 ..]) $- \(pes, slug, new_lambda, i) -> do- let counter_idx =- fromIntegral (i * counters_per_op)- + flat_segment_id- `rem` fromIntegral counters_per_op- reductionStageTwo- pes- virttblock_id- (map Imp.le64 segment_gtids)- first_block_for_segment- blocks_per_segment- slug- new_lambda- counters- sync_arr- counter_idx+ reductionStageTwo+ virttblock_id+ (map Imp.le64 segment_gtids)+ first_block_for_segment+ blocks_per_segment+ (zip3 segred_pess slugs new_lambdas)+ counters+ sync_arr+ (flat_segment_id `rem` fromIntegral num_counters) one_block_per_segment = sComment "first thread in block saves final result to memory" $@@ -633,7 +653,9 @@ { slugOp :: SegBinOp GPUMem, -- | Intermediate arrays needed for the given reduction. slugInterms :: SegRedIntermediateArrays,- -- | Place(s) to store block accumulator(s) in stage 1 reduction.+ -- | Per-thread accumulator(s) for the stage one reduction. After+ -- each block-wide reduction, the accumulator(s) of thread 0 hold+ -- the result for the block. slugAccs :: [(VName, [Imp.TExp Int64])], -- | Global memory destination(s) for the final result(s) for this -- particular reduction.@@ -652,10 +674,10 @@ mkAcc p block_res_arr | Prim t <- paramType p, shapeRank (segBinOpShape op) == 0 = do- block_res_acc <- dPrimS (baseName (paramName p) <> "_block_res_acc") t- pure (block_res_acc, [])+ thread_acc <- dPrimS (baseName (paramName p) <> "_thread_acc") t+ pure (thread_acc, []) -- if this is a non-primitive reduction, the global mem result array will- -- double as accumulator.+ -- double as the per-thread accumulators. | otherwise = pure (block_res_arr, [sExt64 ltid, sExt64 tblock_id]) @@ -674,9 +696,6 @@ slugShape :: SegBinOpSlug -> Shape slugShape = segBinOpShape . slugOp -slugsComm :: [SegBinOpSlug] -> Commutativity-slugsComm = mconcat . map (segBinOpComm . slugOp)- slugSplitParams :: SegBinOpSlug -> ([LParam GPUMem], [LParam GPUMem]) slugSplitParams slug = splitAt (length (slugNeutral slug)) $ slugParams slug @@ -689,7 +708,43 @@ slugCollCopyArrs :: SegBinOpSlug -> [VName] slugCollCopyArrs = collCopyArrs . slugInterms +-- | When 'sequentialStageOneBody' performs the block-wide reduction.+data BlockReduceAt = ReducePerIteration | ReduceAfterLoop+ deriving (Eq)++-- | Each thread sequentially reduces @q@ elements, with block-wide reductions+-- as determined by 'BlockReduceAt'. The function maps the iteration number to+-- the index of the element.+sequentialStageOneBody ::+ [SegBinOpSlug] ->+ DoSegBody ->+ TV Int64 ->+ Imp.TExp Int64 ->+ Imp.TExp Int64 ->+ BlockReduceAt ->+ InKernelGen () ->+ (Imp.TExp Int64 -> Imp.TExp Int64) ->+ InKernelGen ()+sequentialStageOneBody slugs body_cont glb_ind_var q n reduce_at doBlockReduce index = do+ sFor "i" q $ \i -> do+ glb_ind_var <-- index i++ sWhen (tvExp glb_ind_var .<. n) $+ sComment "apply map function(s)" $+ body_cont $ \all_red_res -> do+ let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res++ forM2_ slugs maps_res $ \slug map_res ->+ sLoopNest (slugShape slug) $ \vec_is ->+ applyRedOp slug vec_is map_res++ when (reduce_at == ReducePerIteration) doBlockReduce++ sOp $ Imp.ErrorSync Imp.FenceLocal mempty+ when (reduce_at == ReduceAfterLoop) doBlockReduce+ reductionStageOne ::+ StageOne -> [VName] -> Imp.TExp Int64 -> Imp.TExp Int64 ->@@ -699,39 +754,39 @@ [SegBinOpSlug] -> DoSegBody -> InKernelGen [Lambda GPUMem]-reductionStageOne gtids n global_tid q chunk threads_per_segment slugs body_cont = do+reductionStageOne strategy gtids n global_tid q chunk threads_per_segment slugs body_cont = do constants <- kernelConstants <$> askEnv let glb_ind_var = mkTV (last gtids) ltid = sExt64 $ kernelLocalThreadId constants dScope Nothing $ scopeOfLParams $ concatMap slugParams slugs - sComment "ne-initialise the outer (per-block) accumulator(s)" $ do+ sComment "ne-initialise the per-thread accumulator(s)" $ do forM_ slugs $ \slug -> forM2_ (slugAccs slug) (slugNeutral slug) $ \(acc, acc_is) ne -> sLoopNest (slugShape slug) $ \vec_is -> copyDWIMFix acc (acc_is ++ vec_is) ne [] new_lambdas <- mapM (renameLambda . slugLambda) slugs- let tblock_size = sExt32 $ kernelBlockSize constants let doBlockReduce = forM2_ slugs new_lambdas $ \slug new_lambda -> do let accs = slugAccs slug let params = slugParams slug+ let tblock_size = sExt32 $ kernelBlockSize constants sLoopNest (slugShape slug) $ \vec_is -> do let block_red_arrs = slugBlockRedArrs slug- sComment "store accs. prims go in lmem; non-prims in params (in global mem)" $+ sComment "store reduction accs" $ forM_ (zip3 block_red_arrs accs params) $ \(arr, (acc, acc_is), p) -> if isPrimParam p then copyDWIMFix arr [ltid] (Var acc) (acc_is ++ vec_is) else copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is) - sOp $ Imp.ErrorSync Imp.FenceLocal -- Also implicitly barrier.+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty -- Also implicitly barrier. blockReduce tblock_size new_lambda block_red_arrs sOp $ Imp.Barrier Imp.FenceLocal - sComment "thread 0 updates per-block acc(s); rest reset to ne" $ do+ sComment "thread 0 keeps the block result in its accumulator; rest reset to neutral" $ do sIf (ltid .==. 0) ( forM2_ accs (lambdaParams new_lambda) $@@ -743,8 +798,8 @@ copyDWIMFix acc (acc_is ++ vec_is) ne [] ) - case (slugsComm slugs, all (isPrimSegBinOp . slugOp) slugs) of- (Noncommutative, True) ->+ case strategy of+ StageOneTiled -> noncommPrimParamsStageOneBody slugs body_cont@@ -754,74 +809,43 @@ n chunk doBlockReduce- _ ->- generalStageOneBody- slugs- body_cont- glb_ind_var- global_tid- q- n- threads_per_segment- doBlockReduce+ StageOneStrided ->+ sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $+ \i -> global_tid + threads_per_segment * i+ StageOneContiguous ->+ sequentialStageOneBody slugs body_cont glb_ind_var q n ReduceAfterLoop doBlockReduce $+ \i -> global_tid * q + i+ StageOneWindow -> do+ let window_size = kernelBlockSize constants+ block_base_offset <-+ dPrimVE "block_base_offset" $ (global_tid `quot` window_size) * q * window_size+ sequentialStageOneBody slugs body_cont glb_ind_var q n ReducePerIteration doBlockReduce $+ \i -> block_base_offset + i * window_size + ltid pure new_lambdas -generalStageOneBody ::- [SegBinOpSlug] ->- DoSegBody ->- TV Int64 ->- Imp.TExp Int64 ->- Imp.TExp Int64 ->- Imp.TExp Int64 ->- Imp.TExp Int64 ->- InKernelGen () ->+-- | Combine the accumulators of a reduction with the given next values. The+-- vector indexes select the elements of a vectorised reduction, and are+-- appended to the indexes of both.+applyRedOp ::+ SegBinOpSlug ->+ [Imp.TExp Int64] ->+ [(SubExp, [Imp.TExp Int64])] -> InKernelGen ()-generalStageOneBody slugs body_cont glb_ind_var global_tid q n threads_per_segment doBlockReduce = do- let is_comm = slugsComm slugs == Commutative-- constants <- kernelConstants <$> askEnv- let tblock_size = kernelBlockSize constants- ltid = sExt64 $ kernelLocalThreadId constants-- -- this block's id within its designated segment, and this block's initial- -- global offset.- tblock_id_in_segment <- dPrimVE "tblock_id_in_segment" $ global_tid `quot` tblock_size- block_base_offset <- dPrimVE "block_base_offset" $ tblock_id_in_segment * q * tblock_size-- sFor "i" q $ \i -> do- block_offset <- dPrimVE "block_offset" $ block_base_offset + i * tblock_size- glb_ind_var- <-- if is_comm- then global_tid + threads_per_segment * i- else block_offset + ltid-- sWhen (tvExp glb_ind_var .<. n) $- sComment "apply map function(s)" $- body_cont $ \all_red_res -> do- let maps_res = chunks (map (length . slugNeutral) slugs) all_red_res-- forM2_ slugs maps_res $ \slug map_res ->- sLoopNest (slugShape slug) $ \vec_is -> do- let (acc_params, next_params) = slugSplitParams slug- sComment "load accumulator(s)" $- forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->- copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)- sComment "load next value(s)" $- forM2_ next_params map_res $ \p (res, res_is) ->- copyDWIMFix (paramName p) [] res (res_is ++ vec_is)- sComment "apply reduction operator(s)"- $ compileStms mempty (bodyStms $ slugBody slug)- $ sComment "store in accumulator(s)"- $ forM2_- (slugAccs slug)- (map resSubExp $ bodyResult $ slugBody slug)- $ \(acc, acc_is) se ->- copyDWIMFix acc (acc_is ++ vec_is) se []-- unless is_comm doBlockReduce- sOp $ Imp.ErrorSync Imp.FenceLocal- when is_comm doBlockReduce+applyRedOp slug vec_is next_values = do+ let (acc_params, next_params) = slugSplitParams slug+ sComment "load accumulator(s)" $+ forM2_ acc_params (slugAccs slug) $ \p (acc, acc_is) ->+ copyDWIMFix (paramName p) [] (Var acc) (acc_is ++ vec_is)+ sComment "load next value(s)" $+ forM2_ next_params next_values $ \p (v, v_is) ->+ copyDWIMFix (paramName p) [] v (v_is ++ vec_is)+ sComment "apply reduction operator(s)" $+ compileStms mempty (bodyStms $ slugBody slug) $+ sComment "store in accumulator(s)" $+ forM2_ (slugAccs slug) (map resSubExp $ bodyResult $ slugBody slug) $+ \(acc, acc_is) se ->+ copyDWIMFix acc (acc_is ++ vec_is) se [] noncommPrimParamsStageOneBody :: [SegBinOpSlug] ->@@ -870,7 +894,7 @@ copyDWIMFix priv_chunk [k] ne [] ) - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty sComment "effectualize collective copies in shared memory" $ do forM_ slugs $ \slug -> do@@ -890,58 +914,42 @@ sOp $ Imp.Barrier Imp.FenceLocal - sComment "per-thread sequential reduction of private chunk(s)" $ do+ sComment "per-thread sequential reduction of private chunk(s)" $ chunkLoop $ \k ->- forM_ slugs $ \slug -> do- let accs = map fst $ slugAccs slug- let (acc_ps, next_ps) = slugSplitParams slug- let ps_accs_chunks = zip4 acc_ps next_ps accs (slugPrivChunks slug)-- sComment "load params for all reductions" $ do- forM_ ps_accs_chunks $ \(acc_p, next_p, acc, priv_chunk) -> do- copyDWIM (paramName acc_p) [] (Var acc) []- copyDWIMFix (paramName next_p) [] (Var priv_chunk) [k]-- sComment "apply reduction operator(s)" $ do- let binop_ress = map resSubExp $ bodyResult $ slugBody slug- compileStms mempty (bodyStms $ slugBody slug) $- forM2_ accs binop_ress $ \acc binop_res ->- copyDWIM acc [] binop_res []+ forM_ slugs $ \slug ->+ applyRedOp slug [] [(Var c, [k]) | c <- slugPrivChunks slug] doLMemBlockReduce- sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty +-- | The second stage of a reduction where each segment is handled by more than+-- one block. Each block writes its partial results for all the operators, and+-- the last block of the segment to do so combines them into the final results.+-- The blocks find out which of them is last by incrementing a counter for the+-- segment, which the last block then resets. reductionStageTwo ::- [PatElem LetDecMem] -> Imp.TExp Int32 -> [Imp.TExp Int64] -> Imp.TExp Int64 -> Imp.TExp Int64 ->- SegBinOpSlug ->- Lambda GPUMem ->+ [([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem)] -> VName -> VName -> Imp.TExp Int64 -> InKernelGen ()-reductionStageTwo segred_pes tblock_id segment_gtids first_block_for_segment blocks_per_segment slug new_lambda counters sync_arr counter_idx = do+reductionStageTwo tblock_id segment_gtids first_block_for_segment blocks_per_segment reds counters sync_arr counter_idx = do constants <- kernelConstants <$> askEnv let ltid32 = kernelLocalThreadId constants- ltid = sExt64 ltid32- tblock_size = kernelBlockSize constants - let (acc_params, next_params) = slugSplitParams slug- nes = slugNeutral slug- red_arrs = slugBlockRedArrs slug- block_res_arrs = blockResArrs slug- old_counter <- dPrim "old_counter" (counter_mem, _, counter_offset) <- fullyIndexArray counters [counter_idx]- sComment "first thread in block saves block result to global memory" $+ sComment "first thread in block saves block results to global memory" $ sWhen (ltid32 .==. 0) $ do- forM_ (take (length nes) $ zip block_res_arrs (slugAccs slug)) $ \(v, (acc, acc_is)) ->- writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is- -- Increment the counter, thus stating that our result is- -- available.+ forM_ reds $ \(_, slug, _) ->+ forM_ (take (length (slugNeutral slug)) $ zip (blockResArrs slug) (slugAccs slug)) $+ \(v, (acc, acc_is)) ->+ writeAtomic v [0, sExt64 tblock_id] (Var acc) acc_is+ -- Increment the counter, thus declaring that our results are available. sOp $ Imp.Atomic DefaultSpace $ Imp.AtomicAdd@@ -950,8 +958,8 @@ counter_mem counter_offset $ untyped (1 :: Imp.TExp Int32)- -- Now check if we were the last block to write our result. If- -- so, it is our responsibility to produce the final result.+ -- Now check if we were the last block to write our results. If+ -- so, it is our responsibility to produce the final results. sWrite sync_arr [0] $ untyped $ tvExp old_counter .==. sExt32 (blocks_per_segment - 1) sOp $ Imp.Barrier Imp.FenceGlobal@@ -959,7 +967,7 @@ is_last_block <- dPrim "is_last_block" copyDWIMFix (tvVar is_last_block) [] (Var sync_arr) [0] sWhen (tvExp is_last_block) $ do- -- The final block has written its result (and it was+ -- The final block has written its results (and it was -- us!), so read in all the block results and perform the -- final stage of the reduction. But first, we reset the -- counter so it is ready for next time. This is done@@ -972,74 +980,93 @@ untyped $ sExt32 (negate blocks_per_segment) - sLoopNest (slugShape slug) $ \vec_is -> do- unless (null $ slugShape slug) $- sOp (Imp.Barrier Imp.FenceLocal)+ mapM_ (finalReduction segment_gtids first_block_for_segment blocks_per_segment) reds - -- There is no guarantee that the number of threadblocks for the- -- segment is less than the threadblock size, so each thread may- -- have to read multiple elements. We do this in a sequential- -- way that may induce non-coalesced accesses, but the total- -- number of accesses should be tiny here.- --- -- TODO: here we *could* insert a collective copy of the num_tblocks- -- per-block results. However, it may not be beneficial, since num_tblocks- -- is not necessarily larger than tblock_size, meaning the number of- -- uncoalesced reads here may be insignificant. In fact, if we happen to- -- have a num_tblocks < tblock_size, then the collective copy would add- -- unnecessary overhead. Also, this code is only executed by a single- -- block.- sComment "read in the per-block-results" $ do- read_per_thread <-- dPrimVE "read_per_thread" $- blocks_per_segment `ceilDiv` sExt64 tblock_size+-- | Combine the partial results that the blocks of a segment have written for+-- one operator. Executed by a single block.+finalReduction ::+ [Imp.TExp Int64] ->+ Imp.TExp Int64 ->+ Imp.TExp Int64 ->+ ([PatElem LetDecMem], SegBinOpSlug, Lambda GPUMem) ->+ InKernelGen ()+finalReduction segment_gtids first_block_for_segment blocks_per_segment (segred_pes, slug, new_lambda) = do+ constants <- kernelConstants <$> askEnv - forM2_ acc_params nes $ \p ne ->- copyDWIM (paramName p) [] ne []+ let ltid32 = kernelLocalThreadId constants+ ltid = sExt64 ltid32+ tblock_size = kernelBlockSize constants - sFor "i" read_per_thread $ \i -> do- block_res_id <-- dPrimVE "block_res_id" $- ltid * read_per_thread + i- index_of_block_res <-- dPrimVE "index_of_block_res" $- first_block_for_segment + block_res_id+ let (acc_params, next_params) = slugSplitParams slug+ nes = slugNeutral slug+ red_arrs = slugBlockRedArrs slug+ block_res_arrs = blockResArrs slug - sWhen (block_res_id .<. blocks_per_segment) $ do- forM2_ next_params block_res_arrs $- \p block_res_arr ->- copyDWIMFix- (paramName p)- []- (Var block_res_arr)- ([0, index_of_block_res] ++ vec_is)+ sLoopNest (slugShape slug) $ \vec_is -> do+ unless (null $ slugShape slug) $+ sOp (Imp.Barrier Imp.FenceLocal) - compileStms mempty (bodyStms $ slugBody slug) $- forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->- copyDWIMFix (paramName p) [] se []+ -- There is no guarantee that the number of threadblocks for the segment is+ -- less than the threadblock size, so each thread may have to read multiple+ -- elements. We do this in a sequential way that may induce non-coalesced+ -- accesses, but the total number of accesses should be tiny here.+ --+ -- TODO: here we *could* insert a collective copy of the num_tblocks+ -- per-block results. However, it may not be beneficial, since num_tblocks+ -- is not necessarily larger than tblock_size, meaning the number of+ -- uncoalesced reads here may be insignificant. In fact, if we happen to+ -- have a num_tblocks < tblock_size, then the collective copy would add+ -- unnecessary overhead. Also, this code is only executed by a single block.+ sComment "read in the per-block-results" $ do+ read_per_thread <-+ dPrimVE "read_per_thread" $+ blocks_per_segment `ceilDiv` sExt64 tblock_size - forM2_ acc_params red_arrs $ \p arr ->- when (isPrimParam p) $- copyDWIMFix arr [ltid] (Var $ paramName p) []+ forM2_ acc_params nes $ \p ne ->+ copyDWIM (paramName p) [] ne [] - sOp $ Imp.ErrorSync Imp.FenceLocal+ sFor "i" read_per_thread $ \i -> do+ block_res_id <-+ dPrimVE "block_res_id" $+ ltid * read_per_thread + i+ index_of_block_res <-+ dPrimVE "index_of_block_res" $+ first_block_for_segment + block_res_id - sComment "reduce the per-block results" $ do- blockReduce (sExt32 tblock_size) new_lambda red_arrs+ sWhen (block_res_id .<. blocks_per_segment) $ do+ forM2_ next_params block_res_arrs $ \p block_res_arr ->+ copyDWIMFix+ (paramName p)+ []+ (Var block_res_arr)+ ([0, index_of_block_res] ++ vec_is) - sComment "and back to memory with the final result" $- sWhen (ltid32 .==. 0) $- forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->- copyDWIMFix- (patElemName pe)- (segment_gtids ++ vec_is)- (Var $ paramName p)- []+ compileStms mempty (bodyStms $ slugBody slug) $+ forM2_ acc_params (map resSubExp $ bodyResult $ slugBody slug) $ \p se ->+ copyDWIMFix (paramName p) [] se [] + forM2_ acc_params red_arrs $ \p arr ->+ when (isPrimParam p) $+ copyDWIMFix arr [ltid] (Var $ paramName p) []++ sOp $ Imp.ErrorSync Imp.FenceLocal mempty++ sComment "reduce the per-block results" $ do+ blockReduce (sExt32 tblock_size) new_lambda red_arrs++ sComment "and back to memory with the final result" $+ sWhen (ltid32 .==. 0) $+ forM2_ segred_pes (lambdaParams new_lambda) $ \pe p ->+ copyDWIMFix+ (patElemName pe)+ (segment_gtids ++ vec_is)+ (Var $ paramName p)+ []+ -- Note [IntermArrays] ----- Intermediate memory for the nonsegmented and large segments non-commutative--- reductions with all primitive parameters:+-- Intermediate memory for the nonsegmented and large segments reductions using+-- 'StageOneTiled' (see Note [Stage one strategies]): -- -- These kernels need shared memory for 1) the initial collective copy, 2) the -- (virtualized) block reductions, and (TODO: this one not implemented yet!)@@ -1078,9 +1105,63 @@ -- Aside from shared memory, these kernels also require a CHUNK-sized array of -- thread-private register memory per reduction operator. ----- For all other reductions, ie. commutative reductions, reductions with at--- least one non-primitive operator, and small segments reductions:+-- For all other reductions, ie. those using 'StageOneStrided',+-- 'StageOneContiguous', or 'StageOneWindow', and small segments+-- reductions: -- -- These kernels use shared memory only for the intra-block reductions, and -- since they do not use chunking or CHUNK, they all require onlly `tblock_size -- * max elem_sizes` bytes of shared memory and no thread-private register mem.++-- Note [Stage one strategies]+--+-- In the first stage of a nonsegmented or large-segments reduction, each+-- thread reduces some of the input elements, after which the per-thread+-- results are combined by a block-wide reduction. The strategy decides which+-- elements a thread reduces.+--+-- A commutative operator allows thread @t@ of @T@ to reduce elements @t@,+-- @t+T@, @t+2T@, and so on ('StageOneStrided'). This gives coalesced memory+-- accesses and a single block-wide reduction at the end, and we always pick+-- this strategy for any commutative operator.+--+-- A non-commutative operator requires that each thread reduces a contiguous+-- range of elements. There are three ways of achieving this:+--+-- - 'StageOneContiguous': each thread reduces a contiguous range of @q@+-- elements, followed by a single block-wide reduction. Order is preserved+-- because consecutive threads have consecutive ranges, the block-wide+-- reduction combines the threads in order, and stage two combines the blocks+-- in order. This is similar to 'StageOneStrided', but results in noncoalesced+-- reads. However, this strategy can still be a good choice for complex+-- noncommutative reductions that need fairly little input from global memory,+-- but have large operands.+--+-- - 'StageOneTiled': each block processes a sequence of contiguous tiles of+-- @tblock_size * chunk@ elements. The map function is applied to the elements+-- of a tile with coalesced accesses, its results are stored in thread-private+-- chunks, and the chunks are transposed through shared memory (the collective+-- copies), such that each thread obtains @chunk@ consecutive elements, which+-- it reduces sequentially. Since the elements a thread obtains from+-- consecutive tiles are not adjacent, each tile ends with a block-wide+-- reduction, whose result thread 0 keeps in its accumulator. The chunk+-- size amortises the block-wide reductions, but is bounded by the private+-- storage of @chunk@ copies of the operands (see 'getRedChunkSize'). This+-- requires non-array operands.+--+-- - 'StageOneWindow': each block slides a window of @tblock_size@ elements+-- across a contiguous range of @q * tblock_size@ elements, with coalesced+-- accesses and a block-wide reduction per window, whose result is kept in+-- the accumulator of thread 0. This is like 'StageOneTiled' with a chunk+-- size of 1, but needs no private copies of the operands, and so works for+-- non-primitive operands. It shares its code with 'StageOneStrided',+-- except for the indexing and the placement of the block-wide reduction.+--+-- 'StageOneTiled' is preferable for cheap operators, where the reduction is+-- bound by memory bandwidth. 'StageOneContiguous' is preferable for expensive+-- operators, where the block-wide reductions (which apply the operator several+-- times per element in each wave, often divergently) dominate. We have no+-- good static measure of operator cost, so we use the size of the operands as+-- a crude proxy: we use 'StageOneTiled' when the operands are primitive and+-- take up at most 'maxTiledOperandWords' 32-bit words in total. The threshold+-- is a heuristic.
src/Futhark/CodeGen/ImpGen/GPU/SegScan.hs view
@@ -3,8 +3,6 @@ -- the scan and the chosen abckend. module Futhark.CodeGen.ImpGen.GPU.SegScan (compileSegScan) where -import Control.Monad-import Data.Maybe import Futhark.CodeGen.ImpCode.GPU qualified as Imp import Futhark.CodeGen.ImpGen hiding (compileProg) import Futhark.CodeGen.ImpGen.GPU.Base@@ -37,17 +35,6 @@ (concatMap (bodyResult . lambdaBody) lams) } -bodyHas :: (Exp GPUMem -> Bool) -> Body GPUMem -> Bool-bodyHas f = any (f' . stmExp) . bodyStms- where- f' e- | f e = True- | otherwise = isNothing $ walkExpM walker e- walker =- identityWalker- { walkOnBody = const $ guard . not . bodyHas f- }- canBeSinglePass :: [SegBinOp GPUMem] -> Maybe (SegBinOp GPUMem) canBeSinglePass scan_ops = if all ok scan_ops@@ -57,9 +44,7 @@ ok op = segBinOpShape op == mempty && all primType (lambdaReturnType (segBinOpLambda op))- && not (bodyHas isAssert (lambdaBody (segBinOpLambda op)))- isAssert (BasicOp Assert {}) = True- isAssert _ = False+ && not (bodyMayFail (lambdaBody (segBinOpLambda op))) -- | Compile 'SegScan' instance to host-level code with calls to -- various kernels.
src/Futhark/CodeGen/ImpGen/GPU/SegScan/SinglePass.hs view
@@ -136,14 +136,17 @@ statusA = 1 statusP = 2 +-- | Scan the look-back window held in the given flag and value arrays, using+-- the first 32 threads, which form a single warp. Basically a kind of segmented+-- scan, optimised for only the cases handled by single pass scan (in+-- particular, no array operands). inBlockScanLookback :: KernelConstants ->- Imp.TExp Int64 -> VName -> [VName] -> Lambda GPUMem -> InKernelGen ()-inBlockScanLookback constants arrs_full_size flag_arr arrs scan_lam = do+inBlockScanLookback constants flag_arr arrs scan_lam = do flg_x :: TV Int8 <- dPrim "flg_x" flg_y :: TV Int8 <- dPrim "flg_y" let flg_param_x = Param mempty (tvVar flg_x) (MemPrim p_int8)@@ -155,43 +158,26 @@ dLParams (lambdaParams scan_lam) skip_threads <- dPrim "skip_threads"- let in_block_thread_active =- tvExp skip_threads .<=. in_block_id- actual_params = lambdaParams scan_lam+ let thread_active = tvExp skip_threads .<=. ltid32 (x_params, y_params) =- splitAt (length actual_params `div` 2) actual_params+ splitAt (length (lambdaParams scan_lam) `div` 2) $ lambdaParams scan_lam y_to_x =- forM_ (zip x_params y_params) $ \(x, y) ->- when (primType (paramType x)) $- copyDWIM (paramName x) [] (Var (paramName y)) []- y_to_x_flg =- copyDWIM (tvVar flg_x) [] (Var (tvVar flg_y)) []-- -- Set initial y values- sComment "read input for in-block scan" $ do- zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)- -- Since the final result is expected to be in x_params, we may- -- need to copy it there for the first thread in the block.- sWhen (in_block_id .==. 0) $ do- y_to_x- y_to_x_flg-- when array_scan barrier-- let op_to_x = do+ forM_ (zip (flg_param_x : x_params) (flg_param_y : y_params)) $ \(x, y) ->+ copyDWIM (paramName x) [] (Var (paramName y)) []+ op_to_x = sIf (flg_y_exp .==. statusP_e .||. flg_y_exp .==. statusX_e)- ( do- y_to_x_flg- y_to_x- )+ y_to_x (compileBody' x_params $ lambdaBody scan_lam) - sComment "in-block scan (hopefully no barriers needed)" $ do+ sComment "read input for warp scan" $+ zipWithM_ readInitial (flg_param_y : y_params) (flag_arr : arrs)++ sComment "warp scan" $ do skip_threads <-- 1 - sWhile (tvExp skip_threads .<. block_size) $ do- warpLocal . sWhen in_block_thread_active $ do+ sWhile (tvExp skip_threads .<. warp_size) $ do+ warpLocal . sWhen thread_active $ do sComment "read operands" $ zipWithM_ (readParam (sExt64 $ tvExp skip_threads))@@ -199,7 +185,7 @@ (flag_arr : arrs) sComment "perform operation" op_to_x - warpLocal . sWhen in_block_thread_active . sComment "write result" $+ warpLocal . sWhen thread_active . sComment "write result" $ sequence_ $ zipWith3 writeResult@@ -210,33 +196,18 @@ skip_threads <-- tvExp skip_threads * 2 where p_int8 = IntType Int8- block_size = 32- block_id = ltid32 `quot` block_size- in_block_id = ltid32 - block_id * block_size+ warp_size = 32 ltid32 = kernelLocalThreadId constants ltid = sExt64 ltid32- gtid = sExt64 $ kernelGlobalThreadId constants- array_scan = not $ all primType $ lambdaReturnType scan_lam- barrier- | array_scan =- sOp $ Imp.Barrier Imp.FenceGlobal- | otherwise =- sOp $ Imp.Barrier Imp.FenceLocal - readInitial p arr- | primType $ paramType p =- copyDWIMFix (paramName p) [] (Var arr) [ltid]- | otherwise =- copyDWIMFix (paramName p) [] (Var arr) [gtid]- readParam behind p arr- | primType $ paramType p =- copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]- | otherwise =- copyDWIMFix (paramName p) [] (Var arr) [gtid - behind + arrs_full_size]+ readInitial p arr =+ copyDWIMFix (paramName p) [] (Var arr) [ltid] + readParam behind p arr =+ copyDWIMFix (paramName p) [] (Var arr) [ltid - behind]+ writeResult x y arr = do- when (isPrimParam x) $- copyDWIMFix arr [ltid] (Var $ paramName x) []+ copyDWIMFix arr [ltid] (Var $ paramName x) [] copyDWIM (paramName y) [] (Var $ paramName x) [] -- | Calculate the number of u64 words needed to store n bits@@ -371,9 +342,6 @@ tvSize <$> dPrimV "num_virt_blocks" (n `ceilDiv` (tblock_size_e * chunk)) let num_virt_blocks_e = pe64 num_virt_blocks - num_virt_threads <-- dPrimVE "num_virt_threads" $ num_virt_blocks_e * tblock_size_e- let (gtids, dims) = unzip $ unSegSpace space dims' = map pe64 dims segmented = length dims' > 1@@ -392,7 +360,8 @@ <$> sAllocArray "aggregates" ty (Shape [num_virt_blocks]) (Space "device") <*> sAllocArray "incprefixes" ty (Shape [num_virt_blocks]) (Space "device") - global_id <- genZeroes "global_dynid" 1+ -- The first counter hands out the dynamic IDs.+ global_id <- bookkeepingArray Imp.BookkeepingCounters let attrs' = attrs@@ -555,7 +524,16 @@ sIf (virt_tid .<. n) in_bounds out_of_bounds - sOp $ Imp.ErrorSync Imp.FenceLocal+ -- See Note [Failures in single-pass scan].+ publish_on_failure <- collect . sWhen (ltid32 .==. 0) $ do+ everythingVolatile $+ forM_ (zip incprefixArrays scanop_nes) $ \(incprefixArray, ne) ->+ copyDWIMFix incprefixArray [tvExp dyn_id] ne []+ sOp global_fence+ everythingVolatile $+ copyDWIMFix statusFlags [tvExp dyn_id] (intConst Int8 statusP) []+ sOp $ Imp.ErrorSync Imp.FenceLocal publish_on_failure+ sComment "Transpose scan inputs" $ do forM_ (zip transposedArrays scan_private_chunks) $ \(trans, priv) -> do sFor "i" chunk $ \i -> do@@ -611,11 +589,11 @@ blockScan crossesSegment tblock_size_e- num_virt_threads+ tblock_size_e scan_op1 prefixArrays - sOp $ Imp.ErrorSync Imp.FenceLocal+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty let firstThread acc prefixes = copyDWIMFix (tvVar acc) [] (Var prefixes) [sExt64 tblock_size_e - 1]@@ -725,7 +703,6 @@ lam' <- renameLambda scan_op1 inBlockScanLookback constants- num_virt_threads warpscan exchanges lam'@@ -846,7 +823,7 @@ lambdaParams $ segPostOpLambda post_op - sComment "Compute post op and write to global memory." $ do+ sComment "Compute post op and write to global memory." $ sFor "i" chunk $ \i -> do flat_idx <- dPrimVE "flat_idx" $ thd_offset + i * tblock_size_e sWhen (flat_idx .<. n) $ do@@ -883,5 +860,28 @@ then pure () else copyDWIMFix (patElemName pe) (map le64 gtids) subexp [] - sOp local_barrier+ -- The block has published its prefix, so it may stop here.+ sOp $ Imp.ErrorSync Imp.FenceLocal mempty {-# NOINLINE compileSegScan #-}++-- Note [Failures in single-pass scan]+--+-- Normally, when a thread fails, its whole thread block stops at the next+-- 'Imp.ErrorSync'. The single-pass scan cannot allow that before the block has+-- published its prefix, as each block waits in the lookback for the status+-- flags of the blocks with smaller dynamic IDs, so a block that stops before+-- publishing its flag leaves every later block waiting forever. Stopping after+-- publishing is fine, no block waits for the larger dynamic IDs that are then+-- skipped. The block that fetches the last ID normally resets the dynamic ID+-- counter, so after a failure the counter may be left nonzero. This is+-- harmless, as the runtime system resets the bookkeeping buffers when it+-- reports the failure.+--+-- The only failures before publishing are in the segop body (the "map part"),+-- as 'canBeSinglePass' checks that the scan operator cannot fail. The+-- 'Imp.ErrorSync' after the map therefore has a handler that publishes the+-- neutral element as the block's inclusive prefix before the block stops. The+-- later blocks then compute meaningless results, but as the failure is recorded+-- globally, the results are never used. Failures in the post op reach the+-- 'Imp.ErrorSync' at the end of the virtual block, after the prefix has been+-- published.
src/Futhark/CodeGen/ImpGen/GPU/SegScan/TwoPass.hs view
@@ -257,7 +257,7 @@ to' = sExt64 to + tvExp chunk_offset in f from' to' - sOp $ Imp.ErrorSync fence+ sOp $ Imp.ErrorSync fence mempty -- We need to avoid parameter name clashes. scan_op_renamed <- renameLambda scan_op@@ -386,7 +386,7 @@ -- reload carry (xParams) from the scan output written by the -- previous chunk. Thread 0 reads the last element of the -- previous chunk, unless a segment boundary falls between the- -- chunks.+ -- chunks, or that element is out of bounds. when use_global_carry $ do crosses_seg <- dPrimVE "crosses_seg" $@@ -394,7 +394,11 @@ Nothing -> false Just f -> f prev_chunk_last (prev_chunk_last + 1) sIf- (chunk_id .>. 0 .&&. kernelLocalThreadId constants .==. 0 .&&. bNot crosses_seg)+ ( (chunk_id .>. 0)+ .&&. (kernelLocalThreadId constants .==. 0)+ .&&. (prev_chunk_last .<. product dims')+ .&&. bNot crosses_seg+ ) ( do let carry_is = unflattenIndex dims' prev_chunk_last forM_ (zip scan_x_params pes) $ \(p, pe) ->@@ -425,7 +429,7 @@ \(arr, se) -> copyDWIMFix arr [localArrayIndex constants] se [] - sOp $ Imp.ErrorSync fence+ sOp $ Imp.ErrorSync fence mempty -- crossesSegment' maps block-local thread IDs to element -- indices, adjusting for the current chunk offset.@@ -575,8 +579,9 @@ (Var $ paramName p) [] - sOp $ Imp.Barrier Imp.FenceLocal-+ -- Each thread reads only the elements it wrote itself, so no barrier is+ -- needed, and a failure must be able to reach the 'Imp.ErrorSync' at the+ -- end of the iteration without passing one. if isIdentityPostOp post_op then pure () else do@@ -607,8 +612,6 @@ forM_ (zip (patElems pat) res) $ \(pe, subexp) -> copyDWIMFix (patElemName pe) (map Imp.le64 gtids) subexp [] - sOp $ Imp.Barrier Imp.FenceGlobal- isIdentityPostOp :: SegPostOp rep -> Bool isIdentityPostOp post_op = map resSubExp (bodyResult (lambdaBody lam))@@ -636,12 +639,15 @@ -- The stage-2 block size is a tunable/user-settable parameter. It -- is independent of stage1_num_tblocks so the autotuner can treat- -- it as a fixed knob.+ -- it as a fixed knob. It may not exceed the number of stage-1+ -- threads, because the memory of array-typed operator parameters+ -- is allocated for that many threads. stage2_tblock_size_param <- getSize "segscan_stage2_tblock_size" SizeThreadBlock stage2_tblock_size <- fmap (Imp.Count . tvSize) $ dPrimV "stage2_tblock_size" $- tvExp stage2_tblock_size_param+ sMin64 (tvExp stage2_tblock_size_param) $+ pe64 (unCount (kAttrNumBlocks attrs)) * pe64 (unCount (kAttrBlockSize attrs)) let shpT op = (segBinOpShape op,) <$> lambdaReturnType (segBinOpLambda op) scan_ts = concatMap shpT scans
src/Futhark/CodeGen/ImpGen/GPU/ToOpenCL.hs view
@@ -126,6 +126,7 @@ getSize (ImpOpenCL.CmpSizeLe _ v _) = Just v getSize (ImpOpenCL.GetSizeMax {}) = Nothing getSize (ImpOpenCL.LaunchKernel {}) = Nothing+ getSize (ImpOpenCL.GetBookkeeping {}) = Nothing directUseInFun fun = mapMaybe getSize $ toList $ functionBody fun direct_uses = map (second directUseInFun) $ unFunctions $ defFuns defs @@ -158,18 +159,11 @@ data KernelState = KernelState { kernelSharedMemory :: [SharedMemoryUse], kernelFailures :: [FailureMsg],- kernelNextSync :: Int,- -- | Has a potential failure occurred sine the last- -- ErrorSync?- kernelSyncPending :: Bool, kernelHasBarriers :: Bool } newKernelState :: [FailureMsg] -> KernelState-newKernelState failures = KernelState mempty failures 0 False False--errorLabel :: KernelState -> String-errorLabel = ("error_" ++) . show . kernelNextSync+newKernelState failures = KernelState mempty failures False data ToOpenCL = ToOpenCL { clGPU :: M.Map KernelName (KernelSafety, T.Text),@@ -195,6 +189,7 @@ codeMayFail f (For _ _ x) = codeMayFail f x codeMayFail f (While _ x) = codeMayFail f x codeMayFail f (If _ x y) = codeMayFail f x || codeMayFail f y+codeMayFail f (OnFailure x y) = codeMayFail f x || codeMayFail f y codeMayFail _ _ = False hostOpMayFail :: ImpGPU.HostOp -> Bool@@ -235,6 +230,8 @@ pure $ ImpOpenCL.CmpSizeLe v key x onHostOp _ (ImpGPU.GetSizeMax v size_class) = pure $ ImpOpenCL.GetSizeMax v size_class+onHostOp _ (ImpGPU.GetBookkeeping v b) =+ pure $ ImpOpenCL.GetBookkeeping v b genGPUCode :: Env ->@@ -303,7 +300,8 @@ calledInHostOp _ = mempty calledInKernelOp :: KernelOp -> S.Set Name-calledInKernelOp = const mempty+calledInKernelOp (ErrorSync _ handler) = calledFuncs calledInKernelOp handler+calledInKernelOp _ = mempty ensureDeviceFuns :: ImpGPU.KernelCode -> OnKernelM [Name] ensureDeviceFuns code = do@@ -347,6 +345,7 @@ -- No need to free, as we cannot allocate memory in kernels. mapM_ GC.item =<< GC.declAllocatedMem mapM_ GC.item body+ GC.items =<< GC.errorLabel kstate = GC.compUserState cstate (kernel_consts, (const_defs, const_undefs)) =@@ -359,17 +358,32 @@ let (use_params, unpack_params) = unzip $ mapMaybe useAsParam $ kernelUses kernel - -- The local_failure variable is an int despite only really storing- -- a single bit of information, as some OpenCL implementations- -- (e.g. AMD) does not like byte-sized shared memory (and the others- -- likely pad to a whole word anyway).- let (safety, error_init)+ -- The local_failure variable is an int despite only really storing a single+ -- bit of information, as some OpenCL implementations (e.g. AMD) does not like+ -- byte-sized shared memory (and the others likely pad to a whole word+ -- anyway). The CUDA/HIP prelude #defines __local to nothing, so we need+ -- special handling here to get a shared-memory local variable. Ugly...+ let cuda_local_failure_decl =+ [C.citem|$escstm:("__shared__ volatile int local_failure;")|]+ local_failure_decl =+ case target of+ TargetOpenCL -> [C.citem|volatile __local int local_failure;|]+ TargetCUDA -> cuda_local_failure_decl+ TargetHIP -> cuda_local_failure_decl++ -- No thread may write a failure before all threads have cleared it.+ local_failure_barrier+ | kernelHasBarriers kstate = [C.citems|barrier(CLK_LOCAL_MEM_FENCE);|]+ | otherwise = []++ (safety, error_init) -- We conservatively assume that any called function can fail. | not $ null called = ( SafetyFull,- [C.citems|volatile __local int local_failure;+ [C.citems|$item:local_failure_decl // Harmless for all threads to write this.- local_failure = 0;|]+ local_failure = 0;+ $items:local_failure_barrier|] ) | length (kernelFailures kstate) == length failures = if kernelFailureTolerant kernel@@ -388,14 +402,14 @@ else ( SafetyFull, [C.citems|- volatile __local int local_failure;+ $item:local_failure_decl if (failure_is_an_option) { int failed = *global_failure >= 0; if (failed) { return; } }- // All threads write this value - it looks like CUDA has a compiler bug otherwise.+ // Harmless for all threads to write this. local_failure = 0; barrier(CLK_LOCAL_MEM_FENCE); |]@@ -457,8 +471,6 @@ $items:error_init $items:kernel_body - $id:(errorLabel kstate): return;- $items:const_undefs }|] modify $ \s ->@@ -565,18 +577,6 @@ useToArg (ScalarUse v pt) = Just $ ValueKArg (LeafExp v pt) pt useToArg ConstUse {} = Nothing -nextErrorLabel :: GC.CompilerM KernelOp KernelState String-nextErrorLabel =- errorLabel <$> GC.getUserState--incErrorLabel :: GC.CompilerM KernelOp KernelState ()-incErrorLabel =- GC.modifyUserState $ \s -> s {kernelNextSync = kernelNextSync s + 1}--pendingError :: Bool -> GC.CompilerM KernelOp KernelState ()-pendingError b =- GC.modifyUserState $ \s -> s {kernelSyncPending = b}- hasCommunication :: ImpGPU.KernelCode -> Bool hasCommunication = any communicates where@@ -642,16 +642,15 @@ GC.modifyUserState $ \s -> s {kernelSharedMemory = (name', size) : kernelSharedMemory s} GC.stm [C.cstm|$id:name = (__local unsigned char*) $id:name';|]- kernelOps (ErrorSync f) = do- label <- nextErrorLabel- pending <- kernelSyncPending <$> GC.getUserState- when pending $ do- pendingError False- GC.stm [C.cstm|$id:label: barrier($exp:(fence f));|]- GC.stm [C.cstm|if (local_failure) { return; }|]+ kernelOps (ErrorSync f handler) = do+ error_label <- GC.errorLabel+ unless (null error_label) $ do+ handler' <- GC.collect $ GC.compileCode handler+ GC.items error_label+ GC.stm [C.cstm|barrier($exp:(fence f));|]+ GC.stm [C.cstm|if (local_failure) { $items:handler' return; }|] GC.stm [C.cstm|barrier($exp:(fence f));|] GC.modifyUserState $ \s -> s {kernelHasBarriers = True}- incErrorLabel kernelOps (Atomic space aop) = atomicOps space aop atomicCast s t = do@@ -814,16 +813,20 @@ kernelReadScalar = GC.readScalarPointerWithQuals pointerQuals - whatNext = do- label <- nextErrorLabel- pendingError True- pure $- if has_communication- then [C.citems|local_failure = 1; goto $id:label;|]- else- if mode == FunMode- then [C.citems|return 1;|]- else [C.citems|return;|]+ -- A failing thread jumps to the next error label, which is either at an+ -- 'ErrorSync', before the handler of an 'OnFailure', or at the end of the+ -- kernel. Since 'local_failure' is set before the jump, the handler of an+ -- 'OnFailure' can propagate the failure with a plain jump. Device functions+ -- instead return on failure, which would skip such a handler, so they must+ -- not contain an 'OnFailure'.+ whatNext+ | mode == FunMode = pure [C.citems|return 1;|]+ | has_communication = do+ failure <- GC.failureJump+ pure [C.citems|local_failure = 1; $stm:failure|]+ | otherwise = do+ failure <- GC.failureJump+ pure [C.citems|$stm:failure|] callInKernel dests fname args | functionMayFail fname env = do@@ -897,10 +900,12 @@ typesInCode (If (TPrimExp e) c1 c2) = typesInExp e <> typesInCode c1 <> typesInCode c2 typesInCode (Assert e _ _) = typesInExp e+typesInCode (OnFailure body handler) = typesInCode body <> typesInCode handler typesInCode (Meta _) = mempty typesInCode (DebugPrint _ v) = maybe mempty typesInExp v typesInCode (TracePrint msg) = foldMap typesInExp msg typesInCode (GetUserParam _ _ def) = typesInExp (untyped def)+typesInCode (Op (ErrorSync _ handler)) = typesInCode handler typesInCode Op {} = mempty typesInExp :: Exp -> S.Set PrimType
src/Futhark/CodeGen/ImpGen/Multicore/Base.hs view
@@ -38,6 +38,7 @@ import Futhark.Error import Futhark.IR.MCMem import Futhark.Transform.Rename+import Futhark.Util (nubOrd) import Prelude hiding (quot, rem) -- | Is there an atomic t'BinOp' corresponding to this t'BinOp'?@@ -107,13 +108,15 @@ _ -> pure $ product $ init ns_64 -- When the SegRed's return value is a scalar--- we perform a call by value-result in the segop function+-- we perform a call by value-result in the segop function. Several+-- array results may reside in the same memory block (e.g. after+-- short-circuiting), so the parameters are deduplicated. getReturnParams :: Pat LetDecMem -> SegOp () MCMem -> MulticoreGen [Imp.Param] getReturnParams pat SegRed {} = -- It's a good idea to make sure any prim values are initialised, as -- we will load them (redundantly) in the task code, and -- uninitialised values are UB.- fmap concat . forM (patElems pat) $ \pe -> do+ fmap (nubOrd . concat) . forM (patElems pat) $ \pe -> do case patElemType pe of Prim pt -> patElemName pe <~~ ValueExp (blankPrimValue pt) _ -> pure ()@@ -149,6 +152,7 @@ isLoadBalanced (a Imp.:>>: b) = isLoadBalanced a && isLoadBalanced b isLoadBalanced (Imp.For _ _ a) = isLoadBalanced a isLoadBalanced (Imp.If _ a b) = isLoadBalanced a && isLoadBalanced b+isLoadBalanced (Imp.OnFailure a b) = isLoadBalanced a && isLoadBalanced b isLoadBalanced Imp.While {} = False isLoadBalanced (Imp.Op (Imp.ParLoop _ code _)) = isLoadBalanced code isLoadBalanced (Imp.Op (Imp.ForEachActive _ a)) = isLoadBalanced a@@ -464,8 +468,11 @@ sUnless (lock_acquired .==. 0) $ do dLParams acc_params bind_acc_params- op_body- do_hist+ -- A thread whose operator fails must still release the lock, or other+ -- threads would wait for it forever.+ update <- collect $ op_body >> do_hist+ on_failure <- collect release_lock+ emit $ Imp.OnFailure update on_failure release_lock where writeArray bucket arr val = copyDWIMFix arr bucket val []
src/Futhark/CodeGen/RTS/JavaScript.hs view
@@ -4,7 +4,7 @@ module Futhark.CodeGen.RTS.JavaScript ( serverJs, valuesJs,- wrapperclassesJs,+ wrappersJs, ) where @@ -19,6 +19,6 @@ valuesJs :: T.Text valuesJs = $(embedStringFile "rts/javascript/values.js") --- | @rts/javascript/wrapperclasses.js@-wrapperclassesJs :: T.Text-wrapperclassesJs = $(embedStringFile "rts/javascript/wrapperclasses.js")+-- | @rts/javascript/wrappers.js@+wrappersJs :: T.Text+wrappersJs = $(embedStringFile "rts/javascript/wrappers.js")
src/Futhark/Optimise/ArrayShortCircuiting/ArrayCoalescing.hs view
@@ -1326,8 +1326,12 @@ in -- It is not safe to short circuit if some other pattern -- element is aliased to this one, as this indicates the -- two pattern elements reference the same physical- -- value somehow.+ -- value somehow. Nor is it safe if the destination is+ -- itself bound by the pattern (e.g. the existential memory+ -- of a 'Match' or 'Loop'), as then it is not in scope where+ -- @b@ is created, although it is in @scope td_env@. if any ((`nameIn` aliasesOf patel) . patElemName) pes+ || x_mem `elem` map patElemName pes then failed else case M.lookup b vtab of Nothing ->
src/Futhark/Pass/ExpandAllocations.hs view
@@ -161,7 +161,7 @@ -- allocations inside the update function. lvl = SegThread SegNoVirt Nothing (op_lam', lam_allocs) =- extractLambdaAllocations (lvl, [0]) bound_outside mempty op_lam+ extractLambdaAllocations (lvl, [0], Nothing) bound_outside mempty op_lam variantAlloc (_, Var v, _) = v `notNameIn` bound_outside variantAlloc _ = False (variant_allocs, invariant_allocs) = M.partition variantAlloc lam_allocs@@ -178,7 +178,9 @@ let num_is = shapeRank shape is = map paramName $ take num_is $ lambdaParams op_lam (alloc_stms, alloc_offsets) <-- genericExpandedInvariantAllocations (const $ const (shape, map le64 is)) invariant_allocs+ genericExpandedInvariantAllocations+ (const $ const (map pe64 $ shapeDims shape, map le64 is))+ invariant_allocs scope <- askScope let scope' = scopeOf op_lam <> scope <> scopeOf alloc_stms@@ -217,7 +219,7 @@ ExpandM (Stms GPUMem, (SegLevel, [Lambda GPUMem], KernelBody GPUMem)) transformScanRed lvl space ops kbody = do bound_outside <- asks $ namesFromList . M.keys- let user = (lvl, [le64 $ segFlat space])+ let user = (lvl, [le64 $ segFlat space], Nothing) (kbody', kbody_allocs) = extractKernelBodyAllocations user bound_outside bound_in_kernel kbody (ops', ops_allocs) =@@ -343,8 +345,9 @@ type Exp64 = TPrimExp Int64 VName -- | Identifying the spot where an allocation occurs in terms of its--- level and unique thread ID.-type User = (SegLevel, [Exp64])+-- level and unique thread ID. For a 'SegThreadInBlock' user, also the+-- size of its index space, if that is bound outside the kernel.+type User = (SegLevel, [Exp64], Maybe Exp64) -- | A description of allocations that have been extracted, and how -- much memory (and which space) is needed.@@ -404,7 +407,7 @@ in (set_stms (stmsFromList stms) body, allocs) expandable :: User -> Space -> Bool-expandable (SegBlock {}, _) (Space "shared") = False+expandable (SegBlock {}, _, _) (Space "shared") = False expandable _ ScalarSpace {} = False expandable _ _ = True @@ -447,13 +450,22 @@ tell allocs pure body' - onOp (_, user_ids) (Inner (SegOp op)) =+ onOp (_, user_ids, _) (Inner (SegOp op)) = Inner . SegOp <$> mapSegOpM (opMapper user'') op where+ dims = segSpaceDims $ segSpace op user'' =- (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))])+ (segLevel op, user_ids ++ [le64 (segFlat (segSpace op))], extent)+ extent+ | SegThreadInBlock {} <- segLevel op,+ all boundOutside dims =+ Just $ product $ map pe64 dims+ | otherwise = Nothing onOp _ op = pure op + boundOutside (Var v) = v `nameIn` bound_outside+ boundOutside Constant {} = True+ opMapper user' = identitySegOpMapper { mapOnSegBinOpLambda = onLambda user',@@ -472,7 +484,7 @@ pure lam {lambdaBody = body} genericExpandedInvariantAllocations ::- (User -> Space -> (Shape, [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap)+ (User -> Space -> ([Exp64], [Exp64])) -> Extraction -> ExpandM (Stms GPUMem, RebaseMap) genericExpandedInvariantAllocations getNumUsers invariant_allocs = do -- We expand the invariant allocations by adding an inner dimension -- equal to the number of kernel threads.@@ -485,22 +497,20 @@ allocpat = Pat [PatElem mem $ MemMem space] total_size <- letExp "total_size" <=< toExp . product $- pe64 per_thread_size : map pe64 (shapeDims num_users)+ pe64 per_thread_size : num_users letBind allocpat $ Op $ Alloc (Var total_size) space pure $ M.singleton mem $ newBase user space newBaseThread user space _old_shape =- let (users_shape, user_ids) = getNumUsers user space- dims = map pe64 (shapeDims users_shape)+ let (dims, user_ids) = getNumUsers user space in ( flattenIndex dims user_ids, product dims ) - newBase user@(SegThreadInBlock {}, _) space = newBaseThread user space- newBase user@(SegThread {}, _) space = newBaseThread user space- newBase user@(SegBlock {}, _) space = \_old_shape ->- let (users_shape, user_ids) = getNumUsers user space- dims = map pe64 (shapeDims users_shape)+ newBase user@(SegThreadInBlock {}, _, _) space = newBaseThread user space+ newBase user@(SegThread {}, _, _) space = newBaseThread user space+ newBase user@(SegBlock {}, _, _) space = \_old_shape ->+ let (dims, user_ids) = getNumUsers user space in ( flattenIndex dims user_ids, product dims )@@ -514,16 +524,20 @@ expandedInvariantAllocations num_threads (Count num_tblocks) (Count tblock_size) = genericExpandedInvariantAllocations getNumUsers where- getNumUsers (SegThread {}, [gtid]) _ = (Shape [num_threads], [gtid])- getNumUsers (SegThread {}, [gid, ltid]) _ = (Shape [num_tblocks, tblock_size], [gid, ltid])- getNumUsers (SegThreadInBlock {}, [gtid]) _ = (Shape [num_threads], [gtid])- getNumUsers (SegThreadInBlock {}, [_gid, ltid]) (Space "shared") =- (Shape [tblock_size], [ltid])- getNumUsers (SegThreadInBlock {}, [gid, ltid]) (Space "device") =- (Shape [num_tblocks, tblock_size], [gid, ltid])- getNumUsers (SegBlock {}, [gid]) _ = (Shape [num_tblocks], [gid])+ getNumUsers (SegThread {}, [gtid], _) _ = ([pe64 num_threads], [gtid])+ getNumUsers (SegThread {}, [gid, ltid], _) _ =+ ([pe64 num_tblocks, pe64 tblock_size], [gid, ltid])+ getNumUsers (SegThreadInBlock {}, [gtid], _) _ = ([pe64 num_threads], [gtid])+ getNumUsers (SegThreadInBlock {}, [_gid, ltid], extent) (Space "shared") =+ ([inBlockUsers extent], [ltid])+ getNumUsers (SegThreadInBlock {}, [gid, ltid], extent) (Space "device") =+ ([pe64 num_tblocks, inBlockUsers extent], [gid, ltid])+ getNumUsers (SegBlock {}, [gid], _) _ = ([pe64 num_tblocks], [gid]) getNumUsers user space = error $ "getNumUsers: unhandled " ++ show (user, space) + -- The number of threads in a block that use an in-block allocation.+ inBlockUsers = maybe (pe64 tblock_size) (sMin64 (pe64 tblock_size))+ expandedVariantAllocations :: SubExp -> SegSpace ->@@ -770,9 +784,9 @@ offsetMemoryInLambda :: RebaseMap -> Lambda GPUMem -> OffsetM (Lambda GPUMem) offsetMemoryInLambda offsets lam = do- body <- inScopeOf lam $ offsetMemoryInBody offsets $ lambdaBody lam- let params = map (offsetMemoryInParam offsets) $ lambdaParams lam- pure $ lam {lambdaBody = body, lambdaParams = params}+ let lam' = lam {lambdaParams = map (offsetMemoryInParam offsets) (lambdaParams lam)}+ body <- inScopeOf lam' $ offsetMemoryInBody offsets $ lambdaBody lam'+ pure $ lam' {lambdaBody = body} -- A loop may have memory parameters, and those memory blocks may -- be expanded. We assume (but do not check - FIXME) that if the
src/Futhark/Pass/ExtractMulticore.hs view
@@ -123,6 +123,9 @@ where transformInput (shape, arrs, op) = (shape,arrs,) <$> traverse (bitraverse transformLambda pure) op+transformStm stm@(Let _ aux (Op _))+ | "sequential" `inAttrs` stmAuxAttrs aux =+ pure $ oneStm $ runIdentity $ rephraseStm toMC stm transformStm (Let pat aux (Op op)) = fmap (certify (stmAuxCerts aux)) <$> transformSOAC pat (stmAuxAttrs aux) op @@ -145,8 +148,10 @@ sequentialiseBody :: Body SOACS -> ExtractM (Body MC) sequentialiseBody = pure . runIdentity . rephraseBody toMC- where- toMC = injectSOACS OtherOp++-- | Keep SOACs as they are, to be compiled to sequential code.+toMC :: Rephraser Identity SOACS MC+toMC = injectSOACS OtherOp transformFunDef :: FunDef SOACS -> ExtractM (FunDef MC) transformFunDef (FunDef entry attrs name rettype params body) = do
src/Futhark/Pass/Flatten.hs view
@@ -170,7 +170,9 @@ (\i pe -> DistResult (ResTag i) (DistType [] (Rank 0) (patElemType pe)) (patElemName pe)) [0 ..] (patElems nested_pat)- env <- flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form)+ env <-+ freshenSOAC res $+ flattenScrema ops [] (DistEnv mempty) inps res (nested_pat, aux) (w, arrs, form) forM_ (zip (patNames pat) res) $ \(pat_v, r) -> case resVar (distResTag r) env of Regular v ->@@ -321,7 +323,8 @@ ~[pe] = patElems pat flattenBasicOp ops segments env (inps, res', pe, aux, op) Op (Screma w arrs form) ->- flattenScrema ops segments env inps res (pat, aux) (w, arrs, form)+ freshenSOAC res $+ flattenScrema ops segments env inps res (pat, aux) (w, arrs, form) Match scrutinees cases defaultCase rt -> flattenMatch ops segments env inps res aux scrutinees cases defaultCase rt Apply name args rettype s ->@@ -333,9 +336,11 @@ WithAcc inputs lam -> flattenWithAcc ops segments env inps res pat aux inputs lam Op (Hist w hist_inputs hist_ops bucket_fun) ->- flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun)+ freshenSOAC res $+ flattenHist ops segments env inps res (pat, aux) (w, hist_inputs, hist_ops, bucket_fun) Op (FlatMap w arrs lam) ->- flattenFlatMapNested ops segments env inps res aux w arrs lam+ freshenSOAC res $+ flattenFlatMapNested ops segments env inps res aux w arrs lam Op (Stream {}) -> error "transformDistStm: Stream should have been removed" Op (JVP {}) -> error "Unhandled JVP" Op (VJP {}) -> error "Unhandled VJP"@@ -378,6 +383,39 @@ env' <- foldM (flattenDistStm ops segments) env dstms concat <$> zipWithM (liftRegResult lvl segments w inputs env') rettype result +-- | Flatten a SOAC, copying those of its results that would otherwise alias+-- something. This is because a SOAC result is fresh, and so must be the results+-- of applying flattening to the SOAC, which is not the case if the body is e.g.+-- a slice or a transpose.+freshenSOAC :: [DistResult] -> FlattenM DistEnv -> FlattenM DistEnv+freshenSOAC res m = do+ (env, stms) <- collectStms m+ addStms stms+ let vs = map (repArray . (`resVar` env) . distResTag) res+ -- Transitive aliases of each result, including names bound in 'stms'.+ als = bodyAliases $ analyseBody mempty $ Body () stms (varsRes vs)+ bound = boundByStms stms+ snd <$> foldM (freshen bound) (mempty, env) (zip3 res vs als)+ where+ -- Only the data array of an irregular representation can be consumed.+ repArray (Regular v) = v+ repArray (Irregular irreg) = irregularD irreg++ freshen bound (taken, env) (r, v, v_als) = do+ v_t <- lookupType v+ let v_all = oneName v <> v_als+ aliases_outside = v_all `namesSubtract` bound /= mempty+ case v_t of+ Array {}+ | aliases_outside || v_all `namesIntersect` taken -> do+ v' <- letExp (baseName v <> "_fresh") $ BasicOp $ Replicate mempty $ Var v+ env' <- insertRepM (distResTag r) (replaceArray v' $ resVar (distResTag r) env) env+ pure (taken, env')+ _ -> pure (taken <> v_all, env)++ replaceArray v' (Regular _) = Regular v'+ replaceArray v' (Irregular irreg) = Irregular irreg {irregularD = v'}+ -- | A lifted function must return fresh, non-aliasing arrays (as it -- corresponds to 'map f'; see 'liftRetType'). This is not -- automatically the case: a result may alias a parameter (when a value@@ -570,11 +608,6 @@ pure $ Lambda params ret body' transformStm :: Attrs -> FunHasParallelism -> FunSizeParams -> Stm SOACS -> FlattenM ()-transformStm attrs funHasParallelism funSizeParams (Let pat aux (Op soac))- | "sequential_outer" `inAttrs` stmAuxAttrs aux = do- scope <- askScope- stms <- runBuilderT_ (FOT.transformSOAC pat soac) (castScope scope)- transformStms attrs funHasParallelism funSizeParams $ fmap (certify (stmAuxCerts aux)) stms transformStm _ _ _ stm | "sequential" `inAttrs` stmAuxAttrs (stmAux stm) = addStm $ soacsStmToGPU stm transformStm _ _ _ (Let pat aux (Op (Hist w arrs ops bucket_fun))) =
src/Futhark/Pass/Flatten/Builtins.hs view
@@ -4,6 +4,7 @@ ( BuiltinFn (..), builtinFunDef, determineReduceOp,+ getGridSize, genUniformSegHist, mkSegSpace, segMap,@@ -62,24 +63,17 @@ data ThreadRecommendation = ManyThreads | NoRecommendation SegVirt -numberOfBlocks ::+-- | Compute a sensible grid size given this many (virtual) threads.+getGridSize :: (MonadBuilder m, Op (Rep m) ~ HostOp inner (Rep m)) => Name -> SubExp ->- SubExp ->- m (SubExp, SubExp)-numberOfBlocks desc w tblock_size = do+ Count BlockSize SubExp ->+ m (Count NumBlocks SubExp)+getGridSize desc num_threads tblock_size = do max_num_tblocks_key <- nameFromText . prettyText <$> newVName (desc <> "_num_tblocks")- num_tblocks <-- letSubExp "num_tblocks" $- Op $- SizeOp $- CalcNumBlocks w max_num_tblocks_key tblock_size- num_threads <-- letSubExp "num_threads" $- BasicOp $- BinOp (Mul Int64 OverflowUndef) num_tblocks tblock_size- pure (num_tblocks, num_threads)+ fmap Count . letSubExp "num_tblocks" . Op . SizeOp $+ CalcNumBlocks num_threads max_num_tblocks_key (unCount tblock_size) -- | Like 'segThread', but cap the thread count to the input size. -- This is more efficient for small kernels, e.g. summing a small@@ -104,8 +98,8 @@ let grid = KernelGrid (Count usable_groups) (Count tblock_size) pure $ SegThread SegNoVirt (Just grid) NoRecommendation v -> do- (num_tblocks, _) <- numberOfBlocks desc w tblock_size- let grid = KernelGrid (Count num_tblocks) (Count tblock_size)+ num_tblocks <- getGridSize desc w $ Count tblock_size+ let grid = KernelGrid num_tblocks (Count tblock_size) pure $ SegThread v (Just grid) -- FIXME: We use segThreadCapped here because otherwise we may get
src/Futhark/Pass/Flatten/Incremental.hs view
@@ -55,7 +55,6 @@ -- * Levels defaultSegLevel,- inBlockSegLevel, allowVersioning, -- * Various queries@@ -84,9 +83,6 @@ defaultSegLevel :: SegLevel defaultSegLevel = SegThread SegVirt Nothing -inBlockSegLevel :: SegLevel-inBlockSegLevel = SegThreadInBlock SegNoVirt- allowVersioning :: SegLevel -> Bool allowVersioning SegThreadInBlock {} = False allowVersioning _ = True@@ -191,18 +187,30 @@ onlyExploitIntra attrs = "only_intra" `inAttrs` flatteningAttrs attrs +onlyExploitOuter :: Attrs -> Bool+onlyExploitOuter attrs =+ "only_outer" `inAttrs` flatteningAttrs attrs+ mayExploitOuter :: Attrs -> Bool mayExploitOuter attrs =- not $ "no_outer" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'+ not $ "no_outer" `inAttrs` attrs' || "only_full" `inAttrs` attrs' where attrs' = flatteningAttrs attrs mayExploitIntra :: Attrs -> Bool mayExploitIntra attrs =- not $ "no_intra" `inAttrs` attrs' || "only_inner" `inAttrs` attrs'+ not $ "no_intra" `inAttrs` attrs' || "only_full" `inAttrs` attrs' where attrs' = flatteningAttrs attrs +-- | Is this one of the flattening attributes that select which versions are+-- generated? These are the ones an enclosing @only_full@ overrides.+isVersioningAttr :: Attr -> Bool+isVersioningAttr (AttrComp f [x]) =+ f `elem` ["flattening", "incremental_flattening"]+ && x `elem` ["no_outer", "no_intra", "only_intra", "only_outer", "only_full"]+isVersioningAttr _ = False+ intraBlockAlternative :: Intrablock.IntrablockResult -> FlattenM (SubExp, Body GPU)@@ -215,7 +223,7 @@ BasicOp $ CmpOp (CmpSle Int64)- (Intrablock.intraThreadBlockSize intra)+ (unCount (Intrablock.intraThreadBlockSize intra)) max_tblock_size (intra_suff, _) <- sufficientParallelism@@ -263,7 +271,7 @@ _ | parallel_fun_inside -> full []- | "sequential_inner" `inAttrs` attrs -> do+ | onlyExploitOuter attrs -> do outer_body <- mkOuterBody kernelAlternatives desc result_ts outer_body [] Nothing@@ -317,7 +325,7 @@ scanRedAlternatives desc result_ts attrs parallel_fun_inside allow_versioning ws mkFullBody mkOuterBody | parallel_fun_inside = fullAlternative- | "sequential_inner" `inAttrs` attrs =+ | onlyExploitOuter attrs = outerAlternative | mayExploitOuter attrs && allow_versioning = fullWithOuterAlternative@@ -375,16 +383,14 @@ 0 where attrs = stmAuxAttrs $ stmAux stm- sequential_inner = "sequential_inner" `inAttrs` attrs zeroIfTooSmall (Constant (IntValue x)) | intToInt64 x < 32 = 0 zeroIfTooSmall _ = 1 - mapLike w lam' =- if sequential_inner- then 0- else max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam'))+ mapLike w lam'+ | onlyExploitOuter attrs = 0+ | otherwise = max (zeroIfTooSmall w) (bodyInterest (lambdaBody lam')) -- | A lambda is worth sequentialising if it contains enough nested parallelism -- of an interesting kind, or if distributing it would fragment sequential@@ -404,7 +410,7 @@ 0 :: Int | Op (Screma _ _ form@(ScremaForm lam' _ _ _)) <- stmExp stm, isJust $ isMapSOAC form =- if sequential_inner+ if onlyExploitOuter attrs then 0 else bodyInterest (depth + 1) (lambdaBody lam') | Loop _ _ body <- stmExp stm =@@ -432,7 +438,6 @@ 0 where attrs = stmAuxAttrs $ stmAux stm- sequential_inner = "sequential_inner" `inAttrs` attrs bodyHasParallelism :: FunHasParallelism -> Body SOACS -> Bool bodyHasParallelism funHasParallelism =@@ -521,15 +526,17 @@ <$> preprocessStms scope stms pure $ mkBody stms' $ varsRes $ patNames pat --- | Add the flattening attributes of the enclosing context to a statement. A--- statement that carries flattening attributes of its own is left alone, as--- those are more specific.+-- | Add the flattening attributes of the enclosing context to a statement, and+-- to the statements nested in its sequential control flow (see Note [Flattening+-- attribute propagation]). propagateAttrs :: Attrs -> Stm SOACS -> Stm SOACS propagateAttrs attrs stm | attrs' == mempty = stm- | flatteningAttrs (stmAuxAttrs (stmAux stm)) == mempty =- stm {stmAux = (stmAux stm) {stmAuxAttrs = attrs' <> stmAuxAttrs (stmAux stm)}}- | otherwise = stm+ | otherwise =+ stm+ { stmAux = (stmAux stm) {stmAuxAttrs = stm_attrs},+ stmExp = propagateInExp stm_attrs $ stmExp stm+ } where -- 'flatteningAttrs' strips the enclosing 'flattening', which has to be put -- back for the attributes to be recognised on the statement.@@ -537,17 +544,44 @@ mconcat . mapAttrs (oneAttr . AttrComp "flattening" . pure) $ flatteningAttrs attrs + own = stmAuxAttrs $ stmAux stm++ stm_attrs+ | "only_full" `inAttrs` flatteningAttrs attrs =+ attrs' <> mconcat (mapAttrs keepNonVersioning own)+ | flatteningAttrs own == mempty =+ attrs' <> own+ | otherwise =+ own++ keepNonVersioning attr+ | isVersioningAttr attr = mempty+ | otherwise = oneAttr attr++-- | Propagate attributes into the bodies of sequential control flow, but not+-- into SOAC lambdas.+propagateInExp :: Attrs -> Exp SOACS -> Exp SOACS+propagateInExp attrs (Loop merge form body) =+ Loop merge form $ propagateInBody attrs body+propagateInExp attrs (Match ses cases defbody dec) =+ Match ses (map (fmap $ propagateInBody attrs) cases) (propagateInBody attrs defbody) dec+propagateInExp attrs (WithAcc inputs lam) =+ WithAcc inputs lam {lambdaBody = propagateInBody attrs $ lambdaBody lam}+propagateInExp _ e = e++propagateInBody :: Attrs -> Body SOACS -> Body SOACS+propagateInBody attrs body =+ body {bodyStms = fmap (propagateAttrs attrs) (bodyStms body)}+ -- | Impose outside flattening attributes on a statement. Only SOACs are--- affected, and only those that carry no flattening attributes of their own, as--- those are more specific.+-- affected, following the rules of 'propagateAttrs'. imposeAttrs :: Attrs -> Stm SOACS -> Stm SOACS imposeAttrs attrs stm | Op {} <- stmExp stm = propagateAttrs attrs stm | otherwise = stm --- | Propagate incremental flattening attributes to the statements of--- a map lambda body. Statements that carry their own incremental--- flattening attributes are left alone.+-- | Propagate flattening attributes to the statements of a map lambda body,+-- following the rules of 'propagateAttrs'. propagateVersioningAttrs :: Attrs -> Lambda SOACS -> Lambda SOACS propagateVersioningAttrs attrs lam | flatteningAttrs attrs == mempty = lam@@ -558,3 +592,15 @@ { bodyStms = fmap (propagateAttrs attrs) (bodyStms (lambdaBody lam)) } }++-- Note [Flattening attribute propagation]+--+-- A flattening attribute on a SOAC applies to every SOAC nested inside it,+-- including inside loops, branches and with_acc. A nested statement keeps its+-- own flattening attributes, except that an enclosing @only_full@ replaces its+-- versioning attributes ('isVersioningAttr').+--+-- Attributes are copied onto a SOAC's body only once it has been preprocessed+-- ('propagateVersioningAttrs', 'factorScremaForParallelism'), as preprocessing+-- may rewrite statements and lose them. Nested SOAC lambdas are reached when+-- their own SOAC is flattened.
src/Futhark/Pass/Flatten/Intrablock.hs view
@@ -10,8 +10,10 @@ import Control.Monad import Control.Monad.RWS-import Control.Monad.Writer+import Control.Monad.State+import Data.Bitraversable import Data.Map qualified as M+import Data.Maybe import Data.Set qualified as S import Futhark.Analysis.PrimExp.Convert import Futhark.IR.GPU hiding (HistOp)@@ -21,7 +23,6 @@ import Futhark.Pass.Flatten.General import Futhark.Pass.Flatten.PreProcess (preprocessLambda) import Futhark.Tools-import Futhark.Transform.FirstOrderTransform qualified as FOT import Futhark.Transform.Rename import Futhark.Transform.ToGPU import Prelude hiding (log)@@ -34,13 +35,14 @@ data IntrablockResult = IntrablockResult { intraMinPar :: SubExp, intraAvailPar :: SubExp,- intraThreadBlockSize :: SubExp,+ intraThreadBlockSize :: Count BlockSize SubExp, intraPreludeStms :: Stms GPU, intraKernelStms :: Stms GPU, intraResultNames :: [VName] } type InBlockMapTransformer =+ SegLevel -> Pat Type -> SubExp -> [VName] ->@@ -51,9 +53,10 @@ foldBinOp' _ [] = eSubExp $ intConst Int64 1 foldBinOp' bop (x : xs) = foldBinOp bop x xs --- | Extract the intra-block parallelism used by the body.-findParallelism :: KernelBody GPU -> [[SubExp]]-findParallelism = S.toList . execWriter . onKernelBody+-- | Extract the intra-block parallelism used by the body, paired with whether+-- it is virtualised.+findParallelism :: KernelBody GPU -> [([SubExp], SegVirt)]+findParallelism = S.toList . flip execState mempty . onKernelBody where onKernelBody = mapM_ onStm . bodyStms onBody = mapM_ onStm . bodyStms@@ -64,37 +67,39 @@ walkOnOp = onOp } onOp (SegOp op) = do- tell $ S.singleton $ segSpaceDims $ segSpace op+ modify $ S.insert (segSpaceDims $ segSpace op, segVirt (segLevel op)) onKernelBody $ segBody op onOp _ = pure () -computeThreadBlockSize :: [[SubExp]] -> [[SubExp]] -> FlattenM (SubExp, SubExp)-computeThreadBlockSize wss_min wss_avail = do- ws_min <-- mapM (letSubExp "one_intra_par_min" <=< foldBinOp' (Mul Int64 OverflowUndef)) $- filter (not . null) wss_min- ws_avail <-- mapM (letSubExp "one_intra_par_avail" <=< foldBinOp' (Mul Int64 OverflowUndef)) $- filter (not . null) wss_avail+onlyNonVirt :: [(a, SegVirt)] -> [a]+onlyNonVirt = mapMaybe f+ where+ f (_, SegVirt) = Nothing+ f (ws, _) = Just ws - -- The amount of parallelism available *in the worst case* is- -- equal to the smallest parallel loop, or *at least* 1.+computeThreadBlockSize ::+ [([SubExp], SegVirt)] ->+ FlattenM (SubExp, Count BlockSize SubExp)+computeThreadBlockSize wss = do+ let prod = letSubExp "one_intra_par" <=< foldBinOp' (Mul Int64 OverflowUndef)+ ws <- mapM (bitraverse prod pure) $ filter (not . null . fst) wss++ -- The amount of parallelism available *in the worst case* is equal to the+ -- smallest parallel loop. intra_avail_par <-- letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) ws_avail+ letSubExp "intra_avail_par" =<< foldBinOp' (SMin Int64) (map fst ws) - tblock_size <- newVName "computed_tblock_size"- -- The group size is either the maximum of the minimum parallelism- -- exploited, or the desired parallelism (bounded by the max group- -- size) in case there is no minimum.- letBindNames [tblock_size]- =<< if null ws_min- then- eBinOp- (SMin Int64)- (eSubExp =<< letSubExp "max_tblock_size" (Op $ SizeOp $ GetSizeMax SizeThreadBlock))- (eSubExp intra_avail_par)- else foldBinOp' (SMax Int64) ws_min- pure (intra_avail_par, Var tblock_size)+ -- If we have purely virtualised SegOps, then the thread block size can be set+ -- freely, but otherwise it must fit all the non-virtualised SegOps. Note that+ -- in the virtualised case, we do not cap the block size by the actual+ -- exploitable parallelism. This is to ensure that the block size is+ -- statically known when the kernel is compiled, which can result in much+ -- better code generation.+ tblock_size <-+ case onlyNonVirt ws of+ [] -> getSize "tblock_size" SizeThreadBlock+ ws' -> letSubExp "tblock_size" =<< foldBinOp' (SMax Int64) (intConst Int64 1 : ws')+ pure (intra_avail_par, Count tblock_size) -- | Check whether this result is actually something that is acceptable to use: -- the parallel dimensions must all be bound outside the kernel, as nonuniform@@ -155,13 +160,18 @@ let new_segments = segments <> pure w wss = findParallelism kbody - noNonuniformPar (freeIn wss) $ do+ noNonuniformPar (freeIn $ map fst wss) $ do ((intra_avail_par, tblock_size, kspace, num_tblocks), prelude_stms) <- collectStms $ do- num_tblocks <-- letSubExp "intra_num_tblocks"- =<< foldBinOp' (Mul Int64 OverflowUndef) new_segments- (intra_avail_par, tblock_size) <- computeThreadBlockSize wss wss+ (intra_avail_par, tblock_size) <-+ computeThreadBlockSize wss+ -- CalcNumBlocks needs a "thread count" to cap the actual number of+ -- blocks. Tell a white lie here that is good enough, hopefully.+ num_threads <-+ letSubExp "num_threads" <=< toExp $+ product $+ pe64 (unCount tblock_size) : map pe64 new_segments+ num_tblocks <- getGridSize "intrablock" num_threads tblock_size gtids <- mapM (const $ newVName "gtid") new_segments kspace <- mkSegSpace $ zip gtids new_segments pure (intra_avail_par, tblock_size, kspace, num_tblocks)@@ -176,8 +186,8 @@ let kbody' = kbody {bodyStms = read_input_stms <> bodyStms kbody} rts = map (length new_segments `stripArray`) result_ts- grid = KernelGrid (Count num_tblocks) (Count tblock_size)- lvl = SegBlock SegNoVirt (Just grid)+ grid = KernelGrid num_tblocks tblock_size+ lvl = SegBlock SegVirt (Just grid) kstm = Let nested_pat aux $ Op $ SegOp $ SegMap lvl kspace rts kbody' pure $@@ -250,10 +260,14 @@ lam' <- intrablockLambda map_in_block lam pure (shape, arrs, Just (lam', nes)) +bindPatToRes :: Pat Type -> [VName] -> FlattenM ()+bindPatToRes pat res = forM_ (zip (patNames pat) res) $ \(pe_v, v) ->+ letBindNames [pe_v] $ BasicOp $ SubExp $ Var v+ intrablockStm :: InBlockMapTransformer -> Stm SOACS -> FlattenM () intrablockStm map_in_block stm@(Let pat aux e) = do scope <- askScope- let lvl = SegThreadInBlock SegNoVirt+ let lvl = SegThreadInBlock SegVirt case e of Loop merge form loopbody ->@@ -270,28 +284,19 @@ inputs' <- mapM (intrablockWithAccInput map_in_block) inputs lam' <- intrablockLambda map_in_block lam certifying (stmAuxCerts aux) . letBind pat $ WithAcc inputs' lam'- Op soac- | "sequential_outer" `inAttrs` stmAuxAttrs aux ->- intrablockStms map_in_block . fmap (certify (stmAuxCerts aux))- =<< runBuilder_ (FOT.transformSOAC pat soac) Op (Screma w arrs form) | Just lam <- isMapSOAC form ->- map_in_block pat w arrs lam+ map_in_block lvl pat w arrs lam Op (Screma w arrs form) | Just (post_lam, scans, mapfun) <- isMaposcanomapSOAC form,- -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileGroupOp+ -- FIXME: Futhark.CodeGen.ImpGen.GPU.Block.compileBlockScan -- cannot handle multiple scan operators yet. Scan scanfun nes <- singleScan scans -> do let scanfun' = soacsLambdaToGPU scanfun mapfun' = soacsLambdaToGPU mapfun post_op = soacsLambdaToGPU post_lam- scan_res <- certifying (stmAuxCerts aux) $ genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ())- zipWithM_- ( \pe v ->- letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v- )- (patElems pat)- scan_res+ bindPatToRes pat <=< certifying (stmAuxCerts aux) $+ genUniformSegScanomapWithPost lvl (pure w) "intra_maposcanomap" scanfun' mempty nes post_op mapfun' arrs (const $ pure ()) Op (Screma w arrs form) | Just (reds, map_lam) <- isRedomapSOAC form -> do let onRed red =@@ -302,15 +307,8 @@ in Reduce comm (soacsLambdaToGPU red_lam) (redNeutral red) reds_gpu = map onRed reds map_lam' = soacsLambdaToGPU map_lam- (red_res, stms) <- runBuilder (genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ()))- certifying (stmAuxCerts aux) $ do- addStms stms- zipWithM_- ( \pe v ->- letBindNames [patElemName pe] $ BasicOp $ SubExp $ Var v- )- (patElems pat)- red_res+ bindPatToRes pat <=< certifying (stmAuxCerts aux) $+ genUniformSegRed lvl "intra_redomap" (pure w) reds_gpu mempty map_lam' arrs (const $ pure ()) Op (Screma w arrs form) -> -- This screma is too complicated for us to immediately do -- anything, so split it up and try again.@@ -318,24 +316,12 @@ =<< runBuilderT (dissectScrema pat w form arrs) (scopeForSOACs scope) Op (Hist w arrs ops bucket_fun) -> do let bucket_fun' = soacsLambdaToGPU bucket_fun-- (hist_res, stms) <- runBuilder (genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ()))- certifying (stmAuxCerts aux) $ do- addStms stms- zipWithM_- ( \pe v ->- letBindNames [patElemName pe] $- BasicOp $- SubExp $- Var v- )- (patElems pat)- hist_res+ bindPatToRes pat <=< certifying (stmAuxCerts aux) $+ genUniformSegHist lvl "Uniform_segHist" (pure w) ops bucket_fun' arrs (const $ pure ()) Op (Stream w arrs accs lam) -> do types <- asksScope castScope- ((), stream_stms) <-- runBuilderT (sequentialStreamWholeArray pat w accs lam arrs) types- intrablockStms map_in_block stream_stms+ intrablockStms map_in_block+ =<< runBuilderT_ (sequentialStreamWholeArray pat w accs lam arrs) types _ -> addStm $ soacsStmToGPU stm
src/Futhark/Pass/Flatten/PreProcess.hs view
@@ -52,7 +52,8 @@ rewriteSoacStm scope (Let pat aux (Op soac)) | "sequential_outer" `inAttrs` stmAuxAttrs aux = Just $- preprocessStms scope =<< runSimplifiedBuilder scope (FOT.transformSOAC pat soac)+ preprocessStms scope+ =<< runSimplifiedBuilder scope (certifying (stmAuxCerts aux) $ FOT.transformSOAC pat soac) rewriteSoacStm scope (Let pat aux (Op (Stream w arrs nes lam))) = Just $ do stms <- runSimplifiedBuilder scope (auxing aux $ sequentialStreamWholeArray pat w nes lam arrs) preprocessStms scope stms
src/Futhark/Pass/Flatten/SOAC.hs view
@@ -838,12 +838,13 @@ flattenMapForInBlock :: FlattenOps ->+ SegLevel -> Pat Type -> SubExp -> [VName] -> Lambda SOACS -> FlattenM ()-flattenMapForInBlock ops pat w arrs map_lam = do+flattenMapForInBlock ops lvl pat w arrs map_lam = do scope <- askScope lam <- preprocessLambda (castScope scope) map_lam let arrs' = zipWith MapArray arrs $ map paramType (lambdaParams lam)@@ -851,7 +852,7 @@ distributeMapWith ops' scope pat [w] arrs' lam transformDistributed ops' mempty [w] distributed where- ops' = atSegLevel inBlockSegLevel ops+ ops' = atSegLevel lvl ops resultMapMode :: InnerMapMode -> DistInputs -> Type -> InnerMapMode resultMapMode SingleDim _ _ = SingleDim@@ -1116,7 +1117,7 @@ transformMap ops attrs [] _env _inps pat w arrs map_lam = do -- Top-level map (no enclosing segments). Preprocess the body and then -- propagate the attributes onto it, so they influence how the body is- -- versioned (e.g. only_inner reaching a Screma produced by interchanging a+ -- versioned (e.g. only_full reaching a Screma produced by interchanging a -- 'sequential_outer' loop). Order matters: preprocessing may rewrite a body -- statement, so propagating first would lose the attributes on the rewritten -- form. XXX: this is arguably a bug in preprocessing.