diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,47 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [0.25.30] + +### Added + +* New math functions: `f16.rsqrt`, `f32.rsqrt`, `f64.rsqrt`. + +* New math functions: `cospi`, `sinpi`, `tanpi`, `acospi`, `asinpi`, + `atanpi`, `atan2pi`, in each of the `f16`/`f32`/`f64` modules. (#2243) + +* Slight improvements in the ability of the fusion engine to fuse + across `map` nests separated by `reshape` operations. Only works if + the innermost return type is purely scalar. + +* `futhark pkg` now allows underscores in package paths. + +### Fixed + +* The interpreter no longer crashes when passing a sum-typed value + into AD, but it is unlikely to produce a usable result (#2238). + +* The partial derivatives of comparisons are now always zero. + Previously we had some code that made an attempt at giving these + another interpretation, but it was never mathematically sound, not + useful, and sometimes buggy. (#2239). + +* Out-of-bounds reads in GPU backends when transposing a great many + matrices in parallel (#2241). + +* `vjp` in the interpreter is now asymptotically efficient (#2187, + #2240). Work by Marcus Jensen. + +* The interpreter did not handle `open` correctly. + +* Incorrect handling of some size inference edge cases during + monomorphisation (#2252). + +* Incorrect registration of entry point types when mixing type + abbreviations and arrays (#2253). + +* Reverse mode AD now handles sequential streams. (#2256) + ## [0.25.29] ### Fixed @@ -29,6 +70,10 @@ ways (#2234). * Fusion could crash after AD in some circumstances (#2236). + +* Under very unlikely circumstances, an "intrablock" flat parallel + operation could be hoisted out of its enclosing GPU kernel, causing + a compiler crash. ## [0.25.28] diff --git a/docs/error-index.rst b/docs/error-index.rst --- a/docs/error-index.rst +++ b/docs/error-index.rst @@ -367,6 +367,28 @@ Here the type rules force ``A`` to have size ``x``, leading to a problematic type. It can be fixed using the techniques above. +.. _aliases-previously-returned: + +"Return value for consuming loop parameter *x* aliases previously returned value" +--------------------------------------------------------------------------------- + +This error occurs when you have a loop with multiple loop parameters, +at least one of which is consuming, and the values returned by the +loop body alias each other. This would result in the consuming loop +parameter aliasing another loop parameter, which is not allowed. It is +essentially :ref:`unique-return-aliased` from a loop perspective. + +A (contrived) example of this error is the following: + +.. code-block:: futhark + + loop (acc: []f64, arr: *[][]f64) for i < length arr-1 do + let arr[i] = acc + -- Error, because 'arr[i]' and 'arr' are aliased, yet the latter + -- is consumed. + in (arr[i+1], arr) + + Size errors ----------- diff --git a/docs/language-reference.rst b/docs/language-reference.rst --- a/docs/language-reference.rst +++ b/docs/language-reference.rst @@ -370,15 +370,21 @@ ~~~~~~~~~~~~ Apart from declaring a function with the keyword ``def``, it can also -be declared with ``entry``. When the Futhark program is compiled any -top-level function declared with ``entry`` will be exposed as an entry -point. If the Futhark program has been compiled as a library, these -are the functions that will be exposed. If compiled as an executable, -you can use the ``--entry-point`` command line option of the generated -executable to select the entry point you wish to run. +be declared with ``entry``. When the Futhark program is compiled any +top-level function declared with ``entry`` *in the single file passed +directly to the Futhark compiler* will be exposed as an entry point. +This means that any functions defined with ``entry`` in a file that is +accessed via ``import`` are not considered entry points, but they are +still usable as normal functions. -Any top-level function named ``main`` will always be considered an -entry point, whether it is declared with ``entry`` or not. +If the Futhark program has been compiled as a library, these entry +points are the functions that will be exposed as the library +interface. If compiled as an executable, you can use the +``--entry-point`` command line option of the generated executable to +select the entry point you wish to run. + +Any top-level function named ``main`` is treated as if it had been +defined with ``entry``. The name of an entry point must not contain an apostrophe (``'``), even though that is normally permitted in Futhark identifiers. diff --git a/futhark.cabal b/futhark.cabal --- a/futhark.cabal +++ b/futhark.cabal @@ -1,6 +1,6 @@ cabal-version: 2.4 name: futhark -version: 0.25.29 +version: 0.25.30 synopsis: An optimising compiler for a functional, array-oriented language. description: Futhark is a small programming language designed to be compiled to diff --git a/prelude/ad.fut b/prelude/ad.fut --- a/prelude/ad.fut +++ b/prelude/ad.fut @@ -55,23 +55,23 @@ -- Both `jvp` and `vjp` work by transforming the program to carry -- along extra information associated with each scalar value. -- --- In the case of `vjp`, this extra information takes the form of an +-- In the case of `jvp`, this extra information takes the form of an -- additional scalar representing the tangent, which is then -- propagated in each scalar computation using essentially the [chain -- rule](https://en.wikipedia.org/wiki/Chain_rule). Therefore, `jvp` -- has a memory overhead of approximately *2x*, and a computational -- overhead of slightly more, but usually less than *4x*. -- --- In the case of `jvp`, since our starting point is a *cotangent*, +-- In the case of `vjp`, since our starting point is a *cotangent*, -- the function is essentially first run forward, then backwards (the -- *return sweep*) to propagate the cotangent. During the return -- sweep, all intermediate results computed during the forward sweep -- must still be available, and must therefore be stored in memory --- during the forward sweep. This means that the memory usage of `jvp` --- is essentially proportional to the number of sequential steps of --- the original function (essentially turning *time* into *space*). --- The compiler does a nontrivial amount of optimisation to ameliorate --- this overhead (see [AD for an Array Language with Nested +-- during the forward sweep. This means that the memory usage of `vjp` +-- is proportional to the number of sequential steps of the original +-- function (essentially turning *time* into *space*). The compiler +-- does a nontrivial amount of optimisation to ameliorate this +-- overhead (see [AD for an Array Language with Nested -- Parallelism](https://futhark-lang.org/publications/sc22-ad.pdf)), -- but it can still be substantial for programs with deep sequential -- loops. @@ -85,6 +85,11 @@ -- primitive floating-point operations, without ever converting to or -- from integers. -- +-- Note that a function whose input or output is a sum type with more +-- than one constructor is *not* differentiable (or at least the +-- sum-typed part is not). This is because the choice of constructor +-- is not a continuous quantity. +-- -- ## Limitations -- -- `jvp` is expected to work in all cases. `vjp` has limitations when @@ -95,18 +100,18 @@ -- | Jacobian-Vector Product ("forward mode"), producing also the -- primal result as the first element of the result tuple. -def jvp2 'a 'b (f: a -> b) (x: a) (x': a): (b, b) = +def jvp2 'a 'b (f: a -> b) (x: a) (x': a) : (b, b) = intrinsics.jvp2 f x x' -- | Vector-Jacobian Product ("reverse mode"), producing also the -- primal result as the first element of the result tuple. -def vjp2 'a 'b (f: a -> b) (x: a) (y': b): (b, a) = +def vjp2 'a 'b (f: a -> b) (x: a) (y': b) : (b, a) = intrinsics.vjp2 f x y' -- | Jacobian-Vector Product ("forward mode"). -def jvp 'a 'b (f: a -> b) (x: a) (x': a): b = +def jvp 'a 'b (f: a -> b) (x: a) (x': a) : b = (jvp2 f x x').1 -- | Vector-Jacobian Product ("reverse mode"). -def vjp 'a 'b (f: a -> b) (x: a) (y': b): a = +def vjp 'a 'b (f: a -> b) (x: a) (y': b) : a = (vjp2 f x y').1 diff --git a/prelude/array.fut b/prelude/array.fut --- a/prelude/array.fut +++ b/prelude/array.fut @@ -3,7 +3,7 @@ import "math" import "soacs" import "functional" -open import "zip" -- Rexport. +open import "zip" -- | The size of the outer dimension of an array. -- diff --git a/prelude/functional.fut b/prelude/functional.fut --- a/prelude/functional.fut +++ b/prelude/functional.fut @@ -6,7 +6,7 @@ -- ``` -- x |> f |> g |> h -- ``` -def (|>) '^a '^b (x: a) (f: a -> b): b = f x +def (|>) '^a '^b (x: a) (f: a -> b) : b = f x -- | Right to left application. -- @@ -35,35 +35,35 @@ -- ``` -- -- In such cases you can use the pipe operator `|>`@term instead. -def (>->) '^a '^b '^c (f: a -> b) (g: b -> c) (x: a): c = g (f x) +def (>->) '^a '^b '^c (f: a -> b) (g: b -> c) (x: a) : c = g (f x) -- | Function composition, with values flowing from right to left. -- This is the same as the `∘` operator known from mathematics. -- -- Has the same restrictions with respect to anonymous sizes as -- `>->`@term. -def (<-<) '^a '^b '^c (g: b -> c) (f: a -> b) (x: a): c = g (f x) +def (<-<) '^a '^b '^c (g: b -> c) (f: a -> b) (x: a) : c = g (f x) -- | Flip the arguments passed to a function. -- -- ``` -- f x y == flip f y x -- ``` -def flip '^a '^b '^c (f: a -> b -> c) (b: b) (a: a): c = +def flip '^a '^b '^c (f: a -> b -> c) (b: b) (a: a) : c = f a b -- | Transform a function taking a pair into a function taking two -- arguments. -def curry '^a '^b '^c (f: (a, b) -> c) (a: a) (b: b): c = +def curry '^a '^b '^c (f: (a, b) -> c) (a: a) (b: b) : c = f (a, b) -- | Transform a function taking two arguments in a function taking a -- pair. -def uncurry '^a '^b '^c (f: a -> b -> c) (a: a, b: b): c = +def uncurry '^a '^b '^c (f: a -> b -> c) (a: a, b: b) : c = f a b -- | The constant function. -def const '^a '^b (x: a) (_: b): a = x +def const '^a '^b (x: a) (_: b) : a = x -- | The identity function. def id '^a (x: a) = x diff --git a/prelude/math.fut b/prelude/math.fut --- a/prelude/math.fut +++ b/prelude/math.fut @@ -7,21 +7,21 @@ module type from_prim = { type t - val i8: i8 -> t - val i16: i16 -> t - val i32: i32 -> t - val i64: i64 -> t + val i8 : i8 -> t + val i16 : i16 -> t + val i32 : i32 -> t + val i64 : i64 -> t - val u8: u8 -> t - val u16: u16 -> t - val u32: u32 -> t - val u64: u64 -> t + val u8 : u8 -> t + val u16 : u16 -> t + val u32 : u32 -> t + val u64 : u64 -> t - val f16: f16 -> t - val f32: f32 -> t - val f64: f64 -> t + val f16 : f16 -> t + val f32 : f32 -> t + val f64 : f64 -> t - val bool: bool -> t + val bool : bool -> t } -- | A basic numeric module type that can be implemented for both @@ -29,51 +29,51 @@ module type numeric = { include from_prim - val +: t -> t -> t - val -: t -> t -> t - val *: t -> t -> t - val /: t -> t -> t - val %: t -> t -> t - val **: t -> t -> t + val + : t -> t -> t + val - : t -> t -> t + val * : t -> t -> t + val / : t -> t -> t + val % : t -> t -> t + val ** : t -> t -> t - val to_i64: t -> i64 + val to_i64 : t -> i64 - val ==: t -> t -> bool - val <: t -> t -> bool - val >: t -> t -> bool - val <=: t -> t -> bool - val >=: t -> t -> bool - val !=: t -> t -> bool + val == : t -> t -> bool + val < : t -> t -> bool + val > : t -> t -> bool + val <= : t -> t -> bool + val >= : t -> t -> bool + val != : t -> t -> bool -- | Arithmetic negation (use `!` for bitwise negation). - val neg: t -> t - val max: t -> t -> t - val min: t -> t -> t + val neg : t -> t + val max : t -> t -> t + val min : t -> t -> t - val abs: t -> t + val abs : t -> t -- | Sign function. Produces -1, 0, or 1 if the argument is -- respectively less than, equal to, or greater than zero. - val sgn: t -> t + val sgn : t -> t -- | The most positive representable number. - val highest: t + val highest : t -- | The least positive representable number (most negative for -- signed types). - val lowest: t + val lowest : t -- | Returns zero on empty input. - val sum [n]: [n]t -> t + val sum [n] : [n]t -> t -- | Returns one on empty input. - val product [n]: [n]t -> t + val product [n] : [n]t -> t -- | Returns `lowest` on empty input. - val maximum [n]: [n]t -> t + val maximum [n] : [n]t -> t -- | Returns `highest` on empty input. - val minimum [n]: [n]t -> t + val minimum [n] : [n]t -> t } -- | An extension of `numeric`@mtype that provides facilities that are @@ -83,57 +83,57 @@ -- | Like `/`@term, but rounds towards zero. This only matters when -- one of the operands is negative. May be more efficient. - val //: t -> t -> t + val // : t -> t -> t -- | Like `%`@term, but rounds towards zero. This only matters when -- one of the operands is negative. May be more efficient. - val %%: t -> t -> t + val %% : t -> t -> t -- | Bitwise and. - val &: t -> t -> t + val & : t -> t -> t -- | Bitwise or. - val |: t -> t -> t + val | : t -> t -> t -- | Bitwise xor. - val ^: t -> t -> t + val ^ : t -> t -> t -- | Bitwise negation. - val not: t -> t + val not : t -> t -- | Left shift; inserting zeroes. - val <<: t -> t -> t + val << : t -> t -> t -- | Arithmetic right shift, using sign extension for the leftmost bits. - val >>: t -> t -> t + val >> : t -> t -> t -- | Logical right shift, inserting zeroes for the leftmost bits. - val >>>: t -> t -> t + val >>> : t -> t -> t - val num_bits: i32 - val get_bit: i32 -> t -> i32 - val set_bit: i32 -> t -> i32 -> t + val num_bits : i32 + val get_bit : i32 -> t -> i32 + val set_bit : i32 -> t -> i32 -> t -- | Count number of one bits. - val popc: t -> i32 + val popc : t -> i32 -- | Computes `x * y` and returns the high half of the product of x -- and y. - val mul_hi: (x: t) -> (y: t) -> t + val mul_hi : (x: t) -> (y: t) -> t -- | Computes `mul_hi a b + c`, but perhaps in a more efficient way, -- depending on the target platform. - val mad_hi: (a: t) -> (b: t) -> (c: t) -> t + val mad_hi : (a: t) -> (b: t) -> (c: t) -> t -- | Count number of zero bits preceding the most significant set -- bit. Returns the number of bits in the type if the argument is -- zero. - val clz: t -> i32 + val clz : t -> i32 -- | Count number of trailing zero bits following the least -- significant set bit. Returns the number of bits in the type if -- the argument is zero. - val ctz: t -> i32 + val ctz : t -> i32 } -- | Numbers that model real numbers to some degree. @@ -141,105 +141,126 @@ include numeric -- | Multiplicative inverse. - val recip: t -> t + val recip : t -> t - val from_fraction: i64 -> i64 -> t - val to_i64: t -> i64 - val to_f64: t -> f64 + val from_fraction : i64 -> i64 -> t + val to_i64 : t -> i64 + val to_f64 : t -> f64 -- | Square root. - val sqrt: t -> t + val sqrt : t -> t + -- | Inverse square root. Depending on the backend, this may be + -- faster than `1/sqrt(x)`. + val rsqrt : t -> t + -- | Cube root. - val cbrt: t -> t - val exp: t -> t + val cbrt : t -> t + val exp : t -> t - val sin: t -> t - val cos: t -> t - val tan: t -> t + val sin : t -> t + val cos : t -> t + val tan : t -> t - val asin: t -> t - val acos: t -> t - val atan: t -> t + -- | `sin(pi*x)` - depending on backing, may be faster or more + -- accurate. + val sinpi : t -> t - val sinh: t -> t - val cosh: t -> t - val tanh: t -> t + -- | `cos(pi*x)` - depending on backing, may be faster or more + -- accurate. + val cospi : t -> t - val asinh: t -> t - val acosh: t -> t - val atanh: t -> t + -- | `tan(pi*x)` - depending on backing, may be faster or more + -- accurate. + val tanpi : t -> t - val atan2: t -> t -> t + val asin : t -> t + val acos : t -> t + val atan : t -> t + val asinpi : t -> t + val acospi : t -> t + val atanpi : t -> t + + val sinh : t -> t + val cosh : t -> t + val tanh : t -> t + + val asinh : t -> t + val acosh : t -> t + val atanh : t -> t + + val atan2 : t -> t -> t + val atan2pi : t -> t -> t + -- | Compute the length of the hypotenuse of a right-angled -- triangle. That is, `hypot x y` computes *√(x²+y²)*. Put another -- way, the distance of *(x,y)* from origin in an Euclidean space. -- The calculation is performed without undue overflow or underflow -- during intermediate steps (specific accuracy depends on the -- backend). - val hypot: t -> t -> t + val hypot : t -> t -> t -- | The true Gamma function. - val gamma: t -> t + val gamma : t -> t -- | The natural logarithm of the absolute value of `gamma`@term. - val lgamma: t -> t + val lgamma : t -> t -- | The error function. - val erf: t -> t + val erf : t -> t -- | The complementary error function. - val erfc: t -> t + val erfc : t -> t -- | Linear interpolation. The third argument must be in the range -- `[0,1]` or the results are unspecified. - val lerp: t -> t -> t -> t + val lerp : t -> t -> t -> t -- | Natural logarithm. - val log: t -> t + val log : t -> t -- | Base-2 logarithm. - val log2: t -> t + val log2 : t -> t -- | Base-10 logarithm. - val log10: t -> t + val log10 : t -> t -- | Compute `log (1 + x)` accurately even when `x` is very small. - val log1p: t -> t + val log1p : t -> t -- | Round towards infinity. - val ceil: t -> t + val ceil : t -> t -- | Round towards negative infinity. - val floor: t -> t + val floor : t -> t -- | Round towards zero. - val trunc: t -> t + val trunc : t -> t -- | Round to the nearest integer, with halfway cases rounded to the -- nearest even integer. Note that this differs from `round()` in -- C, but matches more modern languages. - val round: t -> t + val round : t -> t -- | Computes `a*b+c`. Depending on the compiler backend, this may -- be fused into a single operation that is faster but less -- accurate. Do not confuse it with `fma`@term. - val mad: (a: t) -> (b: t) -> (c: t) -> t + val mad : (a: t) -> (b: t) -> (c: t) -> t -- | Computes `a*b+c`, with `a*b` being rounded with infinite -- precision. Rounding of intermediate products shall not -- occur. Edge case behavior is per the IEEE 754-2008 standard. - val fma: (a: t) -> (b: t) -> (c: t) -> t + val fma : (a: t) -> (b: t) -> (c: t) -> t - val isinf: t -> bool - val isnan: t -> bool + val isinf : t -> bool + val isnan : t -> bool - val inf: t - val nan: t + val inf : t + val nan : t - val pi: t - val e: t + val pi : t + val e : t } -- | An extension of `real`@mtype that further gives access to the @@ -255,31 +276,31 @@ -- 't'. type int_t - val from_bits: int_t -> t - val to_bits: t -> int_t + val from_bits : int_t -> t + val to_bits : t -> int_t - val num_bits: i32 - val get_bit: i32 -> t -> i32 - val set_bit: i32 -> t -> i32 -> t + val num_bits : i32 + val get_bit : i32 -> t -> i32 + val set_bit : i32 -> t -> i32 -> t -- | The difference between 1.0 and the next larger representable -- number. - val epsilon: t + val epsilon : t -- | Produces the next representable number from `x` in the -- direction of `y`. - val nextafter: (x: t) -> (y: t) -> t + val nextafter : (x: t) -> (y: t) -> t -- | Multiplies floating-point value by 2 raised to an integer power. - val ldexp: t -> i32 -> t + val ldexp : t -> i32 -> t -- | Compose a floating-point value with the magnitude of `x` and the sign of `y`. - val copysign: (x: t) -> (y: t) -> t + val copysign : (x: t) -> (y: t) -> t } -- | Boolean numbers. When converting from a number to `bool`, 0 is -- considered `false` and any other value is `true`. -module bool: from_prim with t = bool = { +module bool : from_prim with t = bool = { type t = bool def i8 = intrinsics.itob_i8_bool @@ -299,7 +320,7 @@ def bool (x: bool) = x } -module i8: (integral with t = i8) = { +module i8 : (integral with t = i8) = { type t = i8 def (+) (x: i8) (y: i8) = intrinsics.add8 (x, y) @@ -374,7 +395,7 @@ def minimum = reduce min highest } -module i16: (integral with t = i16) = { +module i16 : (integral with t = i16) = { type t = i16 def (+) (x: i16) (y: i16) = intrinsics.add16 (x, y) @@ -449,7 +470,7 @@ def minimum = reduce min highest } -module i32: (integral with t = i32) = { +module i32 : (integral with t = i32) = { type t = i32 def sign (x: u32) = intrinsics.sign_i32 x @@ -527,7 +548,7 @@ def minimum = reduce min highest } -module i64: (integral with t = i64) = { +module i64 : (integral with t = i64) = { type t = i64 def sign (x: u64) = intrinsics.sign_i64 x @@ -605,7 +626,7 @@ def minimum = reduce min highest } -module u8: (integral with t = u8) = { +module u8 : (integral with t = u8) = { type t = u8 def sign (x: u8) = intrinsics.sign_i8 x @@ -683,7 +704,7 @@ def minimum = reduce min highest } -module u16: (integral with t = u16) = { +module u16 : (integral with t = u16) = { type t = u16 def sign (x: u16) = intrinsics.sign_i16 x @@ -761,7 +782,7 @@ def minimum = reduce min highest } -module u32: (integral with t = u32) = { +module u32 : (integral with t = u32) = { type t = u32 def sign (x: u32) = intrinsics.sign_i32 x @@ -839,7 +860,7 @@ def minimum = reduce min highest } -module u64: (integral with t = u64) = { +module u64 : (integral with t = u64) = { type t = u64 def sign (x: u64) = intrinsics.sign_i64 x @@ -917,7 +938,7 @@ def minimum = reduce min highest } -module f64: (float with t = f64 with int_t = u64) = { +module f64 : (float with t = f64 with int_t = u64) = { type t = f64 type int_t = u64 @@ -967,6 +988,7 @@ def abs (x: f64) = intrinsics.fabs64 x def sqrt (x: f64) = intrinsics.sqrt64 x + def rsqrt (x: f64) = intrinsics.rsqrt64 x def cbrt (x: f64) = intrinsics.cbrt64 x def log (x: f64) = intrinsics.log64 x @@ -977,9 +999,15 @@ def sin (x: f64) = intrinsics.sin64 x def cos (x: f64) = intrinsics.cos64 x def tan (x: f64) = intrinsics.tan64 x + def sinpi (x: f64) = intrinsics.sinpi64 x + def cospi (x: f64) = intrinsics.cospi64 x + def tanpi (x: f64) = intrinsics.tanpi64 x def acos (x: f64) = intrinsics.acos64 x def asin (x: f64) = intrinsics.asin64 x def atan (x: f64) = intrinsics.atan64 x + def acospi (x: f64) = intrinsics.acospi64 x + def asinpi (x: f64) = intrinsics.asinpi64 x + def atanpi (x: f64) = intrinsics.atanpi64 x def sinh (x: f64) = intrinsics.sinh64 x def cosh (x: f64) = intrinsics.cosh64 x def tanh (x: f64) = intrinsics.tanh64 x @@ -987,6 +1015,7 @@ def asinh (x: f64) = intrinsics.asinh64 x def atanh (x: f64) = intrinsics.atanh64 x def atan2 (x: f64) (y: f64) = intrinsics.atan2_64 (x, y) + def atan2pi (x: f64) (y: f64) = intrinsics.atan2pi_64 (x, y) def hypot (x: f64) (y: f64) = intrinsics.hypot64 (x, y) def gamma = intrinsics.gamma64 def lgamma = intrinsics.lgamma64 @@ -999,7 +1028,7 @@ def ceil = intrinsics.ceil64 def floor = intrinsics.floor64 - def trunc (x: f64): f64 = i64 (i64m.f64 x) + def trunc (x: f64) : f64 = i64 (i64m.f64 x) def round = intrinsics.round64 @@ -1007,8 +1036,8 @@ def ldexp x y = intrinsics.ldexp64 (x, y) def copysign x y = intrinsics.copysign64 (x, y) - def to_bits (x: f64): u64 = u64m.i64 (intrinsics.to_bits64 x) - def from_bits (x: u64): f64 = intrinsics.from_bits64 (intrinsics.sign_i64 x) + def to_bits (x: f64) : u64 = u64m.i64 (intrinsics.to_bits64 x) + def from_bits (x: u64) : f64 = intrinsics.from_bits64 (intrinsics.sign_i64 x) def num_bits = 64i32 def get_bit (bit: i32) (x: t) = u64m.get_bit bit (to_bits x) @@ -1033,7 +1062,7 @@ def minimum = reduce min highest } -module f32: (float with t = f32 with int_t = u32) = { +module f32 : (float with t = f32 with int_t = u32) = { type t = f32 type int_t = u32 @@ -1084,6 +1113,7 @@ def abs (x: f32) = intrinsics.fabs32 x def sqrt (x: f32) = intrinsics.sqrt32 x + def rsqrt (x: f32) = intrinsics.rsqrt32 x def cbrt (x: f32) = intrinsics.cbrt32 x def log (x: f32) = intrinsics.log32 x @@ -1094,9 +1124,15 @@ def sin (x: f32) = intrinsics.sin32 x def cos (x: f32) = intrinsics.cos32 x def tan (x: f32) = intrinsics.tan32 x + def sinpi (x: f32) = intrinsics.sinpi32 x + def cospi (x: f32) = intrinsics.cospi32 x + def tanpi (x: f32) = intrinsics.tanpi32 x def acos (x: f32) = intrinsics.acos32 x def asin (x: f32) = intrinsics.asin32 x def atan (x: f32) = intrinsics.atan32 x + def acospi (x: f32) = intrinsics.acospi32 x + def asinpi (x: f32) = intrinsics.asinpi32 x + def atanpi (x: f32) = intrinsics.atanpi32 x def sinh (x: f32) = intrinsics.sinh32 x def cosh (x: f32) = intrinsics.cosh32 x def tanh (x: f32) = intrinsics.tanh32 x @@ -1104,6 +1140,7 @@ def asinh (x: f32) = intrinsics.asinh32 x def atanh (x: f32) = intrinsics.atanh32 x def atan2 (x: f32) (y: f32) = intrinsics.atan2_32 (x, y) + def atan2pi (x: f32) (y: f32) = intrinsics.atan2pi_32 (x, y) def hypot (x: f32) (y: f32) = intrinsics.hypot32 (x, y) def gamma = intrinsics.gamma32 def lgamma = intrinsics.lgamma32 @@ -1116,7 +1153,7 @@ def ceil = intrinsics.ceil32 def floor = intrinsics.floor32 - def trunc (x: f32): f32 = i32 (i32m.f32 x) + def trunc (x: f32) : f32 = i32 (i32m.f32 x) def round = intrinsics.round32 @@ -1124,8 +1161,8 @@ def ldexp x y = intrinsics.ldexp32 (x, y) def copysign x y = intrinsics.copysign32 (x, y) - def to_bits (x: f32): u32 = u32m.i32 (intrinsics.to_bits32 x) - def from_bits (x: u32): f32 = intrinsics.from_bits32 (intrinsics.sign_i32 x) + def to_bits (x: f32) : u32 = u32m.i32 (intrinsics.to_bits32 x) + def from_bits (x: u32) : f32 = intrinsics.from_bits32 (intrinsics.sign_i32 x) def num_bits = 32i32 def get_bit (bit: i32) (x: t) = u32m.get_bit bit (to_bits x) @@ -1154,7 +1191,7 @@ -- support half precision. This means you might get more accurate -- results than on real systems, but it is also likely to be -- significantly slower than just using `f32` in the first place. -module f16: (float with t = f16 with int_t = u16) = { +module f16 : (float with t = f16 with int_t = u16) = { type t = f16 type int_t = u16 @@ -1205,6 +1242,7 @@ def abs (x: f16) = intrinsics.fabs16 x def sqrt (x: f16) = intrinsics.sqrt16 x + def rsqrt (x: f16) = intrinsics.rsqrt16 x def cbrt (x: f16) = intrinsics.cbrt16 x def log (x: f16) = intrinsics.log16 x @@ -1215,9 +1253,15 @@ def sin (x: f16) = intrinsics.sin16 x def cos (x: f16) = intrinsics.cos16 x def tan (x: f16) = intrinsics.tan16 x + def sinpi (x: f16) = intrinsics.sinpi16 x + def cospi (x: f16) = intrinsics.cospi16 x + def tanpi (x: f16) = intrinsics.tanpi16 x def acos (x: f16) = intrinsics.acos16 x def asin (x: f16) = intrinsics.asin16 x def atan (x: f16) = intrinsics.atan16 x + def acospi (x: f16) = intrinsics.acospi16 x + def asinpi (x: f16) = intrinsics.asinpi16 x + def atanpi (x: f16) = intrinsics.atanpi16 x def sinh (x: f16) = intrinsics.sinh16 x def cosh (x: f16) = intrinsics.cosh16 x def tanh (x: f16) = intrinsics.tanh16 x @@ -1225,6 +1269,7 @@ def asinh (x: f16) = intrinsics.asinh16 x def atanh (x: f16) = intrinsics.atanh16 x def atan2 (x: f16) (y: f16) = intrinsics.atan2_16 (x, y) + def atan2pi (x: f16) (y: f16) = intrinsics.atan2pi_16 (x, y) def hypot (x: f16) (y: f16) = intrinsics.hypot16 (x, y) def gamma = intrinsics.gamma16 def lgamma = intrinsics.lgamma16 @@ -1237,7 +1282,7 @@ def ceil = intrinsics.ceil16 def floor = intrinsics.floor16 - def trunc (x: f16): f16 = i16 (i16m.f16 x) + def trunc (x: f16) : f16 = i16 (i16m.f16 x) def round = intrinsics.round16 @@ -1245,8 +1290,8 @@ def ldexp x y = intrinsics.ldexp16 (x, y) def copysign x y = intrinsics.copysign16 (x, y) - def to_bits (x: f16): u16 = u16m.i16 (intrinsics.to_bits16 x) - def from_bits (x: u16): f16 = intrinsics.from_bits16 (intrinsics.sign_i16 x) + def to_bits (x: f16) : u16 = u16m.i16 (intrinsics.to_bits16 x) + def from_bits (x: u16) : f16 = intrinsics.from_bits16 (intrinsics.sign_i16 x) def num_bits = 16i32 def get_bit (bit: i32) (x: t) = u16m.get_bit bit (to_bits x) diff --git a/prelude/prelude.fut b/prelude/prelude.fut --- a/prelude/prelude.fut +++ b/prelude/prelude.fut @@ -8,33 +8,35 @@ open import "ad" -- | Create single-precision float from integer. -def r32 (x: i32): f32 = f32.i32 x +def r32 (x: i32) : f32 = f32.i32 x + -- | Create integer from single-precision float. -def t32 (x: f32): i32 = i32.f32 x +def t32 (x: f32) : i32 = i32.f32 x -- | Create double-precision float from integer. -def r64 (x: i32): f64 = f64.i32 x +def r64 (x: i32) : f64 = f64.i32 x + -- | Create integer from double-precision float. -def t64 (x: f64): i32 = i32.f64 x +def t64 (x: f64) : i32 = i32.f64 x -- | Negate a boolean. `not x` is the same as `!x`. This function is -- mostly useful for passing to `map`. -def not (x: bool): bool = !x +def not (x: bool) : bool = !x -- | Semantically just identity, but serves as an optimisation -- inhibitor. The compiler will treat this function as a black box. -- You can use this to work around optimisation deficiencies (or -- bugs), although it should hopefully rarely be necessary. -- Deprecated: use `#[opaque]` attribute instead. -def opaque 't (x: t): t = +def opaque 't (x: t) : t = #[opaque] x -- | Semantically just identity, but at runtime, the argument value -- will be printed. Deprecated: use `#[trace]` attribute instead. -def trace 't (x: t): t = +def trace 't (x: t) : t = #[trace(trace)] x -- | Semantically just identity, but acts as a break point in -- `futhark repl`. Deprecated: use `#[break]` attribute instead. -def break 't (x: t): t = +def break 't (x: t) : t = #[break] x diff --git a/prelude/zip.fut b/prelude/zip.fut --- a/prelude/zip.fut +++ b/prelude/zip.fut @@ -9,7 +9,8 @@ -- We need a map to define some of the zip variants, but this file is -- depended upon by soacs.fut. So we just define a quick-and-dirty -- internal one here that uses the intrinsic version. -local def internal_map 'a [n] 'x (f: a -> x) (as: [n]a) : *[n]x = +local +def internal_map 'a [n] 'x (f: a -> x) (as: [n]a) : *[n]x = intrinsics.map f as -- | Construct an array of pairs from two arrays. diff --git a/rts/c/atomics.h b/rts/c/atomics.h --- a/rts/c/atomics.h +++ b/rts/c/atomics.h @@ -266,7 +266,7 @@ SCALAR_FUN_ATTR int64_t atomic_xchg_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicExch((uint64_t*)p, x); + return atomicExch((unsigned long long*)p, x); #else return atom_xor(p, x); #endif @@ -274,7 +274,7 @@ SCALAR_FUN_ATTR int64_t atomic_xchg_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicExch((uint64_t*)p, x); + return atomicExch((unsigned long long*)p, x); #else return atom_xor(p, x); #endif @@ -283,7 +283,7 @@ SCALAR_FUN_ATTR int64_t atomic_cmpxchg_i64_global(volatile __global int64_t *p, int64_t cmp, int64_t val) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicCAS((uint64_t*)p, cmp, val); + return atomicCAS((unsigned long long*)p, cmp, val); #else return atom_cmpxchg(p, cmp, val); #endif @@ -292,7 +292,7 @@ SCALAR_FUN_ATTR int64_t atomic_cmpxchg_i64_shared(volatile __local int64_t *p, int64_t cmp, int64_t val) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicCAS((uint64_t*)p, cmp, val); + return atomicCAS((unsigned long long*)p, cmp, val); #else return atom_cmpxchg(p, cmp, val); #endif @@ -300,7 +300,7 @@ SCALAR_FUN_ATTR int64_t atomic_add_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicAdd((uint64_t*)p, x); + return atomicAdd((unsigned long long*)p, x); #else return atom_add(p, x); #endif @@ -308,7 +308,7 @@ SCALAR_FUN_ATTR int64_t atomic_add_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicAdd((uint64_t*)p, x); + return atomicAdd((unsigned long long*)p, x); #else return atom_add(p, x); #endif @@ -368,7 +368,7 @@ SCALAR_FUN_ATTR int64_t atomic_smax_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) - return atomicMax((int64_t*)p, x); + return atomicMax((long long*)p, x); #elif defined(FUTHARK_HIP) // Currentely missing in HIP; probably a temporary oversight. int64_t old = *p, assumed; @@ -385,7 +385,7 @@ SCALAR_FUN_ATTR int64_t atomic_smax_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) - return atomicMax((int64_t*)p, x); + return atomicMax((long long*)p, x); #elif defined(FUTHARK_HIP) // Currentely missing in HIP; probably a temporary oversight. int64_t old = *p, assumed; @@ -402,7 +402,7 @@ SCALAR_FUN_ATTR int64_t atomic_smin_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) - return atomicMin((int64_t*)p, x); + return atomicMin((long long*)p, x); #elif defined(FUTHARK_HIP) // Currentely missing in HIP; probably a temporary oversight. int64_t old = *p, assumed; @@ -419,7 +419,7 @@ SCALAR_FUN_ATTR int64_t atomic_smin_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) - return atomicMin((int64_t*)p, x); + return atomicMin((long long*)p, x); #elif defined(FUTHARK_HIP) // Currentely missing in HIP; probably a temporary oversight. int64_t old = *p, assumed; @@ -436,7 +436,7 @@ SCALAR_FUN_ATTR uint64_t atomic_umax_i64_global(volatile __global uint64_t *p, uint64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicMax((uint64_t*)p, x); + return atomicMax((unsigned long long*)p, x); #else return atom_max(p, x); #endif @@ -444,7 +444,7 @@ SCALAR_FUN_ATTR uint64_t atomic_umax_i64_shared(volatile __local uint64_t *p, uint64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicMax((uint64_t*)p, x); + return atomicMax((unsigned long long*)p, x); #else return atom_max(p, x); #endif @@ -452,7 +452,7 @@ SCALAR_FUN_ATTR uint64_t atomic_umin_i64_global(volatile __global uint64_t *p, uint64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicMin((uint64_t*)p, x); + return atomicMin((unsigned long long*)p, x); #else return atom_min(p, x); #endif @@ -460,7 +460,7 @@ SCALAR_FUN_ATTR uint64_t atomic_umin_i64_shared(volatile __local uint64_t *p, uint64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicMin((uint64_t*)p, x); + return atomicMin((unsigned long long*)p, x); #else return atom_min(p, x); #endif @@ -468,7 +468,7 @@ SCALAR_FUN_ATTR int64_t atomic_and_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicAnd((uint64_t*)p, x); + return atomicAnd((unsigned long long*)p, x); #else return atom_and(p, x); #endif @@ -476,7 +476,7 @@ SCALAR_FUN_ATTR int64_t atomic_and_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicAnd((uint64_t*)p, x); + return atomicAnd((unsigned long long*)p, x); #else return atom_and(p, x); #endif @@ -484,7 +484,7 @@ SCALAR_FUN_ATTR int64_t atomic_or_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicOr((uint64_t*)p, x); + return atomicOr((unsigned long long*)p, x); #else return atom_or(p, x); #endif @@ -492,7 +492,7 @@ SCALAR_FUN_ATTR int64_t atomic_or_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicOr((uint64_t*)p, x); + return atomicOr((unsigned long long*)p, x); #else return atom_or(p, x); #endif @@ -500,7 +500,7 @@ SCALAR_FUN_ATTR int64_t atomic_xor_i64_global(volatile __global int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicXor((uint64_t*)p, x); + return atomicXor((unsigned long long*)p, x); #else return atom_xor(p, x); #endif @@ -508,7 +508,7 @@ SCALAR_FUN_ATTR int64_t atomic_xor_i64_shared(volatile __local int64_t *p, int64_t x) { #if defined(FUTHARK_CUDA) || defined(FUTHARK_HIP) - return atomicXor((uint64_t*)p, x); + return atomicXor((unsigned long long*)p, x); #else return atom_xor(p, x); #endif diff --git a/rts/c/scalar.h b/rts/c/scalar.h --- a/rts/c/scalar.h +++ b/rts/c/scalar.h @@ -17,3032 +17,1851 @@ // Double-precision definitions are only included if the preprocessor // macro FUTHARK_F64_ENABLED is set. -SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x); -SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x); - -SCALAR_FUN_ATTR uint8_t add8(uint8_t x, uint8_t y) { - return x + y; -} - -SCALAR_FUN_ATTR uint16_t add16(uint16_t x, uint16_t y) { - return x + y; -} - -SCALAR_FUN_ATTR uint32_t add32(uint32_t x, uint32_t y) { - return x + y; -} - -SCALAR_FUN_ATTR uint64_t add64(uint64_t x, uint64_t y) { - return x + y; -} - -SCALAR_FUN_ATTR uint8_t sub8(uint8_t x, uint8_t y) { - return x - y; -} - -SCALAR_FUN_ATTR uint16_t sub16(uint16_t x, uint16_t y) { - return x - y; -} - -SCALAR_FUN_ATTR uint32_t sub32(uint32_t x, uint32_t y) { - return x - y; -} - -SCALAR_FUN_ATTR uint64_t sub64(uint64_t x, uint64_t y) { - return x - y; -} - -SCALAR_FUN_ATTR uint8_t mul8(uint8_t x, uint8_t y) { - return x * y; -} - -SCALAR_FUN_ATTR uint16_t mul16(uint16_t x, uint16_t y) { - return x * y; -} - -SCALAR_FUN_ATTR uint32_t mul32(uint32_t x, uint32_t y) { - return x * y; -} - -SCALAR_FUN_ATTR uint64_t mul64(uint64_t x, uint64_t y) { - return x * y; -} - -#if defined(ISPC) - -SCALAR_FUN_ATTR uint8_t udiv8(uint8_t x, uint8_t y) { - // This strange pattern is used to prevent the ISPC compiler from - // causing SIGFPEs and bogus results on divisions where inactive lanes - // have 0-valued divisors. It ensures that any inactive lane instead - // has a divisor of 1. https://github.com/ispc/ispc/issues/2292 - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR uint16_t udiv16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR uint32_t udiv32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - - return x / ys; -} - -SCALAR_FUN_ATTR uint64_t udiv64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - - return x / ys; -} - -SCALAR_FUN_ATTR uint8_t udiv_up8(uint8_t x, uint8_t y) { - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - - return (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint16_t udiv_up16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint32_t udiv_up32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint64_t udiv_up64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint8_t umod8(uint8_t x, uint8_t y) { - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR uint16_t umod16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - - return x % ys; -} - -SCALAR_FUN_ATTR uint32_t umod32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR uint64_t umod64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR uint8_t udiv_safe8(uint8_t x, uint8_t y) { - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR uint16_t udiv_safe16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR uint32_t udiv_safe32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR uint64_t udiv_safe64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR uint8_t udiv_up_safe8(uint8_t x, uint8_t y) { - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint16_t udiv_up_safe16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint32_t udiv_up_safe32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint64_t udiv_up_safe64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : (x + y - 1) / ys; -} - -SCALAR_FUN_ATTR uint8_t umod_safe8(uint8_t x, uint8_t y) { - uint8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR uint16_t umod_safe16(uint16_t x, uint16_t y) { - uint16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR uint32_t umod_safe32(uint32_t x, uint32_t y) { - uint32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR uint64_t umod_safe64(uint64_t x, uint64_t y) { - uint64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR int8_t sdiv8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - int8_t q = x / ys; - int8_t r = x % ys; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int16_t sdiv16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - int16_t q = x / ys; - int16_t r = x % ys; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int32_t sdiv32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - int32_t q = x / ys; - int32_t r = x % ys; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int64_t sdiv64(int64_t x, int64_t y) { - int64_t ys = 1; - foreach_active(i){ - ys = y; - } - - int64_t q = x / ys; - int64_t r = x % ys; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int8_t sdiv_up8(int8_t x, int8_t y) { - return sdiv8(x + y - 1, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_up16(int16_t x, int16_t y) { - return sdiv16(x + y - 1, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_up32(int32_t x, int32_t y) { - return sdiv32(x + y - 1, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_up64(int64_t x, int64_t y) { - return sdiv64(x + y - 1, y); -} - -SCALAR_FUN_ATTR int8_t smod8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - int8_t r = x % ys; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int16_t smod16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - int16_t r = x % ys; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int32_t smod32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - - int32_t r = x % ys; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int64_t smod64(int64_t x, int64_t y) { - int64_t ys = 1; - foreach_active(i){ - ys = y; - } - - int64_t r = x % ys; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int8_t sdiv_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : sdiv8(x, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : sdiv16(x, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : sdiv32(x, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : sdiv64(x, y); -} - -SCALAR_FUN_ATTR int8_t sdiv_up_safe8(int8_t x, int8_t y) { - return sdiv_safe8(x + y - 1, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_up_safe16(int16_t x, int16_t y) { - return sdiv_safe16(x + y - 1, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_up_safe32(int32_t x, int32_t y) { - return sdiv_safe32(x + y - 1, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_up_safe64(int64_t x, int64_t y) { - return sdiv_safe64(x + y - 1, y); -} - -SCALAR_FUN_ATTR int8_t smod_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : smod8(x, y); -} - -SCALAR_FUN_ATTR int16_t smod_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : smod16(x, y); -} - -SCALAR_FUN_ATTR int32_t smod_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : smod32(x, y); -} - -SCALAR_FUN_ATTR int64_t smod_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : smod64(x, y); -} - -SCALAR_FUN_ATTR int8_t squot8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR int16_t squot16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR int32_t squot32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR int64_t squot64(int64_t x, int64_t y) { - int64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x / ys; -} - -SCALAR_FUN_ATTR int8_t srem8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR int16_t srem16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR int32_t srem32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR int64_t srem64(int64_t x, int64_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return x % ys; -} - -SCALAR_FUN_ATTR int8_t squot_safe8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR int16_t squot_safe16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR int32_t squot_safe32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR int64_t squot_safe64(int64_t x, int64_t y) { - int64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x / ys; -} - -SCALAR_FUN_ATTR int8_t srem_safe8(int8_t x, int8_t y) { - int8_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR int16_t srem_safe16(int16_t x, int16_t y) { - int16_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR int32_t srem_safe32(int32_t x, int32_t y) { - int32_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -SCALAR_FUN_ATTR int64_t srem_safe64(int64_t x, int64_t y) { - int64_t ys = 1; - foreach_active(i){ - ys = y; - } - - return y == 0 ? 0 : x % ys; -} - -#else - -SCALAR_FUN_ATTR uint8_t udiv8(uint8_t x, uint8_t y) { - return x / y; -} - -SCALAR_FUN_ATTR uint16_t udiv16(uint16_t x, uint16_t y) { - return x / y; -} - -SCALAR_FUN_ATTR uint32_t udiv32(uint32_t x, uint32_t y) { - return x / y; -} - -SCALAR_FUN_ATTR uint64_t udiv64(uint64_t x, uint64_t y) { - return x / y; -} - -SCALAR_FUN_ATTR uint8_t udiv_up8(uint8_t x, uint8_t y) { - return (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint16_t udiv_up16(uint16_t x, uint16_t y) { - return (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint32_t udiv_up32(uint32_t x, uint32_t y) { - return (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint64_t udiv_up64(uint64_t x, uint64_t y) { - return (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint8_t umod8(uint8_t x, uint8_t y) { - return x % y; -} - -SCALAR_FUN_ATTR uint16_t umod16(uint16_t x, uint16_t y) { - return x % y; -} - -SCALAR_FUN_ATTR uint32_t umod32(uint32_t x, uint32_t y) { - return x % y; -} - -SCALAR_FUN_ATTR uint64_t umod64(uint64_t x, uint64_t y) { - return x % y; -} - -SCALAR_FUN_ATTR uint8_t udiv_safe8(uint8_t x, uint8_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR uint16_t udiv_safe16(uint16_t x, uint16_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR uint32_t udiv_safe32(uint32_t x, uint32_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR uint64_t udiv_safe64(uint64_t x, uint64_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR uint8_t udiv_up_safe8(uint8_t x, uint8_t y) { - return y == 0 ? 0 : (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint16_t udiv_up_safe16(uint16_t x, uint16_t y) { - return y == 0 ? 0 : (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint32_t udiv_up_safe32(uint32_t x, uint32_t y) { - return y == 0 ? 0 : (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint64_t udiv_up_safe64(uint64_t x, uint64_t y) { - return y == 0 ? 0 : (x + y - 1) / y; -} - -SCALAR_FUN_ATTR uint8_t umod_safe8(uint8_t x, uint8_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR uint16_t umod_safe16(uint16_t x, uint16_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR uint32_t umod_safe32(uint32_t x, uint32_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR uint64_t umod_safe64(uint64_t x, uint64_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR int8_t sdiv8(int8_t x, int8_t y) { - int8_t q = x / y; - int8_t r = x % y; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int16_t sdiv16(int16_t x, int16_t y) { - int16_t q = x / y; - int16_t r = x % y; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int32_t sdiv32(int32_t x, int32_t y) { - int32_t q = x / y; - int32_t r = x % y; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int64_t sdiv64(int64_t x, int64_t y) { - int64_t q = x / y; - int64_t r = x % y; - - return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); -} - -SCALAR_FUN_ATTR int8_t sdiv_up8(int8_t x, int8_t y) { - return sdiv8(x + y - 1, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_up16(int16_t x, int16_t y) { - return sdiv16(x + y - 1, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_up32(int32_t x, int32_t y) { - return sdiv32(x + y - 1, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_up64(int64_t x, int64_t y) { - return sdiv64(x + y - 1, y); -} - -SCALAR_FUN_ATTR int8_t smod8(int8_t x, int8_t y) { - int8_t r = x % y; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int16_t smod16(int16_t x, int16_t y) { - int16_t r = x % y; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int32_t smod32(int32_t x, int32_t y) { - int32_t r = x % y; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int64_t smod64(int64_t x, int64_t y) { - int64_t r = x % y; - - return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); -} - -SCALAR_FUN_ATTR int8_t sdiv_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : sdiv8(x, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : sdiv16(x, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : sdiv32(x, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : sdiv64(x, y); -} - -SCALAR_FUN_ATTR int8_t sdiv_up_safe8(int8_t x, int8_t y) { - return sdiv_safe8(x + y - 1, y); -} - -SCALAR_FUN_ATTR int16_t sdiv_up_safe16(int16_t x, int16_t y) { - return sdiv_safe16(x + y - 1, y); -} - -SCALAR_FUN_ATTR int32_t sdiv_up_safe32(int32_t x, int32_t y) { - return sdiv_safe32(x + y - 1, y); -} - -SCALAR_FUN_ATTR int64_t sdiv_up_safe64(int64_t x, int64_t y) { - return sdiv_safe64(x + y - 1, y); -} - -SCALAR_FUN_ATTR int8_t smod_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : smod8(x, y); -} - -SCALAR_FUN_ATTR int16_t smod_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : smod16(x, y); -} - -SCALAR_FUN_ATTR int32_t smod_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : smod32(x, y); -} - -SCALAR_FUN_ATTR int64_t smod_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : smod64(x, y); -} - -SCALAR_FUN_ATTR int8_t squot8(int8_t x, int8_t y) { - return x / y; -} - -SCALAR_FUN_ATTR int16_t squot16(int16_t x, int16_t y) { - return x / y; -} - -SCALAR_FUN_ATTR int32_t squot32(int32_t x, int32_t y) { - return x / y; -} - -SCALAR_FUN_ATTR int64_t squot64(int64_t x, int64_t y) { - return x / y; -} - -SCALAR_FUN_ATTR int8_t srem8(int8_t x, int8_t y) { - return x % y; -} - -SCALAR_FUN_ATTR int16_t srem16(int16_t x, int16_t y) { - return x % y; -} - -SCALAR_FUN_ATTR int32_t srem32(int32_t x, int32_t y) { - return x % y; -} - -SCALAR_FUN_ATTR int64_t srem64(int64_t x, int64_t y) { - return x % y; -} - -SCALAR_FUN_ATTR int8_t squot_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR int16_t squot_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR int32_t squot_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR int64_t squot_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : x / y; -} - -SCALAR_FUN_ATTR int8_t srem_safe8(int8_t x, int8_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR int16_t srem_safe16(int16_t x, int16_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR int32_t srem_safe32(int32_t x, int32_t y) { - return y == 0 ? 0 : x % y; -} - -SCALAR_FUN_ATTR int64_t srem_safe64(int64_t x, int64_t y) { - return y == 0 ? 0 : x % y; -} - -#endif - -SCALAR_FUN_ATTR int8_t smin8(int8_t x, int8_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR int16_t smin16(int16_t x, int16_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR int32_t smin32(int32_t x, int32_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR int64_t smin64(int64_t x, int64_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR uint8_t umin8(uint8_t x, uint8_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR uint16_t umin16(uint16_t x, uint16_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR uint32_t umin32(uint32_t x, uint32_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR uint64_t umin64(uint64_t x, uint64_t y) { - return x < y ? x : y; -} - -SCALAR_FUN_ATTR int8_t smax8(int8_t x, int8_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR int16_t smax16(int16_t x, int16_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR int32_t smax32(int32_t x, int32_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR int64_t smax64(int64_t x, int64_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR uint8_t umax8(uint8_t x, uint8_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR uint16_t umax16(uint16_t x, uint16_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR uint32_t umax32(uint32_t x, uint32_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR uint64_t umax64(uint64_t x, uint64_t y) { - return x < y ? y : x; -} - -SCALAR_FUN_ATTR uint8_t shl8(uint8_t x, uint8_t y) { - return (uint8_t)(x << y); -} - -SCALAR_FUN_ATTR uint16_t shl16(uint16_t x, uint16_t y) { - return (uint16_t)(x << y); -} - -SCALAR_FUN_ATTR uint32_t shl32(uint32_t x, uint32_t y) { - return x << y; -} - -SCALAR_FUN_ATTR uint64_t shl64(uint64_t x, uint64_t y) { - return x << y; -} - -SCALAR_FUN_ATTR uint8_t lshr8(uint8_t x, uint8_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR uint16_t lshr16(uint16_t x, uint16_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR uint32_t lshr32(uint32_t x, uint32_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR uint64_t lshr64(uint64_t x, uint64_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR int8_t ashr8(int8_t x, int8_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR int16_t ashr16(int16_t x, int16_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR int32_t ashr32(int32_t x, int32_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR int64_t ashr64(int64_t x, int64_t y) { - return x >> y; -} - -SCALAR_FUN_ATTR uint8_t and8(uint8_t x, uint8_t y) { - return x & y; -} - -SCALAR_FUN_ATTR uint16_t and16(uint16_t x, uint16_t y) { - return x & y; -} - -SCALAR_FUN_ATTR uint32_t and32(uint32_t x, uint32_t y) { - return x & y; -} - -SCALAR_FUN_ATTR uint64_t and64(uint64_t x, uint64_t y) { - return x & y; -} - -SCALAR_FUN_ATTR uint8_t or8(uint8_t x, uint8_t y) { - return x | y; -} - -SCALAR_FUN_ATTR uint16_t or16(uint16_t x, uint16_t y) { - return x | y; -} - -SCALAR_FUN_ATTR uint32_t or32(uint32_t x, uint32_t y) { - return x | y; -} - -SCALAR_FUN_ATTR uint64_t or64(uint64_t x, uint64_t y) { - return x | y; -} - -SCALAR_FUN_ATTR uint8_t xor8(uint8_t x, uint8_t y) { - return x ^ y; -} - -SCALAR_FUN_ATTR uint16_t xor16(uint16_t x, uint16_t y) { - return x ^ y; -} - -SCALAR_FUN_ATTR uint32_t xor32(uint32_t x, uint32_t y) { - return x ^ y; -} - -SCALAR_FUN_ATTR uint64_t xor64(uint64_t x, uint64_t y) { - return x ^ y; -} - -SCALAR_FUN_ATTR bool ult8(uint8_t x, uint8_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool ult16(uint16_t x, uint16_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool ult32(uint32_t x, uint32_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool ult64(uint64_t x, uint64_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool ule8(uint8_t x, uint8_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool ule16(uint16_t x, uint16_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool ule32(uint32_t x, uint32_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool ule64(uint64_t x, uint64_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool slt8(int8_t x, int8_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool slt16(int16_t x, int16_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool slt32(int32_t x, int32_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool slt64(int64_t x, int64_t y) { - return x < y; -} - -SCALAR_FUN_ATTR bool sle8(int8_t x, int8_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool sle16(int16_t x, int16_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool sle32(int32_t x, int32_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR bool sle64(int64_t x, int64_t y) { - return x <= y; -} - -SCALAR_FUN_ATTR uint8_t pow8(uint8_t x, uint8_t y) { - uint8_t res = 1, rem = y; - - while (rem != 0) { - if (rem & 1) - res *= x; - rem >>= 1; - x *= x; - } - return res; -} - -SCALAR_FUN_ATTR uint16_t pow16(uint16_t x, uint16_t y) { - uint16_t res = 1, rem = y; - - while (rem != 0) { - if (rem & 1) - res *= x; - rem >>= 1; - x *= x; - } - return res; -} - -SCALAR_FUN_ATTR uint32_t pow32(uint32_t x, uint32_t y) { - uint32_t res = 1, rem = y; - - while (rem != 0) { - if (rem & 1) - res *= x; - rem >>= 1; - x *= x; - } - return res; -} - -SCALAR_FUN_ATTR uint64_t pow64(uint64_t x, uint64_t y) { - uint64_t res = 1, rem = y; - - while (rem != 0) { - if (rem & 1) - res *= x; - rem >>= 1; - x *= x; - } - return res; -} - -SCALAR_FUN_ATTR bool itob_i8_bool(int8_t x) { - return x != 0; -} - -SCALAR_FUN_ATTR bool itob_i16_bool(int16_t x) { - return x != 0; -} - -SCALAR_FUN_ATTR bool itob_i32_bool(int32_t x) { - return x != 0; -} - -SCALAR_FUN_ATTR bool itob_i64_bool(int64_t x) { - return x != 0; -} - -SCALAR_FUN_ATTR int8_t btoi_bool_i8(bool x) { - return x; -} - -SCALAR_FUN_ATTR int16_t btoi_bool_i16(bool x) { - return x; -} - -SCALAR_FUN_ATTR int32_t btoi_bool_i32(bool x) { - return x; -} - -SCALAR_FUN_ATTR int64_t btoi_bool_i64(bool x) { - return x; -} - -#define sext_i8_i8(x) ((int8_t) (int8_t) (x)) -#define sext_i8_i16(x) ((int16_t) (int8_t) (x)) -#define sext_i8_i32(x) ((int32_t) (int8_t) (x)) -#define sext_i8_i64(x) ((int64_t) (int8_t) (x)) -#define sext_i16_i8(x) ((int8_t) (int16_t) (x)) -#define sext_i16_i16(x) ((int16_t) (int16_t) (x)) -#define sext_i16_i32(x) ((int32_t) (int16_t) (x)) -#define sext_i16_i64(x) ((int64_t) (int16_t) (x)) -#define sext_i32_i8(x) ((int8_t) (int32_t) (x)) -#define sext_i32_i16(x) ((int16_t) (int32_t) (x)) -#define sext_i32_i32(x) ((int32_t) (int32_t) (x)) -#define sext_i32_i64(x) ((int64_t) (int32_t) (x)) -#define sext_i64_i8(x) ((int8_t) (int64_t) (x)) -#define sext_i64_i16(x) ((int16_t) (int64_t) (x)) -#define sext_i64_i32(x) ((int32_t) (int64_t) (x)) -#define sext_i64_i64(x) ((int64_t) (int64_t) (x)) -#define zext_i8_i8(x) ((int8_t) (uint8_t) (x)) -#define zext_i8_i16(x) ((int16_t) (uint8_t) (x)) -#define zext_i8_i32(x) ((int32_t) (uint8_t) (x)) -#define zext_i8_i64(x) ((int64_t) (uint8_t) (x)) -#define zext_i16_i8(x) ((int8_t) (uint16_t) (x)) -#define zext_i16_i16(x) ((int16_t) (uint16_t) (x)) -#define zext_i16_i32(x) ((int32_t) (uint16_t) (x)) -#define zext_i16_i64(x) ((int64_t) (uint16_t) (x)) -#define zext_i32_i8(x) ((int8_t) (uint32_t) (x)) -#define zext_i32_i16(x) ((int16_t) (uint32_t) (x)) -#define zext_i32_i32(x) ((int32_t) (uint32_t) (x)) -#define zext_i32_i64(x) ((int64_t) (uint32_t) (x)) -#define zext_i64_i8(x) ((int8_t) (uint64_t) (x)) -#define zext_i64_i16(x) ((int16_t) (uint64_t) (x)) -#define zext_i64_i32(x) ((int32_t) (uint64_t) (x)) -#define zext_i64_i64(x) ((int64_t) (uint64_t) (x)) - -SCALAR_FUN_ATTR int8_t abs8(int8_t x) { - return (int8_t)abs(x); -} - -SCALAR_FUN_ATTR int16_t abs16(int16_t x) { - return (int16_t)abs(x); -} - -SCALAR_FUN_ATTR int32_t abs32(int32_t x) { - return abs(x); -} - -SCALAR_FUN_ATTR int64_t abs64(int64_t x) { -#if defined(__OPENCL_VERSION__) || defined(ISPC) - return abs(x); -#else - return llabs(x); -#endif -} - -#if defined(__OPENCL_VERSION__) -SCALAR_FUN_ATTR int32_t futrts_popc8(int8_t x) { - return popcount(x); -} - -SCALAR_FUN_ATTR int32_t futrts_popc16(int16_t x) { - return popcount(x); -} - -SCALAR_FUN_ATTR int32_t futrts_popc32(int32_t x) { - return popcount(x); -} - -SCALAR_FUN_ATTR int32_t futrts_popc64(int64_t x) { - return popcount(x); -} -#elif defined(__CUDA_ARCH__) - -SCALAR_FUN_ATTR int32_t futrts_popc8(int8_t x) { - return __popc(zext_i8_i32(x)); -} - -SCALAR_FUN_ATTR int32_t futrts_popc16(int16_t x) { - return __popc(zext_i16_i32(x)); -} - -SCALAR_FUN_ATTR int32_t futrts_popc32(int32_t x) { - return __popc(x); -} - -SCALAR_FUN_ATTR int32_t futrts_popc64(int64_t x) { - return __popcll(x); -} - -#else // Not OpenCL or CUDA, but plain C. - -SCALAR_FUN_ATTR int32_t futrts_popc8(uint8_t x) { - int c = 0; - for (; x; ++c) { x &= x - 1; } - return c; -} - -SCALAR_FUN_ATTR int32_t futrts_popc16(uint16_t x) { - int c = 0; - for (; x; ++c) { x &= x - 1; } - return c; -} - -SCALAR_FUN_ATTR int32_t futrts_popc32(uint32_t x) { - int c = 0; - for (; x; ++c) { x &= x - 1; } - return c; -} - -SCALAR_FUN_ATTR int32_t futrts_popc64(uint64_t x) { - int c = 0; - for (; x; ++c) { x &= x - 1; } - return c; -} -#endif - -#if defined(__OPENCL_VERSION__) -SCALAR_FUN_ATTR uint8_t futrts_umul_hi8 ( uint8_t a, uint8_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint16_t futrts_smul_hi16(int16_t a, int16_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint32_t futrts_smul_hi32(int32_t a, int32_t b) { return mul_hi(a, b); } -SCALAR_FUN_ATTR uint64_t futrts_smul_hi64(int64_t a, int64_t b) { return mul_hi(a, b); } -#elif defined(__CUDA_ARCH__) -SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } -SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } -SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return __umulhi(a, b); } -SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return __umul64hi(a, b); } -SCALAR_FUN_ATTR uint8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return ((int16_t)a) * ((int16_t)b) >> 8; } -SCALAR_FUN_ATTR uint16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((int32_t)a) * ((int32_t)b) >> 16; } -SCALAR_FUN_ATTR uint32_t futrts_smul_hi32(int32_t a, int32_t b) { return __mulhi(a, b); } -SCALAR_FUN_ATTR uint64_t futrts_smul_hi64(int64_t a, int64_t b) { return __mul64hi(a, b); } -#elif defined(ISPC) -SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } -SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } -SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } -SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { - uint64_t ah = a >> 32; - uint64_t al = a & 0xffffffff; - uint64_t bh = b >> 32; - uint64_t bl = b & 0xffffffff; - - uint64_t p1 = al * bl; - uint64_t p2 = al * bh; - uint64_t p3 = ah * bl; - uint64_t p4 = ah * bh; - - uint64_t p1h = p1 >> 32; - uint64_t p2h = p2 >> 32; - uint64_t p3h = p3 >> 32; - uint64_t p2l = p2 & 0xffffffff; - uint64_t p3l = p3 & 0xffffffff; - - uint64_t l = p1h + p2l + p3l; - uint64_t m = (p2 >> 32) + (p3 >> 32); - uint64_t h = (l >> 32) + m + p4; - - return h; -} -SCALAR_FUN_ATTR int8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } -SCALAR_FUN_ATTR int16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } -SCALAR_FUN_ATTR int32_t futrts_smul_hi32(int32_t a, int32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } -SCALAR_FUN_ATTR int64_t futrts_smul_hi64(int64_t a, int64_t b) { - uint64_t ah = a >> 32; - uint64_t al = a & 0xffffffff; - uint64_t bh = b >> 32; - uint64_t bl = b & 0xffffffff; - - uint64_t p1 = al * bl; - int64_t p2 = al * bh; - int64_t p3 = ah * bl; - uint64_t p4 = ah * bh; - - uint64_t p1h = p1 >> 32; - uint64_t p2h = p2 >> 32; - uint64_t p3h = p3 >> 32; - uint64_t p2l = p2 & 0xffffffff; - uint64_t p3l = p3 & 0xffffffff; - - uint64_t l = p1h + p2l + p3l; - uint64_t m = (p2 >> 32) + (p3 >> 32); - uint64_t h = (l >> 32) + m + p4; - - return h; -} - -#else // Not OpenCL, ISPC, or CUDA, but plain C. -SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } -SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } -SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } -SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return ((__uint128_t)a) * ((__uint128_t)b) >> 64; } -SCALAR_FUN_ATTR int8_t futrts_smul_hi8(int8_t a, int8_t b) { return ((int16_t)a) * ((int16_t)b) >> 8; } -SCALAR_FUN_ATTR int16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((int32_t)a) * ((int32_t)b) >> 16; } -SCALAR_FUN_ATTR int32_t futrts_smul_hi32(int32_t a, int32_t b) { return ((int64_t)a) * ((int64_t)b) >> 32; } -SCALAR_FUN_ATTR int64_t futrts_smul_hi64(int64_t a, int64_t b) { return ((__int128_t)a) * ((__int128_t)b) >> 64; } -#endif - -#if defined(__OPENCL_VERSION__) -SCALAR_FUN_ATTR uint8_t futrts_umad_hi8 ( uint8_t a, uint8_t b, uint8_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint16_t futrts_umad_hi16(uint16_t a, uint16_t b, uint16_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint32_t futrts_umad_hi32(uint32_t a, uint32_t b, uint32_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint64_t futrts_umad_hi64(uint64_t a, uint64_t b, uint64_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint8_t futrts_smad_hi8( int8_t a, int8_t b, int8_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint16_t futrts_smad_hi16(int16_t a, int16_t b, int16_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint32_t futrts_smad_hi32(int32_t a, int32_t b, int32_t c) { return mad_hi(a, b, c); } -SCALAR_FUN_ATTR uint64_t futrts_smad_hi64(int64_t a, int64_t b, int64_t c) { return mad_hi(a, b, c); } -#else // Not OpenCL - -SCALAR_FUN_ATTR uint8_t futrts_umad_hi8( uint8_t a, uint8_t b, uint8_t c) { return futrts_umul_hi8(a, b) + c; } -SCALAR_FUN_ATTR uint16_t futrts_umad_hi16(uint16_t a, uint16_t b, uint16_t c) { return futrts_umul_hi16(a, b) + c; } -SCALAR_FUN_ATTR uint32_t futrts_umad_hi32(uint32_t a, uint32_t b, uint32_t c) { return futrts_umul_hi32(a, b) + c; } -SCALAR_FUN_ATTR uint64_t futrts_umad_hi64(uint64_t a, uint64_t b, uint64_t c) { return futrts_umul_hi64(a, b) + c; } -SCALAR_FUN_ATTR uint8_t futrts_smad_hi8 ( int8_t a, int8_t b, int8_t c) { return futrts_smul_hi8(a, b) + c; } -SCALAR_FUN_ATTR uint16_t futrts_smad_hi16(int16_t a, int16_t b, int16_t c) { return futrts_smul_hi16(a, b) + c; } -SCALAR_FUN_ATTR uint32_t futrts_smad_hi32(int32_t a, int32_t b, int32_t c) { return futrts_smul_hi32(a, b) + c; } -SCALAR_FUN_ATTR uint64_t futrts_smad_hi64(int64_t a, int64_t b, int64_t c) { return futrts_smul_hi64(a, b) + c; } -#endif - -#if defined(__OPENCL_VERSION__) -SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { - return clz(x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { - return clz(x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { - return clz(x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { - return clz(x); -} - -#elif defined(__CUDA_ARCH__) - -SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { - return __clz(zext_i8_i32(x)) - 24; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { - return __clz(zext_i16_i32(x)) - 16; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { - return __clz(x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { - return __clzll(x); -} - -#elif defined(ISPC) - -SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { - return count_leading_zeros((int32_t)(uint8_t)x)-24; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { - return count_leading_zeros((int32_t)(uint16_t)x)-16; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { - return count_leading_zeros(x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { - return count_leading_zeros(x); -} - -#else // Not OpenCL, ISPC or CUDA, but plain C. - -SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { - return x == 0 ? 8 : __builtin_clz((uint32_t)zext_i8_i32(x)) - 24; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { - return x == 0 ? 16 : __builtin_clz((uint32_t)zext_i16_i32(x)) - 16; -} - -SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { - return x == 0 ? 32 : __builtin_clz((uint32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { - return x == 0 ? 64 : __builtin_clzll((uint64_t)x); -} -#endif - -#if defined(__OPENCL_VERSION__) -SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { - int i = 0; - for (; i < 8 && (x & 1) == 0; i++, x >>= 1) - ; - return i; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { - int i = 0; - for (; i < 16 && (x & 1) == 0; i++, x >>= 1) - ; - return i; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { - int i = 0; - for (; i < 32 && (x & 1) == 0; i++, x >>= 1) - ; - return i; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { - int i = 0; - for (; i < 64 && (x & 1) == 0; i++, x >>= 1) - ; - return i; -} - -#elif defined(__CUDA_ARCH__) - -SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { - int y = __ffs(x); - return y == 0 ? 8 : y - 1; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { - int y = __ffs(x); - return y == 0 ? 16 : y - 1; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { - int y = __ffs(x); - return y == 0 ? 32 : y - 1; -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { - int y = __ffsll(x); - return y == 0 ? 64 : y - 1; -} - -#elif defined(ISPC) - -SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { - return x == 0 ? 8 : count_trailing_zeros((int32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { - return x == 0 ? 16 : count_trailing_zeros((int32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { - return count_trailing_zeros(x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { - return count_trailing_zeros(x); -} - -#else // Not OpenCL or CUDA, but plain C. - -SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { - return x == 0 ? 8 : __builtin_ctz((uint32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { - return x == 0 ? 16 : __builtin_ctz((uint32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { - return x == 0 ? 32 : __builtin_ctz((uint32_t)x); -} - -SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { - return x == 0 ? 64 : __builtin_ctzll((uint64_t)x); -} -#endif - -SCALAR_FUN_ATTR float fdiv32(float x, float y) { - return x / y; -} - -SCALAR_FUN_ATTR float fadd32(float x, float y) { - return x + y; -} - -SCALAR_FUN_ATTR float fsub32(float x, float y) { - return x - y; -} - -SCALAR_FUN_ATTR float fmul32(float x, float y) { - return x * y; -} - -SCALAR_FUN_ATTR bool cmplt32(float x, float y) { - return x < y; -} - -SCALAR_FUN_ATTR bool cmple32(float x, float y) { - return x <= y; -} - -SCALAR_FUN_ATTR float sitofp_i8_f32(int8_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float sitofp_i16_f32(int16_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float sitofp_i32_f32(int32_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float sitofp_i64_f32(int64_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float uitofp_i8_f32(uint8_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float uitofp_i16_f32(uint16_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float uitofp_i32_f32(uint32_t x) { - return (float) x; -} - -SCALAR_FUN_ATTR float uitofp_i64_f32(uint64_t x) { - return (float) x; -} - -#ifdef __OPENCL_VERSION__ -SCALAR_FUN_ATTR float fabs32(float x) { - return fabs(x); -} - -SCALAR_FUN_ATTR float fmax32(float x, float y) { - return fmax(x, y); -} - -SCALAR_FUN_ATTR float fmin32(float x, float y) { - return fmin(x, y); -} - -SCALAR_FUN_ATTR float fpow32(float x, float y) { - return pow(x, y); -} - -#elif defined(ISPC) - -SCALAR_FUN_ATTR float fabs32(float x) { - return abs(x); -} - -SCALAR_FUN_ATTR float fmax32(float x, float y) { - return isnan(x) ? y : isnan(y) ? x : max(x, y); -} - -SCALAR_FUN_ATTR float fmin32(float x, float y) { - return isnan(x) ? y : isnan(y) ? x : min(x, y); -} - -SCALAR_FUN_ATTR float fpow32(float a, float b) { - float ret; - foreach_active (i) { - uniform float r = pow(extract(a, i), extract(b, i)); - ret = insert(ret, i, r); - } - return ret; -} - -#else // Not OpenCL, but CUDA or plain C. - -SCALAR_FUN_ATTR float fabs32(float x) { - return fabsf(x); -} - -SCALAR_FUN_ATTR float fmax32(float x, float y) { - return fmaxf(x, y); -} - -SCALAR_FUN_ATTR float fmin32(float x, float y) { - return fminf(x, y); -} - -SCALAR_FUN_ATTR float fpow32(float x, float y) { - return powf(x, y); -} -#endif - -SCALAR_FUN_ATTR bool futrts_isnan32(float x) { - return isnan(x); -} - -#if defined(ISPC) - -SCALAR_FUN_ATTR bool futrts_isinf32(float x) { - return !isnan(x) && isnan(x - x); -} - -SCALAR_FUN_ATTR bool futrts_isfinite32(float x) { - return !isnan(x) && !futrts_isinf32(x); -} - -#else - -SCALAR_FUN_ATTR bool futrts_isinf32(float x) { - return isinf(x); -} - -#endif - -SCALAR_FUN_ATTR int8_t fptosi_f32_i8(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (int8_t) x; - } -} - -SCALAR_FUN_ATTR int16_t fptosi_f32_i16(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (int16_t) x; - } -} - -SCALAR_FUN_ATTR int32_t fptosi_f32_i32(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (int32_t) x; - } -} - -SCALAR_FUN_ATTR int64_t fptosi_f32_i64(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (int64_t) x; - }; -} - -SCALAR_FUN_ATTR uint8_t fptoui_f32_i8(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (uint8_t) (int8_t) x; - } -} - -SCALAR_FUN_ATTR uint16_t fptoui_f32_i16(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (uint16_t) (int16_t) x; - } -} - -SCALAR_FUN_ATTR uint32_t fptoui_f32_i32(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (uint32_t) (int32_t) x; - } -} - -SCALAR_FUN_ATTR uint64_t fptoui_f32_i64(float x) { - if (futrts_isnan32(x) || futrts_isinf32(x)) { - return 0; - } else { - return (uint64_t) (int64_t) x; - } -} - -SCALAR_FUN_ATTR bool ftob_f32_bool(float x) { - return x != 0; -} - -SCALAR_FUN_ATTR float btof_bool_f32(bool x) { - return x ? 1 : 0; -} - -#ifdef __OPENCL_VERSION__ -SCALAR_FUN_ATTR float futrts_log32(float x) { - return log(x); -} - -SCALAR_FUN_ATTR float futrts_log2_32(float x) { - return log2(x); -} - -SCALAR_FUN_ATTR float futrts_log10_32(float x) { - return log10(x); -} - -SCALAR_FUN_ATTR float futrts_log1p_32(float x) { - return log1p(x); -} - -SCALAR_FUN_ATTR float futrts_sqrt32(float x) { - return sqrt(x); -} - -SCALAR_FUN_ATTR float futrts_cbrt32(float x) { - return cbrt(x); -} - -SCALAR_FUN_ATTR float futrts_exp32(float x) { - return exp(x); -} - -SCALAR_FUN_ATTR float futrts_cos32(float x) { - return cos(x); -} - -SCALAR_FUN_ATTR float futrts_sin32(float x) { - return sin(x); -} - -SCALAR_FUN_ATTR float futrts_tan32(float x) { - return tan(x); -} - -SCALAR_FUN_ATTR float futrts_acos32(float x) { - return acos(x); -} - -SCALAR_FUN_ATTR float futrts_asin32(float x) { - return asin(x); -} - -SCALAR_FUN_ATTR float futrts_atan32(float x) { - return atan(x); -} - -SCALAR_FUN_ATTR float futrts_cosh32(float x) { - return cosh(x); -} - -SCALAR_FUN_ATTR float futrts_sinh32(float x) { - return sinh(x); -} - -SCALAR_FUN_ATTR float futrts_tanh32(float x) { - return tanh(x); -} - -SCALAR_FUN_ATTR float futrts_acosh32(float x) { - return acosh(x); -} - -SCALAR_FUN_ATTR float futrts_asinh32(float x) { - return asinh(x); -} - -SCALAR_FUN_ATTR float futrts_atanh32(float x) { - return atanh(x); -} - -SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) { - return atan2(x, y); -} - -SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { - return hypot(x, y); -} - -SCALAR_FUN_ATTR float futrts_gamma32(float x) { - return tgamma(x); -} - -SCALAR_FUN_ATTR float futrts_lgamma32(float x) { - return lgamma(x); -} - -SCALAR_FUN_ATTR float futrts_erf32(float x) { - return erf(x); -} - -SCALAR_FUN_ATTR float futrts_erfc32(float x) { - return erfc(x); -} - -SCALAR_FUN_ATTR float fmod32(float x, float y) { - return fmod(x, y); -} - -SCALAR_FUN_ATTR float futrts_round32(float x) { - return rint(x); -} - -SCALAR_FUN_ATTR float futrts_floor32(float x) { - return floor(x); -} - -SCALAR_FUN_ATTR float futrts_ceil32(float x) { - return ceil(x); -} - -SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { - return nextafter(x, y); -} - -SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { - return mix(v0, v1, t); -} - -SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { - return ldexp(x, y); -} - -SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { - return copysign(x, y); -} - -SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { - return mad(a, b, c); -} - -SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { - return fma(a, b, c); -} - -#elif defined(ISPC) - -SCALAR_FUN_ATTR float futrts_log32(float x) { - return futrts_isfinite32(x) || (futrts_isinf32(x) && x < 0)? log(x) : x; -} - -SCALAR_FUN_ATTR float futrts_log2_32(float x) { - return futrts_log32(x) / log(2.0f); -} - -SCALAR_FUN_ATTR float futrts_log10_32(float x) { - return futrts_log32(x) / log(10.0f); -} - -SCALAR_FUN_ATTR float futrts_log1p_32(float x) { - if(x == -1.0f || (futrts_isinf32(x) && x > 0.0f)) return x / 0.0f; - float y = 1.0f + x; - float z = y - 1.0f; - return log(y) - (z-x)/y; -} - -SCALAR_FUN_ATTR float futrts_sqrt32(float x) { - return sqrt(x); -} - -extern "C" unmasked uniform float cbrtf(uniform float); -SCALAR_FUN_ATTR float futrts_cbrt32(float x) { - float res; - foreach_active (i) { - uniform float r = cbrtf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR float futrts_exp32(float x) { - return exp(x); -} - -SCALAR_FUN_ATTR float futrts_cos32(float x) { - return cos(x); -} - -SCALAR_FUN_ATTR float futrts_sin32(float x) { - return sin(x); -} - -SCALAR_FUN_ATTR float futrts_tan32(float x) { - return tan(x); -} - -SCALAR_FUN_ATTR float futrts_acos32(float x) { - return acos(x); -} - -SCALAR_FUN_ATTR float futrts_asin32(float x) { - return asin(x); -} - -SCALAR_FUN_ATTR float futrts_atan32(float x) { - return atan(x); -} - -SCALAR_FUN_ATTR float futrts_cosh32(float x) { - return (exp(x)+exp(-x)) / 2.0f; -} - -SCALAR_FUN_ATTR float futrts_sinh32(float x) { - return (exp(x)-exp(-x)) / 2.0f; -} - -SCALAR_FUN_ATTR float futrts_tanh32(float x) { - return futrts_sinh32(x)/futrts_cosh32(x); -} - -SCALAR_FUN_ATTR float futrts_acosh32(float x) { - float f = x+sqrt(x*x-1); - if(futrts_isfinite32(f)) return log(f); - return f; -} - -SCALAR_FUN_ATTR float futrts_asinh32(float x) { - float f = x+sqrt(x*x+1); - if(futrts_isfinite32(f)) return log(f); - return f; - -} - -SCALAR_FUN_ATTR float futrts_atanh32(float x) { - float f = (1+x)/(1-x); - if(futrts_isfinite32(f)) return log(f)/2.0f; - return f; - -} - -SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) { - return (x == 0.0f && y == 0.0f) ? 0.0f : atan2(x, y); -} - -SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { - if (futrts_isfinite32(x) && futrts_isfinite32(y)) { - x = abs(x); - y = abs(y); - float a; - float b; - if (x >= y){ - a = x; - b = y; - } else { - a = y; - b = x; - } - if(b == 0){ - return a; - } - - int e; - float an; - float bn; - an = frexp (a, &e); - bn = ldexp (b, - e); - float cn; - cn = sqrt (an * an + bn * bn); - return ldexp (cn, e); - } else { - if (futrts_isinf32(x) || futrts_isinf32(y)) return INFINITY; - else return x + y; - } - -} - -extern "C" unmasked uniform float tgammaf(uniform float x); -SCALAR_FUN_ATTR float futrts_gamma32(float x) { - float res; - foreach_active (i) { - uniform float r = tgammaf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform float lgammaf(uniform float x); -SCALAR_FUN_ATTR float futrts_lgamma32(float x) { - float res; - foreach_active (i) { - uniform float r = lgammaf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform float erff(uniform float x); -SCALAR_FUN_ATTR float futrts_erf32(float x) { - float res; - foreach_active (i) { - uniform float r = erff(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform float erfcf(uniform float x); -SCALAR_FUN_ATTR float futrts_erfc32(float x) { - float res; - foreach_active (i) { - uniform float r = erfcf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR float fmod32(float x, float y) { - return x - y * trunc(x/y); -} - -SCALAR_FUN_ATTR float futrts_round32(float x) { - return round(x); -} - -SCALAR_FUN_ATTR float futrts_floor32(float x) { - return floor(x); -} - -SCALAR_FUN_ATTR float futrts_ceil32(float x) { - return ceil(x); -} - -extern "C" unmasked uniform float nextafterf(uniform float x, uniform float y); -SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { - float res; - foreach_active (i) { - uniform float r = nextafterf(extract(x, i), extract(y, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { - return v0 + (v1 - v0) * t; -} - -SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { - return x * pow((uniform float)2.0, (float)y); -} - -SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { - int32_t xb = futrts_to_bits32(x); - int32_t yb = futrts_to_bits32(y); - return futrts_from_bits32((xb & ~(1<<31)) | (yb & (1<<31))); -} - -SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { - return a * b + c; -} - -SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { - return a * b + c; -} - -#else // Not OpenCL or ISPC, but CUDA or plain C. - -SCALAR_FUN_ATTR float futrts_log32(float x) { - return logf(x); -} - -SCALAR_FUN_ATTR float futrts_log2_32(float x) { - return log2f(x); -} - -SCALAR_FUN_ATTR float futrts_log10_32(float x) { - return log10f(x); -} - -SCALAR_FUN_ATTR float futrts_log1p_32(float x) { - return log1pf(x); -} - -SCALAR_FUN_ATTR float futrts_sqrt32(float x) { - return sqrtf(x); -} - -SCALAR_FUN_ATTR float futrts_cbrt32(float x) { - return cbrtf(x); -} - -SCALAR_FUN_ATTR float futrts_exp32(float x) { - return expf(x); -} - -SCALAR_FUN_ATTR float futrts_cos32(float x) { - return cosf(x); -} - -SCALAR_FUN_ATTR float futrts_sin32(float x) { - return sinf(x); -} - -SCALAR_FUN_ATTR float futrts_tan32(float x) { - return tanf(x); -} - -SCALAR_FUN_ATTR float futrts_acos32(float x) { - return acosf(x); -} - -SCALAR_FUN_ATTR float futrts_asin32(float x) { - return asinf(x); -} - -SCALAR_FUN_ATTR float futrts_atan32(float x) { - return atanf(x); -} - -SCALAR_FUN_ATTR float futrts_cosh32(float x) { - return coshf(x); -} - -SCALAR_FUN_ATTR float futrts_sinh32(float x) { - return sinhf(x); -} - -SCALAR_FUN_ATTR float futrts_tanh32(float x) { - return tanhf(x); -} - -SCALAR_FUN_ATTR float futrts_acosh32(float x) { - return acoshf(x); -} - -SCALAR_FUN_ATTR float futrts_asinh32(float x) { - return asinhf(x); -} - -SCALAR_FUN_ATTR float futrts_atanh32(float x) { - return atanhf(x); -} - -SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) { - return atan2f(x, y); -} - -SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { - return hypotf(x, y); -} - -SCALAR_FUN_ATTR float futrts_gamma32(float x) { - return tgammaf(x); -} - -SCALAR_FUN_ATTR float futrts_lgamma32(float x) { - return lgammaf(x); -} - -SCALAR_FUN_ATTR float futrts_erf32(float x) { - return erff(x); -} - -SCALAR_FUN_ATTR float futrts_erfc32(float x) { - return erfcf(x); -} - -SCALAR_FUN_ATTR float fmod32(float x, float y) { - return fmodf(x, y); -} - -SCALAR_FUN_ATTR float futrts_round32(float x) { - return rintf(x); -} - -SCALAR_FUN_ATTR float futrts_floor32(float x) { - return floorf(x); -} - -SCALAR_FUN_ATTR float futrts_ceil32(float x) { - return ceilf(x); -} - -SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { - return nextafterf(x, y); -} - -SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { - return v0 + (v1 - v0) * t; -} - -SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { - return ldexpf(x, y); -} - -SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { - return copysignf(x, y); -} - -SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { - return a * b + c; -} - -SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { - return fmaf(a, b, c); -} -#endif - -#if defined(ISPC) -SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x) { - return intbits(x); -} - -SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x) { - return floatbits(x); -} -#else -SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x) { - union { - float f; - int32_t t; - } p; - - p.f = x; - return p.t; -} - -SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x) { - union { - int32_t f; - float t; - } p; - - p.f = x; - return p.t; -} -#endif - -SCALAR_FUN_ATTR float fsignum32(float x) { - return futrts_isnan32(x) ? x : (x > 0 ? 1 : 0) - (x < 0 ? 1 : 0); -} - -#ifdef FUTHARK_F64_ENABLED - -SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x); -SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x); - -#if defined(ISPC) -SCALAR_FUN_ATTR bool futrts_isinf64(float x) { - return !isnan(x) && isnan(x - x); -} - -SCALAR_FUN_ATTR bool futrts_isfinite64(float x) { - return !isnan(x) && !futrts_isinf64(x); -} - -SCALAR_FUN_ATTR double fdiv64(double x, double y) { - return x / y; -} - -SCALAR_FUN_ATTR double fadd64(double x, double y) { - return x + y; -} - -SCALAR_FUN_ATTR double fsub64(double x, double y) { - return x - y; -} - -SCALAR_FUN_ATTR double fmul64(double x, double y) { - return x * y; -} - -SCALAR_FUN_ATTR bool cmplt64(double x, double y) { - return x < y; -} - -SCALAR_FUN_ATTR bool cmple64(double x, double y) { - return x <= y; -} - -SCALAR_FUN_ATTR double sitofp_i8_f64(int8_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i16_f64(int16_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i32_f64(int32_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i64_f64(int64_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i8_f64(uint8_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i16_f64(uint16_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i32_f64(uint32_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i64_f64(uint64_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double fabs64(double x) { - return abs(x); -} - -SCALAR_FUN_ATTR double fmax64(double x, double y) { - return isnan(x) ? y : isnan(y) ? x : max(x, y); -} - -SCALAR_FUN_ATTR double fmin64(double x, double y) { - return isnan(x) ? y : isnan(y) ? x : min(x, y); -} - -SCALAR_FUN_ATTR double fpow64(double a, double b) { - float ret; - foreach_active (i) { - uniform float r = pow(extract(a, i), extract(b, i)); - ret = insert(ret, i, r); - } - return ret; -} - -SCALAR_FUN_ATTR double futrts_log64(double x) { - return futrts_isfinite64(x) || (futrts_isinf64(x) && x < 0)? log(x) : x; -} - -SCALAR_FUN_ATTR double futrts_log2_64(double x) { - return futrts_log64(x)/log(2.0d); -} - -SCALAR_FUN_ATTR double futrts_log10_64(double x) { - return futrts_log64(x)/log(10.0d); -} - -SCALAR_FUN_ATTR double futrts_log1p_64(double x) { - if(x == -1.0d || (futrts_isinf64(x) && x > 0.0d)) return x / 0.0d; - double y = 1.0d + x; - double z = y - 1.0d; - return log(y) - (z-x)/y; -} - -SCALAR_FUN_ATTR double futrts_sqrt64(double x) { - return sqrt(x); -} - -extern "C" unmasked uniform double cbrt(uniform double); -SCALAR_FUN_ATTR double futrts_cbrt64(double x) { - double res; - foreach_active (i) { - uniform double r = cbrtf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR double futrts_exp64(double x) { - return exp(x); -} - -SCALAR_FUN_ATTR double futrts_cos64(double x) { - return cos(x); -} - -SCALAR_FUN_ATTR double futrts_sin64(double x) { - return sin(x); -} - -SCALAR_FUN_ATTR double futrts_tan64(double x) { - return tan(x); -} - -SCALAR_FUN_ATTR double futrts_acos64(double x) { - return acos(x); -} - -SCALAR_FUN_ATTR double futrts_asin64(double x) { - return asin(x); -} - -SCALAR_FUN_ATTR double futrts_atan64(double x) { - return atan(x); -} - -SCALAR_FUN_ATTR double futrts_cosh64(double x) { - return (exp(x)+exp(-x)) / 2.0d; -} - -SCALAR_FUN_ATTR double futrts_sinh64(double x) { - return (exp(x)-exp(-x)) / 2.0d; -} - -SCALAR_FUN_ATTR double futrts_tanh64(double x) { - return futrts_sinh64(x)/futrts_cosh64(x); -} - -SCALAR_FUN_ATTR double futrts_acosh64(double x) { - double f = x+sqrt(x*x-1.0d); - if(futrts_isfinite64(f)) return log(f); - return f; -} - -SCALAR_FUN_ATTR double futrts_asinh64(double x) { - double f = x+sqrt(x*x+1.0d); - if(futrts_isfinite64(f)) return log(f); - return f; -} - -SCALAR_FUN_ATTR double futrts_atanh64(double x) { - double f = (1.0d+x)/(1.0d-x); - if(futrts_isfinite64(f)) return log(f)/2.0d; - return f; - -} - -SCALAR_FUN_ATTR double futrts_atan2_64(double x, double y) { - return atan2(x, y); -} - -extern "C" unmasked uniform double hypot(uniform double x, uniform double y); -SCALAR_FUN_ATTR double futrts_hypot64(double x, double y) { - double res; - foreach_active (i) { - uniform double r = hypot(extract(x, i), extract(y, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform double tgamma(uniform double x); -SCALAR_FUN_ATTR double futrts_gamma64(double x) { - double res; - foreach_active (i) { - uniform double r = tgamma(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform double lgamma(uniform double x); -SCALAR_FUN_ATTR double futrts_lgamma64(double x) { - double res; - foreach_active (i) { - uniform double r = lgamma(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform double erf(uniform double x); -SCALAR_FUN_ATTR double futrts_erf64(double x) { - double res; - foreach_active (i) { - uniform double r = erf(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -extern "C" unmasked uniform double erfc(uniform double x); -SCALAR_FUN_ATTR double futrts_erfc64(double x) { - double res; - foreach_active (i) { - uniform double r = erfc(extract(x, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR double futrts_fma64(double a, double b, double c) { - return a * b + c; -} - -SCALAR_FUN_ATTR double futrts_round64(double x) { - return round(x); -} - -SCALAR_FUN_ATTR double futrts_ceil64(double x) { - return ceil(x); -} - -extern "C" unmasked uniform double nextafter(uniform float x, uniform double y); -SCALAR_FUN_ATTR float futrts_nextafter64(double x, double y) { - double res; - foreach_active (i) { - uniform double r = nextafter(extract(x, i), extract(y, i)); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR double futrts_floor64(double x) { - return floor(x); -} - -SCALAR_FUN_ATTR bool futrts_isnan64(double x) { - return isnan(x); -} - -SCALAR_FUN_ATTR int8_t fptosi_f64_i8(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int8_t) x; - } -} - -SCALAR_FUN_ATTR int16_t fptosi_f64_i16(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int16_t) x; - } -} - -SCALAR_FUN_ATTR int32_t fptosi_f64_i32(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int32_t) x; - } -} - -SCALAR_FUN_ATTR int64_t fptosi_f64_i64(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int64_t) x; - } -} - -SCALAR_FUN_ATTR uint8_t fptoui_f64_i8(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint8_t) (int8_t) x; - } -} - -SCALAR_FUN_ATTR uint16_t fptoui_f64_i16(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint16_t) (int16_t) x; - } -} - -SCALAR_FUN_ATTR uint32_t fptoui_f64_i32(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint32_t) (int32_t) x; - } -} - -SCALAR_FUN_ATTR uint64_t fptoui_f64_i64(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint64_t) (int64_t) x; - } -} - -SCALAR_FUN_ATTR bool ftob_f64_bool(double x) { - return x != 0.0; -} - -SCALAR_FUN_ATTR double btof_bool_f64(bool x) { - return x ? 1.0 : 0.0; -} - -SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x) { - int64_t res; - foreach_active (i) { - uniform double tmp = extract(x, i); - uniform int64_t r = *((uniform int64_t* uniform)&tmp); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x) { - double res; - foreach_active (i) { - uniform int64_t tmp = extract(x, i); - uniform double r = *((uniform double* uniform)&tmp); - res = insert(res, i, r); - } - return res; -} - -SCALAR_FUN_ATTR double fmod64(double x, double y) { - return x - y * trunc(x/y); -} - -SCALAR_FUN_ATTR double fsignum64(double x) { - return futrts_isnan64(x) ? x : (x > 0 ? 1.0d : 0.0d) - (x < 0 ? 1.0d : 0.0d); -} - -SCALAR_FUN_ATTR double futrts_lerp64(double v0, double v1, double t) { - return v0 + (v1 - v0) * t; -} - -SCALAR_FUN_ATTR double futrts_ldexp64(double x, int32_t y) { - return x * pow((uniform double)2.0, (double)y); -} - -SCALAR_FUN_ATTR double futrts_copysign64(double x, double y) { - int64_t xb = futrts_to_bits64(x); - int64_t yb = futrts_to_bits64(y); - return futrts_from_bits64((xb & ~(((int64_t)1)<<63)) | (yb & (((int64_t)1)<<63))); -} - -SCALAR_FUN_ATTR double futrts_mad64(double a, double b, double c) { - return a * b + c; -} - -SCALAR_FUN_ATTR float fpconv_f32_f32(float x) { - return (float) x; -} - -SCALAR_FUN_ATTR double fpconv_f32_f64(float x) { - return (double) x; -} - -SCALAR_FUN_ATTR float fpconv_f64_f32(double x) { - return (float) x; -} - -SCALAR_FUN_ATTR double fpconv_f64_f64(double x) { - return (double) x; -} - -#else - -SCALAR_FUN_ATTR double fdiv64(double x, double y) { - return x / y; -} - -SCALAR_FUN_ATTR double fadd64(double x, double y) { - return x + y; -} - -SCALAR_FUN_ATTR double fsub64(double x, double y) { - return x - y; -} - -SCALAR_FUN_ATTR double fmul64(double x, double y) { - return x * y; -} - -SCALAR_FUN_ATTR bool cmplt64(double x, double y) { - return x < y; -} - -SCALAR_FUN_ATTR bool cmple64(double x, double y) { - return x <= y; -} - -SCALAR_FUN_ATTR double sitofp_i8_f64(int8_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i16_f64(int16_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i32_f64(int32_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double sitofp_i64_f64(int64_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i8_f64(uint8_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i16_f64(uint16_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i32_f64(uint32_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double uitofp_i64_f64(uint64_t x) { - return (double) x; -} - -SCALAR_FUN_ATTR double fabs64(double x) { - return fabs(x); -} - -SCALAR_FUN_ATTR double fmax64(double x, double y) { - return fmax(x, y); -} - -SCALAR_FUN_ATTR double fmin64(double x, double y) { - return fmin(x, y); -} - -SCALAR_FUN_ATTR double fpow64(double x, double y) { - return pow(x, y); -} - -SCALAR_FUN_ATTR double futrts_log64(double x) { - return log(x); -} - -SCALAR_FUN_ATTR double futrts_log2_64(double x) { - return log2(x); -} - -SCALAR_FUN_ATTR double futrts_log10_64(double x) { - return log10(x); -} - -SCALAR_FUN_ATTR double futrts_log1p_64(double x) { - return log1p(x); -} - -SCALAR_FUN_ATTR double futrts_sqrt64(double x) { - return sqrt(x); -} - -SCALAR_FUN_ATTR double futrts_cbrt64(double x) { - return cbrt(x); -} - -SCALAR_FUN_ATTR double futrts_exp64(double x) { - return exp(x); -} - -SCALAR_FUN_ATTR double futrts_cos64(double x) { - return cos(x); -} - -SCALAR_FUN_ATTR double futrts_sin64(double x) { - return sin(x); -} - -SCALAR_FUN_ATTR double futrts_tan64(double x) { - return tan(x); -} - -SCALAR_FUN_ATTR double futrts_acos64(double x) { - return acos(x); -} - -SCALAR_FUN_ATTR double futrts_asin64(double x) { - return asin(x); -} - -SCALAR_FUN_ATTR double futrts_atan64(double x) { - return atan(x); -} - -SCALAR_FUN_ATTR double futrts_cosh64(double x) { - return cosh(x); -} - -SCALAR_FUN_ATTR double futrts_sinh64(double x) { - return sinh(x); -} - -SCALAR_FUN_ATTR double futrts_tanh64(double x) { - return tanh(x); -} - -SCALAR_FUN_ATTR double futrts_acosh64(double x) { - return acosh(x); -} - -SCALAR_FUN_ATTR double futrts_asinh64(double x) { - return asinh(x); -} - -SCALAR_FUN_ATTR double futrts_atanh64(double x) { - return atanh(x); -} - -SCALAR_FUN_ATTR double futrts_atan2_64(double x, double y) { - return atan2(x, y); -} - -SCALAR_FUN_ATTR double futrts_hypot64(double x, double y) { - return hypot(x, y); -} - -SCALAR_FUN_ATTR double futrts_gamma64(double x) { - return tgamma(x); -} - -SCALAR_FUN_ATTR double futrts_lgamma64(double x) { - return lgamma(x); -} - -SCALAR_FUN_ATTR double futrts_erf64(double x) { - return erf(x); -} - -SCALAR_FUN_ATTR double futrts_erfc64(double x) { - return erfc(x); -} - -SCALAR_FUN_ATTR double futrts_fma64(double a, double b, double c) { - return fma(a, b, c); -} - -SCALAR_FUN_ATTR double futrts_round64(double x) { - return rint(x); -} - -SCALAR_FUN_ATTR double futrts_ceil64(double x) { - return ceil(x); -} - -SCALAR_FUN_ATTR float futrts_nextafter64(float x, float y) { - return nextafter(x, y); -} - -SCALAR_FUN_ATTR double futrts_floor64(double x) { - return floor(x); -} - -SCALAR_FUN_ATTR bool futrts_isnan64(double x) { - return isnan(x); -} - -SCALAR_FUN_ATTR bool futrts_isinf64(double x) { - return isinf(x); -} - -SCALAR_FUN_ATTR int8_t fptosi_f64_i8(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int8_t) x; - } -} - -SCALAR_FUN_ATTR int16_t fptosi_f64_i16(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int16_t) x; - } -} - -SCALAR_FUN_ATTR int32_t fptosi_f64_i32(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int32_t) x; - } -} - -SCALAR_FUN_ATTR int64_t fptosi_f64_i64(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (int64_t) x; - } -} - -SCALAR_FUN_ATTR uint8_t fptoui_f64_i8(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint8_t) (int8_t) x; - } -} - -SCALAR_FUN_ATTR uint16_t fptoui_f64_i16(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint16_t) (int16_t) x; - } -} - -SCALAR_FUN_ATTR uint32_t fptoui_f64_i32(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint32_t) (int32_t) x; - } -} - -SCALAR_FUN_ATTR uint64_t fptoui_f64_i64(double x) { - if (futrts_isnan64(x) || futrts_isinf64(x)) { - return 0; - } else { - return (uint64_t) (int64_t) x; - } -} - -SCALAR_FUN_ATTR bool ftob_f64_bool(double x) { - return x != 0; -} - -SCALAR_FUN_ATTR double btof_bool_f64(bool x) { - return x ? 1 : 0; -} - -SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x) { - union { - double f; - int64_t t; - } p; - - p.f = x; - return p.t; -} - -SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x) { - union { - int64_t f; - double t; - } p; - - p.f = x; - return p.t; -} - -SCALAR_FUN_ATTR double fmod64(double x, double y) { - return fmod(x, y); -} - -SCALAR_FUN_ATTR double fsignum64(double x) { - return futrts_isnan64(x) ? x : (x > 0) - (x < 0); -} - -SCALAR_FUN_ATTR double futrts_lerp64(double v0, double v1, double t) { -#ifdef __OPENCL_VERSION__ - return mix(v0, v1, t); -#else - return v0 + (v1 - v0) * t; -#endif -} - -SCALAR_FUN_ATTR double futrts_ldexp64(double x, int32_t y) { - return ldexp(x, y); -} - -SCALAR_FUN_ATTR float futrts_copysign64(double x, double y) { - return copysign(x, y); -} - -SCALAR_FUN_ATTR double futrts_mad64(double a, double b, double c) { -#ifdef __OPENCL_VERSION__ - return mad(a, b, c); -#else - return a * b + c; -#endif -} - -SCALAR_FUN_ATTR float fpconv_f32_f32(float x) { - return (float) x; -} - -SCALAR_FUN_ATTR double fpconv_f32_f64(float x) { - return (double) x; -} - -SCALAR_FUN_ATTR float fpconv_f64_f32(double x) { - return (float) x; -} - -SCALAR_FUN_ATTR double fpconv_f64_f64(double x) { - return (double) x; -} +#ifndef M_PI +#define M_PI 3.141592653589793 +#endif + +SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x); +SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x); + +SCALAR_FUN_ATTR uint8_t add8(uint8_t x, uint8_t y) { return x + y; } +SCALAR_FUN_ATTR uint16_t add16(uint16_t x, uint16_t y) { return x + y; } +SCALAR_FUN_ATTR uint32_t add32(uint32_t x, uint32_t y) { return x + y; } +SCALAR_FUN_ATTR uint64_t add64(uint64_t x, uint64_t y) { return x + y; } + +SCALAR_FUN_ATTR uint8_t sub8(uint8_t x, uint8_t y) { return x - y; } +SCALAR_FUN_ATTR uint16_t sub16(uint16_t x, uint16_t y) { return x - y; } +SCALAR_FUN_ATTR uint32_t sub32(uint32_t x, uint32_t y) { return x - y; } +SCALAR_FUN_ATTR uint64_t sub64(uint64_t x, uint64_t y) { return x - y; } + +SCALAR_FUN_ATTR uint8_t mul8(uint8_t x, uint8_t y) { return x * y; } +SCALAR_FUN_ATTR uint16_t mul16(uint16_t x, uint16_t y) { return x * y; } +SCALAR_FUN_ATTR uint32_t mul32(uint32_t x, uint32_t y) { return x * y; } +SCALAR_FUN_ATTR uint64_t mul64(uint64_t x, uint64_t y) { return x * y; } + +#if defined(ISPC) + +SCALAR_FUN_ATTR uint8_t udiv8(uint8_t x, uint8_t y) { + // This strange pattern is used to prevent the ISPC compiler from + // causing SIGFPEs and bogus results on divisions where inactive lanes + // have 0-valued divisors. It ensures that any inactive lane instead + // has a divisor of 1. https://github.com/ispc/ispc/issues/2292 + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR uint16_t udiv16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR uint32_t udiv32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR uint64_t udiv64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR uint8_t udiv_up8(uint8_t x, uint8_t y) { + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint16_t udiv_up16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint32_t udiv_up32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint64_t udiv_up64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint8_t umod8(uint8_t x, uint8_t y) { + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR uint16_t umod16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR uint32_t umod32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR uint64_t umod64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR uint8_t udiv_safe8(uint8_t x, uint8_t y) { + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR uint16_t udiv_safe16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR uint32_t udiv_safe32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR uint64_t udiv_safe64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR uint8_t udiv_up_safe8(uint8_t x, uint8_t y) { + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint16_t udiv_up_safe16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint32_t udiv_up_safe32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint64_t udiv_up_safe64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : (x + y - 1) / ys; +} + +SCALAR_FUN_ATTR uint8_t umod_safe8(uint8_t x, uint8_t y) { + uint8_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR uint16_t umod_safe16(uint16_t x, uint16_t y) { + uint16_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR uint32_t umod_safe32(uint32_t x, uint32_t y) { + uint32_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR uint64_t umod_safe64(uint64_t x, uint64_t y) { + uint64_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR int8_t sdiv8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + int8_t q = x / ys; + int8_t r = x % ys; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int16_t sdiv16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + int16_t q = x / ys; + int16_t r = x % ys; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int32_t sdiv32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + int32_t q = x / ys; + int32_t r = x % ys; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int64_t sdiv64(int64_t x, int64_t y) { + int64_t ys = 1; + foreach_active(i) { ys = y; } + int64_t q = x / ys; + int64_t r = x % ys; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int8_t sdiv_up8(int8_t x, int8_t y) { return sdiv8(x + y - 1, y); } +SCALAR_FUN_ATTR int16_t sdiv_up16(int16_t x, int16_t y) { return sdiv16(x + y - 1, y); } +SCALAR_FUN_ATTR int32_t sdiv_up32(int32_t x, int32_t y) { return sdiv32(x + y - 1, y); } +SCALAR_FUN_ATTR int64_t sdiv_up64(int64_t x, int64_t y) { return sdiv64(x + y - 1, y); } + +SCALAR_FUN_ATTR int8_t smod8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + int8_t r = x % ys; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int16_t smod16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + int16_t r = x % ys; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int32_t smod32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + int32_t r = x % ys; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int64_t smod64(int64_t x, int64_t y) { + int64_t ys = 1; + foreach_active(i) { ys = y; } + int64_t r = x % ys; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int8_t sdiv_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : sdiv8(x, y); } +SCALAR_FUN_ATTR int16_t sdiv_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : sdiv16(x, y); } +SCALAR_FUN_ATTR int32_t sdiv_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : sdiv32(x, y); } +SCALAR_FUN_ATTR int64_t sdiv_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : sdiv64(x, y); } + +SCALAR_FUN_ATTR int8_t sdiv_up_safe8(int8_t x, int8_t y) { return sdiv_safe8(x + y - 1, y); } +SCALAR_FUN_ATTR int16_t sdiv_up_safe16(int16_t x, int16_t y) { return sdiv_safe16(x + y - 1, y); } +SCALAR_FUN_ATTR int32_t sdiv_up_safe32(int32_t x, int32_t y) { return sdiv_safe32(x + y - 1, y); } +SCALAR_FUN_ATTR int64_t sdiv_up_safe64(int64_t x, int64_t y) { return sdiv_safe64(x + y - 1, y); } + +SCALAR_FUN_ATTR int8_t smod_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : smod8(x, y); } +SCALAR_FUN_ATTR int16_t smod_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : smod16(x, y); } +SCALAR_FUN_ATTR int32_t smod_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : smod32(x, y); } +SCALAR_FUN_ATTR int64_t smod_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : smod64(x, y); } + +SCALAR_FUN_ATTR int8_t squot8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR int16_t squot16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR int32_t squot32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR int64_t squot64(int64_t x, int64_t y) { + int64_t ys = 1; + foreach_active(i) { ys = y; } + return x / ys; +} + +SCALAR_FUN_ATTR int8_t srem8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR int16_t srem16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR int32_t srem32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR int64_t srem64(int64_t x, int64_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + return x % ys; +} + +SCALAR_FUN_ATTR int8_t squot_safe8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR int16_t squot_safe16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR int32_t squot_safe32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR int64_t squot_safe64(int64_t x, int64_t y) { + int64_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x / ys; +} + +SCALAR_FUN_ATTR int8_t srem_safe8(int8_t x, int8_t y) { + int8_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR int16_t srem_safe16(int16_t x, int16_t y) { + int16_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR int32_t srem_safe32(int32_t x, int32_t y) { + int32_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +SCALAR_FUN_ATTR int64_t srem_safe64(int64_t x, int64_t y) { + int64_t ys = 1; + foreach_active(i) { ys = y; } + return y == 0 ? 0 : x % ys; +} + +#else + +SCALAR_FUN_ATTR uint8_t udiv8(uint8_t x, uint8_t y) { return x / y; } +SCALAR_FUN_ATTR uint16_t udiv16(uint16_t x, uint16_t y) { return x / y; } +SCALAR_FUN_ATTR uint32_t udiv32(uint32_t x, uint32_t y) { return x / y; } +SCALAR_FUN_ATTR uint64_t udiv64(uint64_t x, uint64_t y) { return x / y; } + +SCALAR_FUN_ATTR uint8_t udiv_up8(uint8_t x, uint8_t y) { return (x + y - 1) / y; } +SCALAR_FUN_ATTR uint16_t udiv_up16(uint16_t x, uint16_t y) { return (x + y - 1) / y; } +SCALAR_FUN_ATTR uint32_t udiv_up32(uint32_t x, uint32_t y) { return (x + y - 1) / y; } +SCALAR_FUN_ATTR uint64_t udiv_up64(uint64_t x, uint64_t y) { return (x + y - 1) / y; } + +SCALAR_FUN_ATTR uint8_t umod8(uint8_t x, uint8_t y) { return x % y; } +SCALAR_FUN_ATTR uint16_t umod16(uint16_t x, uint16_t y) { return x % y; } +SCALAR_FUN_ATTR uint32_t umod32(uint32_t x, uint32_t y) { return x % y; } +SCALAR_FUN_ATTR uint64_t umod64(uint64_t x, uint64_t y) { return x % y; } + +SCALAR_FUN_ATTR uint8_t udiv_safe8(uint8_t x, uint8_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR uint16_t udiv_safe16(uint16_t x, uint16_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR uint32_t udiv_safe32(uint32_t x, uint32_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR uint64_t udiv_safe64(uint64_t x, uint64_t y) { return y == 0 ? 0 : x / y; } + +SCALAR_FUN_ATTR uint8_t udiv_up_safe8(uint8_t x, uint8_t y) { return y == 0 ? 0 : (x + y - 1) / y; } +SCALAR_FUN_ATTR uint16_t udiv_up_safe16(uint16_t x, uint16_t y) { return y == 0 ? 0 : (x + y - 1) / y; } +SCALAR_FUN_ATTR uint32_t udiv_up_safe32(uint32_t x, uint32_t y) { return y == 0 ? 0 : (x + y - 1) / y; } +SCALAR_FUN_ATTR uint64_t udiv_up_safe64(uint64_t x, uint64_t y) { return y == 0 ? 0 : (x + y - 1) / y; } + +SCALAR_FUN_ATTR uint8_t umod_safe8(uint8_t x, uint8_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR uint16_t umod_safe16(uint16_t x, uint16_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR uint32_t umod_safe32(uint32_t x, uint32_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR uint64_t umod_safe64(uint64_t x, uint64_t y) { return y == 0 ? 0 : x % y; } + +SCALAR_FUN_ATTR int8_t sdiv8(int8_t x, int8_t y) { + int8_t q = x / y; + int8_t r = x % y; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int16_t sdiv16(int16_t x, int16_t y) { + int16_t q = x / y; + int16_t r = x % y; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int32_t sdiv32(int32_t x, int32_t y) { + int32_t q = x / y; + int32_t r = x % y; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int64_t sdiv64(int64_t x, int64_t y) { + int64_t q = x / y; + int64_t r = x % y; + return q - ((r != 0 && r < 0 != y < 0) ? 1 : 0); +} + +SCALAR_FUN_ATTR int8_t sdiv_up8(int8_t x, int8_t y) { return sdiv8(x + y - 1, y); } +SCALAR_FUN_ATTR int16_t sdiv_up16(int16_t x, int16_t y) { return sdiv16(x + y - 1, y); } +SCALAR_FUN_ATTR int32_t sdiv_up32(int32_t x, int32_t y) { return sdiv32(x + y - 1, y); } +SCALAR_FUN_ATTR int64_t sdiv_up64(int64_t x, int64_t y) { return sdiv64(x + y - 1, y); } + +SCALAR_FUN_ATTR int8_t smod8(int8_t x, int8_t y) { + int8_t r = x % y; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int16_t smod16(int16_t x, int16_t y) { + int16_t r = x % y; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int32_t smod32(int32_t x, int32_t y) { + int32_t r = x % y; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int64_t smod64(int64_t x, int64_t y) { + int64_t r = x % y; + return r + (r == 0 || (x > 0 && y > 0) || (x < 0 && y < 0) ? 0 : y); +} + +SCALAR_FUN_ATTR int8_t sdiv_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : sdiv8(x, y); } +SCALAR_FUN_ATTR int16_t sdiv_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : sdiv16(x, y); } +SCALAR_FUN_ATTR int32_t sdiv_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : sdiv32(x, y); } +SCALAR_FUN_ATTR int64_t sdiv_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : sdiv64(x, y); } + +SCALAR_FUN_ATTR int8_t sdiv_up_safe8(int8_t x, int8_t y) { return sdiv_safe8(x + y - 1, y);} +SCALAR_FUN_ATTR int16_t sdiv_up_safe16(int16_t x, int16_t y) { return sdiv_safe16(x + y - 1, y); } +SCALAR_FUN_ATTR int32_t sdiv_up_safe32(int32_t x, int32_t y) { return sdiv_safe32(x + y - 1, y); } +SCALAR_FUN_ATTR int64_t sdiv_up_safe64(int64_t x, int64_t y) { return sdiv_safe64(x + y - 1, y); } + +SCALAR_FUN_ATTR int8_t smod_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : smod8(x, y); } +SCALAR_FUN_ATTR int16_t smod_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : smod16(x, y); } +SCALAR_FUN_ATTR int32_t smod_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : smod32(x, y); } +SCALAR_FUN_ATTR int64_t smod_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : smod64(x, y); } + +SCALAR_FUN_ATTR int8_t squot8(int8_t x, int8_t y) { return x / y; } +SCALAR_FUN_ATTR int16_t squot16(int16_t x, int16_t y) { return x / y; } +SCALAR_FUN_ATTR int32_t squot32(int32_t x, int32_t y) { return x / y; } +SCALAR_FUN_ATTR int64_t squot64(int64_t x, int64_t y) { return x / y; } + +SCALAR_FUN_ATTR int8_t srem8(int8_t x, int8_t y) { return x % y; } +SCALAR_FUN_ATTR int16_t srem16(int16_t x, int16_t y) { return x % y; } +SCALAR_FUN_ATTR int32_t srem32(int32_t x, int32_t y) { return x % y; } +SCALAR_FUN_ATTR int64_t srem64(int64_t x, int64_t y) { return x % y; } + +SCALAR_FUN_ATTR int8_t squot_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR int16_t squot_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR int32_t squot_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : x / y; } +SCALAR_FUN_ATTR int64_t squot_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : x / y; } + +SCALAR_FUN_ATTR int8_t srem_safe8(int8_t x, int8_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR int16_t srem_safe16(int16_t x, int16_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR int32_t srem_safe32(int32_t x, int32_t y) { return y == 0 ? 0 : x % y; } +SCALAR_FUN_ATTR int64_t srem_safe64(int64_t x, int64_t y) { return y == 0 ? 0 : x % y; } + +#endif + +SCALAR_FUN_ATTR int8_t smin8(int8_t x, int8_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR int16_t smin16(int16_t x, int16_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR int32_t smin32(int32_t x, int32_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR int64_t smin64(int64_t x, int64_t y) { return x < y ? x : y; } + +SCALAR_FUN_ATTR uint8_t umin8(uint8_t x, uint8_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR uint16_t umin16(uint16_t x, uint16_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR uint32_t umin32(uint32_t x, uint32_t y) { return x < y ? x : y; } +SCALAR_FUN_ATTR uint64_t umin64(uint64_t x, uint64_t y) { return x < y ? x : y; } + +SCALAR_FUN_ATTR int8_t smax8(int8_t x, int8_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR int16_t smax16(int16_t x, int16_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR int32_t smax32(int32_t x, int32_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR int64_t smax64(int64_t x, int64_t y) { return x < y ? y : x; } + +SCALAR_FUN_ATTR uint8_t umax8(uint8_t x, uint8_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR uint16_t umax16(uint16_t x, uint16_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR uint32_t umax32(uint32_t x, uint32_t y) { return x < y ? y : x; } +SCALAR_FUN_ATTR uint64_t umax64(uint64_t x, uint64_t y) { return x < y ? y : x; } + +SCALAR_FUN_ATTR uint8_t shl8(uint8_t x, uint8_t y) { return (uint8_t)(x << y); } +SCALAR_FUN_ATTR uint16_t shl16(uint16_t x, uint16_t y) { return (uint16_t)(x << y); } +SCALAR_FUN_ATTR uint32_t shl32(uint32_t x, uint32_t y) { return x << y; } +SCALAR_FUN_ATTR uint64_t shl64(uint64_t x, uint64_t y) { return x << y; } + +SCALAR_FUN_ATTR uint8_t lshr8(uint8_t x, uint8_t y) { return x >> y; } +SCALAR_FUN_ATTR uint16_t lshr16(uint16_t x, uint16_t y) { return x >> y; } +SCALAR_FUN_ATTR uint32_t lshr32(uint32_t x, uint32_t y) { return x >> y; } +SCALAR_FUN_ATTR uint64_t lshr64(uint64_t x, uint64_t y) { return x >> y; } + +SCALAR_FUN_ATTR int8_t ashr8(int8_t x, int8_t y) { return x >> y; } +SCALAR_FUN_ATTR int16_t ashr16(int16_t x, int16_t y) { return x >> y; } +SCALAR_FUN_ATTR int32_t ashr32(int32_t x, int32_t y) { return x >> y; } +SCALAR_FUN_ATTR int64_t ashr64(int64_t x, int64_t y) { return x >> y; } + +SCALAR_FUN_ATTR uint8_t and8(uint8_t x, uint8_t y) { return x & y; } +SCALAR_FUN_ATTR uint16_t and16(uint16_t x, uint16_t y) { return x & y; } +SCALAR_FUN_ATTR uint32_t and32(uint32_t x, uint32_t y) { return x & y; } +SCALAR_FUN_ATTR uint64_t and64(uint64_t x, uint64_t y) { return x & y; } + +SCALAR_FUN_ATTR uint8_t or8(uint8_t x, uint8_t y) { return x | y; } +SCALAR_FUN_ATTR uint16_t or16(uint16_t x, uint16_t y) { return x | y; } +SCALAR_FUN_ATTR uint32_t or32(uint32_t x, uint32_t y) { return x | y; } +SCALAR_FUN_ATTR uint64_t or64(uint64_t x, uint64_t y) { return x | y; } + +SCALAR_FUN_ATTR uint8_t xor8(uint8_t x, uint8_t y) { return x ^ y; } +SCALAR_FUN_ATTR uint16_t xor16(uint16_t x, uint16_t y) { return x ^ y; } +SCALAR_FUN_ATTR uint32_t xor32(uint32_t x, uint32_t y) { return x ^ y; } +SCALAR_FUN_ATTR uint64_t xor64(uint64_t x, uint64_t y) { return x ^ y; } + +SCALAR_FUN_ATTR bool ult8(uint8_t x, uint8_t y) { return x < y; } +SCALAR_FUN_ATTR bool ult16(uint16_t x, uint16_t y) { return x < y; } +SCALAR_FUN_ATTR bool ult32(uint32_t x, uint32_t y) { return x < y; } +SCALAR_FUN_ATTR bool ult64(uint64_t x, uint64_t y) { return x < y; } + +SCALAR_FUN_ATTR bool ule8(uint8_t x, uint8_t y) { return x <= y; } +SCALAR_FUN_ATTR bool ule16(uint16_t x, uint16_t y) { return x <= y; } +SCALAR_FUN_ATTR bool ule32(uint32_t x, uint32_t y) { return x <= y; } +SCALAR_FUN_ATTR bool ule64(uint64_t x, uint64_t y) { return x <= y; } + +SCALAR_FUN_ATTR bool slt8(int8_t x, int8_t y) { return x < y; } +SCALAR_FUN_ATTR bool slt16(int16_t x, int16_t y) { return x < y; } +SCALAR_FUN_ATTR bool slt32(int32_t x, int32_t y) { return x < y; } +SCALAR_FUN_ATTR bool slt64(int64_t x, int64_t y) { return x < y; } + +SCALAR_FUN_ATTR bool sle8(int8_t x, int8_t y) { return x <= y; } +SCALAR_FUN_ATTR bool sle16(int16_t x, int16_t y) { return x <= y; } +SCALAR_FUN_ATTR bool sle32(int32_t x, int32_t y) { return x <= y; } +SCALAR_FUN_ATTR bool sle64(int64_t x, int64_t y) { return x <= y; } + +SCALAR_FUN_ATTR uint8_t pow8(uint8_t x, uint8_t y) { + uint8_t res = 1, rem = y; + while (rem != 0) { + if (rem & 1) + res *= x; + rem >>= 1; + x *= x; + } + return res; +} + +SCALAR_FUN_ATTR uint16_t pow16(uint16_t x, uint16_t y) { + uint16_t res = 1, rem = y; + while (rem != 0) { + if (rem & 1) + res *= x; + rem >>= 1; + x *= x; + } + return res; +} + +SCALAR_FUN_ATTR uint32_t pow32(uint32_t x, uint32_t y) { + uint32_t res = 1, rem = y; + while (rem != 0) { + if (rem & 1) + res *= x; + rem >>= 1; + x *= x; + } + return res; +} + +SCALAR_FUN_ATTR uint64_t pow64(uint64_t x, uint64_t y) { + uint64_t res = 1, rem = y; + while (rem != 0) { + if (rem & 1) + res *= x; + rem >>= 1; + x *= x; + } + return res; +} + +SCALAR_FUN_ATTR bool itob_i8_bool(int8_t x) { return x != 0; } +SCALAR_FUN_ATTR bool itob_i16_bool(int16_t x) { return x != 0; } +SCALAR_FUN_ATTR bool itob_i32_bool(int32_t x) { return x != 0; } +SCALAR_FUN_ATTR bool itob_i64_bool(int64_t x) { return x != 0; } + +SCALAR_FUN_ATTR int8_t btoi_bool_i8(bool x) { return x; } +SCALAR_FUN_ATTR int16_t btoi_bool_i16(bool x) { return x; } +SCALAR_FUN_ATTR int32_t btoi_bool_i32(bool x) { return x; } +SCALAR_FUN_ATTR int64_t btoi_bool_i64(bool x) { return x; } + +#define sext_i8_i8(x) ((int8_t) (int8_t) (x)) +#define sext_i8_i16(x) ((int16_t) (int8_t) (x)) +#define sext_i8_i32(x) ((int32_t) (int8_t) (x)) +#define sext_i8_i64(x) ((int64_t) (int8_t) (x)) +#define sext_i16_i8(x) ((int8_t) (int16_t) (x)) +#define sext_i16_i16(x) ((int16_t) (int16_t) (x)) +#define sext_i16_i32(x) ((int32_t) (int16_t) (x)) +#define sext_i16_i64(x) ((int64_t) (int16_t) (x)) +#define sext_i32_i8(x) ((int8_t) (int32_t) (x)) +#define sext_i32_i16(x) ((int16_t) (int32_t) (x)) +#define sext_i32_i32(x) ((int32_t) (int32_t) (x)) +#define sext_i32_i64(x) ((int64_t) (int32_t) (x)) +#define sext_i64_i8(x) ((int8_t) (int64_t) (x)) +#define sext_i64_i16(x) ((int16_t) (int64_t) (x)) +#define sext_i64_i32(x) ((int32_t) (int64_t) (x)) +#define sext_i64_i64(x) ((int64_t) (int64_t) (x)) +#define zext_i8_i8(x) ((int8_t) (uint8_t) (x)) +#define zext_i8_i16(x) ((int16_t) (uint8_t) (x)) +#define zext_i8_i32(x) ((int32_t) (uint8_t) (x)) +#define zext_i8_i64(x) ((int64_t) (uint8_t) (x)) +#define zext_i16_i8(x) ((int8_t) (uint16_t) (x)) +#define zext_i16_i16(x) ((int16_t) (uint16_t) (x)) +#define zext_i16_i32(x) ((int32_t) (uint16_t) (x)) +#define zext_i16_i64(x) ((int64_t) (uint16_t) (x)) +#define zext_i32_i8(x) ((int8_t) (uint32_t) (x)) +#define zext_i32_i16(x) ((int16_t) (uint32_t) (x)) +#define zext_i32_i32(x) ((int32_t) (uint32_t) (x)) +#define zext_i32_i64(x) ((int64_t) (uint32_t) (x)) +#define zext_i64_i8(x) ((int8_t) (uint64_t) (x)) +#define zext_i64_i16(x) ((int16_t) (uint64_t) (x)) +#define zext_i64_i32(x) ((int32_t) (uint64_t) (x)) +#define zext_i64_i64(x) ((int64_t) (uint64_t) (x)) + +SCALAR_FUN_ATTR int8_t abs8(int8_t x) { return (int8_t)abs(x); } +SCALAR_FUN_ATTR int16_t abs16(int16_t x) { return (int16_t)abs(x); } +SCALAR_FUN_ATTR int32_t abs32(int32_t x) { return abs(x); } +SCALAR_FUN_ATTR int64_t abs64(int64_t x) { +#if defined(__OPENCL_VERSION__) || defined(ISPC) + return abs(x); +#else + return llabs(x); +#endif +} + +#if defined(__OPENCL_VERSION__) + +SCALAR_FUN_ATTR int32_t futrts_popc8(int8_t x) { return popcount(x); } +SCALAR_FUN_ATTR int32_t futrts_popc16(int16_t x) { return popcount(x); } +SCALAR_FUN_ATTR int32_t futrts_popc32(int32_t x) { return popcount(x); } +SCALAR_FUN_ATTR int32_t futrts_popc64(int64_t x) { return popcount(x); } + +#elif defined(__CUDA_ARCH__) + +SCALAR_FUN_ATTR int32_t futrts_popc8(int8_t x) { return __popc(zext_i8_i32(x)); } +SCALAR_FUN_ATTR int32_t futrts_popc16(int16_t x) { return __popc(zext_i16_i32(x)); } +SCALAR_FUN_ATTR int32_t futrts_popc32(int32_t x) { return __popc(x); } +SCALAR_FUN_ATTR int32_t futrts_popc64(int64_t x) { return __popcll(x); } + +#else // Not OpenCL or CUDA, but plain C. + +SCALAR_FUN_ATTR int32_t futrts_popc8(uint8_t x) { + int c = 0; + for (; x; ++c) { x &= x - 1; } + return c; +} + +SCALAR_FUN_ATTR int32_t futrts_popc16(uint16_t x) { + int c = 0; + for (; x; ++c) { x &= x - 1; } + return c; +} + +SCALAR_FUN_ATTR int32_t futrts_popc32(uint32_t x) { + int c = 0; + for (; x; ++c) { x &= x - 1; } + return c; +} + +SCALAR_FUN_ATTR int32_t futrts_popc64(uint64_t x) { + int c = 0; + for (; x; ++c) { x &= x - 1; } + return c; +} +#endif + +#if defined(__OPENCL_VERSION__) +SCALAR_FUN_ATTR uint8_t futrts_umul_hi8 ( uint8_t a, uint8_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint16_t futrts_smul_hi16(int16_t a, int16_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint32_t futrts_smul_hi32(int32_t a, int32_t b) { return mul_hi(a, b); } +SCALAR_FUN_ATTR uint64_t futrts_smul_hi64(int64_t a, int64_t b) { return mul_hi(a, b); } +#elif defined(__CUDA_ARCH__) +SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } +SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } +SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return __umulhi(a, b); } +SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return __umul64hi(a, b); } +SCALAR_FUN_ATTR uint8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return ((int16_t)a) * ((int16_t)b) >> 8; } +SCALAR_FUN_ATTR uint16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((int32_t)a) * ((int32_t)b) >> 16; } +SCALAR_FUN_ATTR uint32_t futrts_smul_hi32(int32_t a, int32_t b) { return __mulhi(a, b); } +SCALAR_FUN_ATTR uint64_t futrts_smul_hi64(int64_t a, int64_t b) { return __mul64hi(a, b); } +#elif defined(ISPC) +SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } +SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } +SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } +SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { + uint64_t ah = a >> 32; + uint64_t al = a & 0xffffffff; + uint64_t bh = b >> 32; + uint64_t bl = b & 0xffffffff; + + uint64_t p1 = al * bl; + uint64_t p2 = al * bh; + uint64_t p3 = ah * bl; + uint64_t p4 = ah * bh; + + uint64_t p1h = p1 >> 32; + uint64_t p2h = p2 >> 32; + uint64_t p3h = p3 >> 32; + uint64_t p2l = p2 & 0xffffffff; + uint64_t p3l = p3 & 0xffffffff; + + uint64_t l = p1h + p2l + p3l; + uint64_t m = (p2 >> 32) + (p3 >> 32); + uint64_t h = (l >> 32) + m + p4; + + return h; +} +SCALAR_FUN_ATTR int8_t futrts_smul_hi8 ( int8_t a, int8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } +SCALAR_FUN_ATTR int16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } +SCALAR_FUN_ATTR int32_t futrts_smul_hi32(int32_t a, int32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } +SCALAR_FUN_ATTR int64_t futrts_smul_hi64(int64_t a, int64_t b) { + uint64_t ah = a >> 32; + uint64_t al = a & 0xffffffff; + uint64_t bh = b >> 32; + uint64_t bl = b & 0xffffffff; + + uint64_t p1 = al * bl; + int64_t p2 = al * bh; + int64_t p3 = ah * bl; + uint64_t p4 = ah * bh; + + uint64_t p1h = p1 >> 32; + uint64_t p2h = p2 >> 32; + uint64_t p3h = p3 >> 32; + uint64_t p2l = p2 & 0xffffffff; + uint64_t p3l = p3 & 0xffffffff; + + uint64_t l = p1h + p2l + p3l; + uint64_t m = (p2 >> 32) + (p3 >> 32); + uint64_t h = (l >> 32) + m + p4; + + return h; +} + +#else // Not OpenCL, ISPC, or CUDA, but plain C. +SCALAR_FUN_ATTR uint8_t futrts_umul_hi8(uint8_t a, uint8_t b) { return ((uint16_t)a) * ((uint16_t)b) >> 8; } +SCALAR_FUN_ATTR uint16_t futrts_umul_hi16(uint16_t a, uint16_t b) { return ((uint32_t)a) * ((uint32_t)b) >> 16; } +SCALAR_FUN_ATTR uint32_t futrts_umul_hi32(uint32_t a, uint32_t b) { return ((uint64_t)a) * ((uint64_t)b) >> 32; } +SCALAR_FUN_ATTR uint64_t futrts_umul_hi64(uint64_t a, uint64_t b) { return ((__uint128_t)a) * ((__uint128_t)b) >> 64; } +SCALAR_FUN_ATTR int8_t futrts_smul_hi8(int8_t a, int8_t b) { return ((int16_t)a) * ((int16_t)b) >> 8; } +SCALAR_FUN_ATTR int16_t futrts_smul_hi16(int16_t a, int16_t b) { return ((int32_t)a) * ((int32_t)b) >> 16; } +SCALAR_FUN_ATTR int32_t futrts_smul_hi32(int32_t a, int32_t b) { return ((int64_t)a) * ((int64_t)b) >> 32; } +SCALAR_FUN_ATTR int64_t futrts_smul_hi64(int64_t a, int64_t b) { return ((__int128_t)a) * ((__int128_t)b) >> 64; } +#endif + +#if defined(__OPENCL_VERSION__) +SCALAR_FUN_ATTR uint8_t futrts_umad_hi8 ( uint8_t a, uint8_t b, uint8_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint16_t futrts_umad_hi16(uint16_t a, uint16_t b, uint16_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint32_t futrts_umad_hi32(uint32_t a, uint32_t b, uint32_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint64_t futrts_umad_hi64(uint64_t a, uint64_t b, uint64_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint8_t futrts_smad_hi8( int8_t a, int8_t b, int8_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint16_t futrts_smad_hi16(int16_t a, int16_t b, int16_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint32_t futrts_smad_hi32(int32_t a, int32_t b, int32_t c) { return mad_hi(a, b, c); } +SCALAR_FUN_ATTR uint64_t futrts_smad_hi64(int64_t a, int64_t b, int64_t c) { return mad_hi(a, b, c); } +#else // Not OpenCL + +SCALAR_FUN_ATTR uint8_t futrts_umad_hi8( uint8_t a, uint8_t b, uint8_t c) { return futrts_umul_hi8(a, b) + c; } +SCALAR_FUN_ATTR uint16_t futrts_umad_hi16(uint16_t a, uint16_t b, uint16_t c) { return futrts_umul_hi16(a, b) + c; } +SCALAR_FUN_ATTR uint32_t futrts_umad_hi32(uint32_t a, uint32_t b, uint32_t c) { return futrts_umul_hi32(a, b) + c; } +SCALAR_FUN_ATTR uint64_t futrts_umad_hi64(uint64_t a, uint64_t b, uint64_t c) { return futrts_umul_hi64(a, b) + c; } +SCALAR_FUN_ATTR uint8_t futrts_smad_hi8 ( int8_t a, int8_t b, int8_t c) { return futrts_smul_hi8(a, b) + c; } +SCALAR_FUN_ATTR uint16_t futrts_smad_hi16(int16_t a, int16_t b, int16_t c) { return futrts_smul_hi16(a, b) + c; } +SCALAR_FUN_ATTR uint32_t futrts_smad_hi32(int32_t a, int32_t b, int32_t c) { return futrts_smul_hi32(a, b) + c; } +SCALAR_FUN_ATTR uint64_t futrts_smad_hi64(int64_t a, int64_t b, int64_t c) { return futrts_smul_hi64(a, b) + c; } +#endif + +#if defined(__OPENCL_VERSION__) +SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { return clz(x); } +SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { return clz(x); } +SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { return clz(x); } +SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { return clz(x); } + +#elif defined(__CUDA_ARCH__) + +SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { return __clz(zext_i8_i32(x)) - 24; } +SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { return __clz(zext_i16_i32(x)) - 16; } +SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { return __clz(x); } +SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { return __clzll(x); } + +#elif defined(ISPC) + +SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) { return count_leading_zeros((int32_t)(uint8_t)x)-24; } +SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) { return count_leading_zeros((int32_t)(uint16_t)x)-16; } +SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) { return count_leading_zeros(x); } +SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) { return count_leading_zeros(x); } + +#else // Not OpenCL, ISPC or CUDA, but plain C. + +SCALAR_FUN_ATTR int32_t futrts_clzz8(int8_t x) +{ return x == 0 ? 8 : __builtin_clz((uint32_t)zext_i8_i32(x)) - 24; } +SCALAR_FUN_ATTR int32_t futrts_clzz16(int16_t x) +{ return x == 0 ? 16 : __builtin_clz((uint32_t)zext_i16_i32(x)) - 16; } +SCALAR_FUN_ATTR int32_t futrts_clzz32(int32_t x) +{ return x == 0 ? 32 : __builtin_clz((uint32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_clzz64(int64_t x) +{ return x == 0 ? 64 : __builtin_clzll((uint64_t)x); } +#endif + +#if defined(__OPENCL_VERSION__) +SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { + int i = 0; + for (; i < 8 && (x & 1) == 0; i++, x >>= 1) ; + return i; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { + int i = 0; + for (; i < 16 && (x & 1) == 0; i++, x >>= 1) ; + return i; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { + int i = 0; + for (; i < 32 && (x & 1) == 0; i++, x >>= 1) ; + return i; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { + int i = 0; + for (; i < 64 && (x & 1) == 0; i++, x >>= 1) ; + return i; +} + +#elif defined(__CUDA_ARCH__) + +SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { + int y = __ffs(x); + return y == 0 ? 8 : y - 1; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { + int y = __ffs(x); + return y == 0 ? 16 : y - 1; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { + int y = __ffs(x); + return y == 0 ? 32 : y - 1; +} + +SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { + int y = __ffsll(x); + return y == 0 ? 64 : y - 1; +} + +#elif defined(ISPC) + +SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { return x == 0 ? 8 : count_trailing_zeros((int32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { return x == 0 ? 16 : count_trailing_zeros((int32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { return count_trailing_zeros(x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { return count_trailing_zeros(x); } + +#else // Not OpenCL or CUDA, but plain C. + +SCALAR_FUN_ATTR int32_t futrts_ctzz8(int8_t x) { return x == 0 ? 8 : __builtin_ctz((uint32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz16(int16_t x) { return x == 0 ? 16 : __builtin_ctz((uint32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz32(int32_t x) { return x == 0 ? 32 : __builtin_ctz((uint32_t)x); } +SCALAR_FUN_ATTR int32_t futrts_ctzz64(int64_t x) { return x == 0 ? 64 : __builtin_ctzll((uint64_t)x); } +#endif + +SCALAR_FUN_ATTR float fdiv32(float x, float y) { return x / y; } +SCALAR_FUN_ATTR float fadd32(float x, float y) { return x + y; } +SCALAR_FUN_ATTR float fsub32(float x, float y) { return x - y; } +SCALAR_FUN_ATTR float fmul32(float x, float y) { return x * y; } +SCALAR_FUN_ATTR bool cmplt32(float x, float y) { return x < y; } +SCALAR_FUN_ATTR bool cmple32(float x, float y) { return x <= y; } +SCALAR_FUN_ATTR float sitofp_i8_f32(int8_t x) { return (float) x; } + +SCALAR_FUN_ATTR float sitofp_i16_f32(int16_t x) { return (float) x; } +SCALAR_FUN_ATTR float sitofp_i32_f32(int32_t x) { return (float) x; } +SCALAR_FUN_ATTR float sitofp_i64_f32(int64_t x) { return (float) x; } +SCALAR_FUN_ATTR float uitofp_i8_f32(uint8_t x) { return (float) x; } +SCALAR_FUN_ATTR float uitofp_i16_f32(uint16_t x) { return (float) x; } +SCALAR_FUN_ATTR float uitofp_i32_f32(uint32_t x) { return (float) x; } +SCALAR_FUN_ATTR float uitofp_i64_f32(uint64_t x) { return (float) x; } + +#ifdef __OPENCL_VERSION__ +SCALAR_FUN_ATTR float fabs32(float x) { return fabs(x); } +SCALAR_FUN_ATTR float fmax32(float x, float y) { return fmax(x, y); } +SCALAR_FUN_ATTR float fmin32(float x, float y) { return fmin(x, y); } +SCALAR_FUN_ATTR float fpow32(float x, float y) { return pow(x, y); } + +#elif defined(ISPC) + +SCALAR_FUN_ATTR float fabs32(float x) { return abs(x); } +SCALAR_FUN_ATTR float fmax32(float x, float y) { return isnan(x) ? y : isnan(y) ? x : max(x, y); } +SCALAR_FUN_ATTR float fmin32(float x, float y) { return isnan(x) ? y : isnan(y) ? x : min(x, y); } +SCALAR_FUN_ATTR float fpow32(float a, float b) { + float ret; + foreach_active (i) { + uniform float r = pow(extract(a, i), extract(b, i)); + ret = insert(ret, i, r); + } + return ret; +} + +#else // Not OpenCL, but CUDA or plain C. + +SCALAR_FUN_ATTR float fabs32(float x) { return fabsf(x); } +SCALAR_FUN_ATTR float fmax32(float x, float y) { return fmaxf(x, y); } +SCALAR_FUN_ATTR float fmin32(float x, float y) { return fminf(x, y); } +SCALAR_FUN_ATTR float fpow32(float x, float y) { return powf(x, y); } +#endif + +SCALAR_FUN_ATTR bool futrts_isnan32(float x) { return isnan(x); } + +#if defined(ISPC) + +SCALAR_FUN_ATTR bool futrts_isinf32(float x) { return !isnan(x) && isnan(x - x); } + +SCALAR_FUN_ATTR bool futrts_isfinite32(float x) { return !isnan(x) && !futrts_isinf32(x); } + +#else + +SCALAR_FUN_ATTR bool futrts_isinf32(float x) { return isinf(x); } + +#endif + +SCALAR_FUN_ATTR int8_t fptosi_f32_i8(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (int8_t) x; + } +} + +SCALAR_FUN_ATTR int16_t fptosi_f32_i16(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (int16_t) x; + } +} + +SCALAR_FUN_ATTR int32_t fptosi_f32_i32(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (int32_t) x; + } +} + +SCALAR_FUN_ATTR int64_t fptosi_f32_i64(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (int64_t) x; + }; +} + +SCALAR_FUN_ATTR uint8_t fptoui_f32_i8(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (uint8_t) (int8_t) x; + } +} + +SCALAR_FUN_ATTR uint16_t fptoui_f32_i16(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (uint16_t) (int16_t) x; + } +} + +SCALAR_FUN_ATTR uint32_t fptoui_f32_i32(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (uint32_t) (int32_t) x; + } +} + +SCALAR_FUN_ATTR uint64_t fptoui_f32_i64(float x) { + if (futrts_isnan32(x) || futrts_isinf32(x)) { + return 0; + } else { + return (uint64_t) (int64_t) x; + } +} + +SCALAR_FUN_ATTR bool ftob_f32_bool(float x) { return x != 0; } +SCALAR_FUN_ATTR float btof_bool_f32(bool x) { return x ? 1 : 0; } + +#ifdef __OPENCL_VERSION__ +SCALAR_FUN_ATTR float futrts_log32(float x) { return log(x); } +SCALAR_FUN_ATTR float futrts_log2_32(float x) { return log2(x); } +SCALAR_FUN_ATTR float futrts_log10_32(float x) { return log10(x); } +SCALAR_FUN_ATTR float futrts_log1p_32(float x) { return log1p(x); } +SCALAR_FUN_ATTR float futrts_sqrt32(float x) { return sqrt(x); } +SCALAR_FUN_ATTR float futrts_rsqrt32(float x) { return rsqrt(x); } +SCALAR_FUN_ATTR float futrts_cbrt32(float x) { return cbrt(x); } +SCALAR_FUN_ATTR float futrts_exp32(float x) { return exp(x); } +SCALAR_FUN_ATTR float futrts_cos32(float x) { return cos(x); } +SCALAR_FUN_ATTR float futrts_cospi32(float x) { return cospi(x); } +SCALAR_FUN_ATTR float futrts_sin32(float x) { return sin(x); } +SCALAR_FUN_ATTR float futrts_sinpi32(float x) { return sinpi(x); } +SCALAR_FUN_ATTR float futrts_tan32(float x) { return tan(x); } +SCALAR_FUN_ATTR float futrts_tanpi32(float x) { return tanpi(x); } +SCALAR_FUN_ATTR float futrts_acos32(float x) { return acos(x); } +SCALAR_FUN_ATTR float futrts_acospi32(float x) { return acospi(x); } +SCALAR_FUN_ATTR float futrts_asin32(float x) { return asin(x); } +SCALAR_FUN_ATTR float futrts_asinpi32(float x) { return asinpi(x); } +SCALAR_FUN_ATTR float futrts_atan32(float x) { return atan(x); } +SCALAR_FUN_ATTR float futrts_atanpi32(float x) { return atanpi(x); } +SCALAR_FUN_ATTR float futrts_cosh32(float x) { return cosh(x); } +SCALAR_FUN_ATTR float futrts_sinh32(float x) { return sinh(x); } +SCALAR_FUN_ATTR float futrts_tanh32(float x) { return tanh(x); } +SCALAR_FUN_ATTR float futrts_acosh32(float x) { return acosh(x); } +SCALAR_FUN_ATTR float futrts_asinh32(float x) { return asinh(x); } +SCALAR_FUN_ATTR float futrts_atanh32(float x) { return atanh(x); } +SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) { return atan2(x, y); } +SCALAR_FUN_ATTR float futrts_atan2pi_32(float x, float y) { return atan2pi(x, y); } +SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { return hypot(x, y); } +SCALAR_FUN_ATTR float futrts_gamma32(float x) { return tgamma(x); } +SCALAR_FUN_ATTR float futrts_lgamma32(float x) { return lgamma(x); } +SCALAR_FUN_ATTR float futrts_erf32(float x) { return erf(x); } +SCALAR_FUN_ATTR float futrts_erfc32(float x) { return erfc(x); } +SCALAR_FUN_ATTR float fmod32(float x, float y) { return fmod(x, y); } +SCALAR_FUN_ATTR float futrts_round32(float x) { return rint(x); } +SCALAR_FUN_ATTR float futrts_floor32(float x) { return floor(x); } +SCALAR_FUN_ATTR float futrts_ceil32(float x) { return ceil(x); } +SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { return nextafter(x, y); } +SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { return mix(v0, v1, t); } +SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { return ldexp(x, y); } +SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { return copysign(x, y); } +SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { return mad(a, b, c); } +SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { return fma(a, b, c); } + +#elif defined(ISPC) + +SCALAR_FUN_ATTR float futrts_log32(float x) { return futrts_isfinite32(x) || (futrts_isinf32(x) && x < 0)? log(x) : x; } +SCALAR_FUN_ATTR float futrts_log2_32(float x) { return futrts_log32(x) / log(2.0f); } +SCALAR_FUN_ATTR float futrts_log10_32(float x) { return futrts_log32(x) / log(10.0f); } + +SCALAR_FUN_ATTR float futrts_log1p_32(float x) { + if(x == -1.0f || (futrts_isinf32(x) && x > 0.0f)) return x / 0.0f; + float y = 1.0f + x; + float z = y - 1.0f; + return log(y) - (z-x)/y; +} + +SCALAR_FUN_ATTR float futrts_sqrt32(float x) { return sqrt(x); } +SCALAR_FUN_ATTR float futrts_rsqrt32(float x) { return 1/sqrt(x); } + +extern "C" unmasked uniform float cbrtf(uniform float); +SCALAR_FUN_ATTR float futrts_cbrt32(float x) { + float res; + foreach_active (i) { + uniform float r = cbrtf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR float futrts_exp32(float x) { return exp(x); } +SCALAR_FUN_ATTR float futrts_cos32(float x) { return cos(x); } +SCALAR_FUN_ATTR float futrts_cospi32(float x) { return cos((float)M_PI*x); } +SCALAR_FUN_ATTR float futrts_sin32(float x) { return sin(x); } +SCALAR_FUN_ATTR float futrts_sinpi32(float x) { return sin(M_PI*x); } +SCALAR_FUN_ATTR float futrts_tan32(float x) { return tan(x); } +SCALAR_FUN_ATTR float futrts_tanpi32(float x) { return tan((float)M_PI*x); } +SCALAR_FUN_ATTR float futrts_acos32(float x) { return acos(x); } +SCALAR_FUN_ATTR float futrts_acospi32(float x) { return acos(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_asin32(float x) { return asin(x); } +SCALAR_FUN_ATTR float futrts_asinpi32(float x) { return asin(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_atan32(float x) { return atan(x); } +SCALAR_FUN_ATTR float futrts_atanpi32(float x) { return atan(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_cosh32(float x) { return (exp(x)+exp(-x)) / 2.0f; } +SCALAR_FUN_ATTR float futrts_sinh32(float x) { return (exp(x)-exp(-x)) / 2.0f; } +SCALAR_FUN_ATTR float futrts_tanh32(float x) { return futrts_sinh32(x)/futrts_cosh32(x); } + +SCALAR_FUN_ATTR float futrts_acosh32(float x) { + float f = x+sqrt(x*x-1); + if (futrts_isfinite32(f)) return log(f); + return f; +} + +SCALAR_FUN_ATTR float futrts_asinh32(float x) { + float f = x+sqrt(x*x+1); + if (futrts_isfinite32(f)) return log(f); + return f; +} + +SCALAR_FUN_ATTR float futrts_atanh32(float x) { + float f = (1+x)/(1-x); + if (futrts_isfinite32(f)) return log(f)/2.0f; + return f; +} + +SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) +{ return (x == 0.0f && y == 0.0f) ? 0.0f : atan2(x, y); } +SCALAR_FUN_ATTR float futrts_atan2pi_32(float x, float y) +{ return (x == 0.0f && y == 0.0f) ? 0.0f : atan2(x, y) / (float)M_PI; } + +SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { + if (futrts_isfinite32(x) && futrts_isfinite32(y)) { + x = abs(x); + y = abs(y); + float a; + float b; + if (x >= y){ + a = x; + b = y; + } else { + a = y; + b = x; + } + if(b == 0){ + return a; + } + + int e; + float an; + float bn; + an = frexp (a, &e); + bn = ldexp (b, - e); + float cn; + cn = sqrt (an * an + bn * bn); + return ldexp (cn, e); + } else { + if (futrts_isinf32(x) || futrts_isinf32(y)) return INFINITY; + else return x + y; + } + +} + +extern "C" unmasked uniform float tgammaf(uniform float x); +SCALAR_FUN_ATTR float futrts_gamma32(float x) { + float res; + foreach_active (i) { + uniform float r = tgammaf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform float lgammaf(uniform float x); +SCALAR_FUN_ATTR float futrts_lgamma32(float x) { + float res; + foreach_active (i) { + uniform float r = lgammaf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform float erff(uniform float x); +SCALAR_FUN_ATTR float futrts_erf32(float x) { + float res; + foreach_active (i) { + uniform float r = erff(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform float erfcf(uniform float x); +SCALAR_FUN_ATTR float futrts_erfc32(float x) { + float res; + foreach_active (i) { + uniform float r = erfcf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR float fmod32(float x, float y) { return x - y * trunc(x/y); } +SCALAR_FUN_ATTR float futrts_round32(float x) { return round(x); } +SCALAR_FUN_ATTR float futrts_floor32(float x) { return floor(x); } +SCALAR_FUN_ATTR float futrts_ceil32(float x) { return ceil(x); } + +extern "C" unmasked uniform float nextafterf(uniform float x, uniform float y); +SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { + float res; + foreach_active (i) { + uniform float r = nextafterf(extract(x, i), extract(y, i)); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { + return v0 + (v1 - v0) * t; +} + +SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { + return x * pow((uniform float)2.0, (float)y); +} + +SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { + int32_t xb = futrts_to_bits32(x); + int32_t yb = futrts_to_bits32(y); + return futrts_from_bits32((xb & ~(1<<31)) | (yb & (1<<31))); +} + +SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { + return a * b + c; +} + +SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { + return a * b + c; +} + +#else // Not OpenCL or ISPC, but CUDA or plain C. + +SCALAR_FUN_ATTR float futrts_log32(float x) { return logf(x); } +SCALAR_FUN_ATTR float futrts_log2_32(float x) { return log2f(x); } +SCALAR_FUN_ATTR float futrts_log10_32(float x) { return log10f(x); } +SCALAR_FUN_ATTR float futrts_log1p_32(float x) { return log1pf(x); } +SCALAR_FUN_ATTR float futrts_sqrt32(float x) { return sqrtf(x); } +SCALAR_FUN_ATTR float futrts_rsqrt32(float x) { return 1/sqrtf(x); } +SCALAR_FUN_ATTR float futrts_cbrt32(float x) { return cbrtf(x); } +SCALAR_FUN_ATTR float futrts_exp32(float x) { return expf(x); } +SCALAR_FUN_ATTR float futrts_cos32(float x) { return cosf(x); } + +SCALAR_FUN_ATTR float futrts_cospi32(float x) { +#if defined(__CUDA_ARCH__) + return cospif(x); +#else + return cosf(((float)M_PI)*x); +#endif +} +SCALAR_FUN_ATTR float futrts_sin32(float x) { return sinf(x); } + +SCALAR_FUN_ATTR float futrts_sinpi32(float x) { +#if defined(__CUDA_ARCH__) + return sinpif(x); +#else + return sinf((float)M_PI*x); +#endif +} + +SCALAR_FUN_ATTR float futrts_tan32(float x) { return tanf(x); } +SCALAR_FUN_ATTR float futrts_tanpi32(float x) { return tanf((float)M_PI*x); } +SCALAR_FUN_ATTR float futrts_acos32(float x) { return acosf(x); } +SCALAR_FUN_ATTR float futrts_acospi32(float x) { return acosf(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_asin32(float x) { return asinf(x); } +SCALAR_FUN_ATTR float futrts_asinpi32(float x) { return asinf(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_atan32(float x) { return atanf(x); } +SCALAR_FUN_ATTR float futrts_atanpi32(float x) { return atanf(x)/(float)M_PI; } +SCALAR_FUN_ATTR float futrts_cosh32(float x) { return coshf(x); } +SCALAR_FUN_ATTR float futrts_sinh32(float x) { return sinhf(x); } +SCALAR_FUN_ATTR float futrts_tanh32(float x) { return tanhf(x); } +SCALAR_FUN_ATTR float futrts_acosh32(float x) { return acoshf(x); } +SCALAR_FUN_ATTR float futrts_asinh32(float x) { return asinhf(x); } +SCALAR_FUN_ATTR float futrts_atanh32(float x) { return atanhf(x); } +SCALAR_FUN_ATTR float futrts_atan2_32(float x, float y) { return atan2f(x, y); } +SCALAR_FUN_ATTR float futrts_atan2pi_32(float x, float y) { return atan2f(x, y) / (float)M_PI; } +SCALAR_FUN_ATTR float futrts_hypot32(float x, float y) { return hypotf(x, y); } +SCALAR_FUN_ATTR float futrts_gamma32(float x) { return tgammaf(x); } +SCALAR_FUN_ATTR float futrts_lgamma32(float x) { return lgammaf(x); } +SCALAR_FUN_ATTR float futrts_erf32(float x) { return erff(x); } +SCALAR_FUN_ATTR float futrts_erfc32(float x) { return erfcf(x); } +SCALAR_FUN_ATTR float fmod32(float x, float y) { return fmodf(x, y); } +SCALAR_FUN_ATTR float futrts_round32(float x) { return rintf(x); } +SCALAR_FUN_ATTR float futrts_floor32(float x) { return floorf(x); } +SCALAR_FUN_ATTR float futrts_ceil32(float x) { return ceilf(x); } +SCALAR_FUN_ATTR float futrts_nextafter32(float x, float y) { return nextafterf(x, y); } +SCALAR_FUN_ATTR float futrts_lerp32(float v0, float v1, float t) { return v0 + (v1 - v0) * t; } +SCALAR_FUN_ATTR float futrts_ldexp32(float x, int32_t y) { return ldexpf(x, y); } +SCALAR_FUN_ATTR float futrts_copysign32(float x, float y) { return copysignf(x, y); } +SCALAR_FUN_ATTR float futrts_mad32(float a, float b, float c) { return a * b + c; } +SCALAR_FUN_ATTR float futrts_fma32(float a, float b, float c) { return fmaf(a, b, c); } + +#endif + +#if defined(ISPC) + +SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x) { return intbits(x); } +SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x) { return floatbits(x); } + +#else + +SCALAR_FUN_ATTR int32_t futrts_to_bits32(float x) { + union { + float f; + int32_t t; + } p; + + p.f = x; + return p.t; +} + +SCALAR_FUN_ATTR float futrts_from_bits32(int32_t x) { + union { + int32_t f; + float t; + } p; + + p.f = x; + return p.t; +} +#endif + +SCALAR_FUN_ATTR float fsignum32(float x) { + return futrts_isnan32(x) ? x : (x > 0 ? 1 : 0) - (x < 0 ? 1 : 0); +} + +#ifdef FUTHARK_F64_ENABLED + +SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x); +SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x); + +#if defined(ISPC) + +SCALAR_FUN_ATTR bool futrts_isinf64(float x) { return !isnan(x) && isnan(x - x); } +SCALAR_FUN_ATTR bool futrts_isfinite64(float x) { return !isnan(x) && !futrts_isinf64(x); } +SCALAR_FUN_ATTR double fdiv64(double x, double y) { return x / y; } +SCALAR_FUN_ATTR double fadd64(double x, double y) { return x + y; } +SCALAR_FUN_ATTR double fsub64(double x, double y) { return x - y; } +SCALAR_FUN_ATTR double fmul64(double x, double y) { return x * y; } +SCALAR_FUN_ATTR bool cmplt64(double x, double y) { return x < y; } +SCALAR_FUN_ATTR bool cmple64(double x, double y) { return x <= y; } +SCALAR_FUN_ATTR double sitofp_i8_f64(int8_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i16_f64(int16_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i32_f64(int32_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i64_f64(int64_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i8_f64(uint8_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i16_f64(uint16_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i32_f64(uint32_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i64_f64(uint64_t x) { return (double) x; } +SCALAR_FUN_ATTR double fabs64(double x) { return abs(x); } +SCALAR_FUN_ATTR double fmax64(double x, double y) { return isnan(x) ? y : isnan(y) ? x : max(x, y); } +SCALAR_FUN_ATTR double fmin64(double x, double y) { return isnan(x) ? y : isnan(y) ? x : min(x, y); } + +SCALAR_FUN_ATTR double fpow64(double a, double b) { + float ret; + foreach_active (i) { + uniform float r = pow(extract(a, i), extract(b, i)); + ret = insert(ret, i, r); + } + return ret; +} +SCALAR_FUN_ATTR double futrts_log64(double x) { return futrts_isfinite64(x) || (futrts_isinf64(x) && x < 0)? log(x) : x; } +SCALAR_FUN_ATTR double futrts_log2_64(double x) { return futrts_log64(x)/log(2.0d); } +SCALAR_FUN_ATTR double futrts_log10_64(double x) { return futrts_log64(x)/log(10.0d); } + +SCALAR_FUN_ATTR double futrts_log1p_64(double x) { + if(x == -1.0d || (futrts_isinf64(x) && x > 0.0d)) return x / 0.0d; + double y = 1.0d + x; + double z = y - 1.0d; + return log(y) - (z-x)/y; +} + +SCALAR_FUN_ATTR double futrts_sqrt64(double x) { return sqrt(x); } +SCALAR_FUN_ATTR double futrts_rsqrt64(double x) { return 1/sqrt(x); } + +extern "C" unmasked uniform double cbrt(uniform double); +SCALAR_FUN_ATTR double futrts_cbrt64(double x) { + double res; + foreach_active (i) { + uniform double r = cbrtf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} +SCALAR_FUN_ATTR double futrts_exp64(double x) { return exp(x); } +SCALAR_FUN_ATTR double futrts_cos64(double x) { return cos(x); } +SCALAR_FUN_ATTR double futrts_cospi64(double x) { return cos(M_PI*x); } +SCALAR_FUN_ATTR double futrts_sin64(double x) { return sin(x); } +SCALAR_FUN_ATTR double futrts_sinpi64(double x) { return sin(M_PI*x); } +SCALAR_FUN_ATTR double futrts_tan64(double x) { return tan(x); } +SCALAR_FUN_ATTR double futrts_tanpi64(double x) { return tan(M_PI*x); } +SCALAR_FUN_ATTR double futrts_acos64(double x) { return acos(x); } +SCALAR_FUN_ATTR double futrts_acospi64(double x) { return acos(x)/M_PI; } +SCALAR_FUN_ATTR double futrts_asin64(double x) { return asin(x); } +SCALAR_FUN_ATTR double futrts_asinpi64(double x) { return asin(x)/M_PI; } +SCALAR_FUN_ATTR double futrts_atan64(double x) { return atan(x); } +SCALAR_FUN_ATTR double futrts_atanpi64(double x) { return atan(x)/M_PI; } +SCALAR_FUN_ATTR double futrts_cosh64(double x) { return (exp(x)+exp(-x)) / 2.0d; } +SCALAR_FUN_ATTR double futrts_sinh64(double x) { return (exp(x)-exp(-x)) / 2.0d; } +SCALAR_FUN_ATTR double futrts_tanh64(double x) { return futrts_sinh64(x)/futrts_cosh64(x); } + +SCALAR_FUN_ATTR double futrts_acosh64(double x) { + double f = x+sqrt(x*x-1.0d); + if(futrts_isfinite64(f)) return log(f); + return f; +} + +SCALAR_FUN_ATTR double futrts_asinh64(double x) { + double f = x+sqrt(x*x+1.0d); + if(futrts_isfinite64(f)) return log(f); + return f; +} + +SCALAR_FUN_ATTR double futrts_atanh64(double x) { + double f = (1.0d+x)/(1.0d-x); + if(futrts_isfinite64(f)) return log(f)/2.0d; + return f; +} +SCALAR_FUN_ATTR double futrts_atan2_64(double x, double y) { return atan2(x, y); } + +SCALAR_FUN_ATTR double futrts_atan2pi_64(double x, double y) { return atan2(x, y) / M_PI; } + +extern "C" unmasked uniform double hypot(uniform double x, uniform double y); +SCALAR_FUN_ATTR double futrts_hypot64(double x, double y) { + double res; + foreach_active (i) { + uniform double r = hypot(extract(x, i), extract(y, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform double tgamma(uniform double x); +SCALAR_FUN_ATTR double futrts_gamma64(double x) { + double res; + foreach_active (i) { + uniform double r = tgamma(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform double lgamma(uniform double x); +SCALAR_FUN_ATTR double futrts_lgamma64(double x) { + double res; + foreach_active (i) { + uniform double r = lgamma(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform double erf(uniform double x); +SCALAR_FUN_ATTR double futrts_erf64(double x) { + double res; + foreach_active (i) { + uniform double r = erf(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +extern "C" unmasked uniform double erfc(uniform double x); +SCALAR_FUN_ATTR double futrts_erfc64(double x) { + double res; + foreach_active (i) { + uniform double r = erfc(extract(x, i)); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR double futrts_fma64(double a, double b, double c) { return a * b + c; } +SCALAR_FUN_ATTR double futrts_round64(double x) { return round(x); } +SCALAR_FUN_ATTR double futrts_ceil64(double x) { return ceil(x); } + +extern "C" unmasked uniform double nextafter(uniform float x, uniform double y); +SCALAR_FUN_ATTR float futrts_nextafter64(double x, double y) { + double res; + foreach_active (i) { + uniform double r = nextafter(extract(x, i), extract(y, i)); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR double futrts_floor64(double x) { return floor(x); } +SCALAR_FUN_ATTR bool futrts_isnan64(double x) { return isnan(x); } + +SCALAR_FUN_ATTR int8_t fptosi_f64_i8(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int8_t) x; + } +} + +SCALAR_FUN_ATTR int16_t fptosi_f64_i16(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int16_t) x; + } +} + +SCALAR_FUN_ATTR int32_t fptosi_f64_i32(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int32_t) x; + } +} + +SCALAR_FUN_ATTR int64_t fptosi_f64_i64(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int64_t) x; + } +} + +SCALAR_FUN_ATTR uint8_t fptoui_f64_i8(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint8_t) (int8_t) x; + } +} + +SCALAR_FUN_ATTR uint16_t fptoui_f64_i16(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint16_t) (int16_t) x; + } +} + +SCALAR_FUN_ATTR uint32_t fptoui_f64_i32(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint32_t) (int32_t) x; + } +} + +SCALAR_FUN_ATTR uint64_t fptoui_f64_i64(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint64_t) (int64_t) x; + } +} + +SCALAR_FUN_ATTR bool ftob_f64_bool(double x) { return x != 0.0; } +SCALAR_FUN_ATTR double btof_bool_f64(bool x) { return x ? 1.0 : 0.0; } + +SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x) { + int64_t res; + foreach_active (i) { + uniform double tmp = extract(x, i); + uniform int64_t r = *((uniform int64_t* uniform)&tmp); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x) { + double res; + foreach_active (i) { + uniform int64_t tmp = extract(x, i); + uniform double r = *((uniform double* uniform)&tmp); + res = insert(res, i, r); + } + return res; +} + +SCALAR_FUN_ATTR double fmod64(double x, double y) { + return x - y * trunc(x/y); +} + +SCALAR_FUN_ATTR double fsignum64(double x) { + return futrts_isnan64(x) ? x : (x > 0 ? 1.0d : 0.0d) - (x < 0 ? 1.0d : 0.0d); +} + +SCALAR_FUN_ATTR double futrts_lerp64(double v0, double v1, double t) { + return v0 + (v1 - v0) * t; +} + +SCALAR_FUN_ATTR double futrts_ldexp64(double x, int32_t y) { + return x * pow((uniform double)2.0, (double)y); +} + +SCALAR_FUN_ATTR double futrts_copysign64(double x, double y) { + int64_t xb = futrts_to_bits64(x); + int64_t yb = futrts_to_bits64(y); + return futrts_from_bits64((xb & ~(((int64_t)1)<<63)) | (yb & (((int64_t)1)<<63))); +} + +SCALAR_FUN_ATTR double futrts_mad64(double a, double b, double c) { return a * b + c; } +SCALAR_FUN_ATTR float fpconv_f32_f32(float x) { return (float) x; } +SCALAR_FUN_ATTR double fpconv_f32_f64(float x) { return (double) x; } +SCALAR_FUN_ATTR float fpconv_f64_f32(double x) { return (float) x; } +SCALAR_FUN_ATTR double fpconv_f64_f64(double x) { return (double) x; } + +#else + +SCALAR_FUN_ATTR double fdiv64(double x, double y) { return x / y; } +SCALAR_FUN_ATTR double fadd64(double x, double y) { return x + y; } +SCALAR_FUN_ATTR double fsub64(double x, double y) { return x - y; } +SCALAR_FUN_ATTR double fmul64(double x, double y) { return x * y; } +SCALAR_FUN_ATTR bool cmplt64(double x, double y) { return x < y; } +SCALAR_FUN_ATTR bool cmple64(double x, double y) { return x <= y; } +SCALAR_FUN_ATTR double sitofp_i8_f64(int8_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i16_f64(int16_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i32_f64(int32_t x) { return (double) x; } +SCALAR_FUN_ATTR double sitofp_i64_f64(int64_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i8_f64(uint8_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i16_f64(uint16_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i32_f64(uint32_t x) { return (double) x; } +SCALAR_FUN_ATTR double uitofp_i64_f64(uint64_t x) { return (double) x; } +SCALAR_FUN_ATTR double fabs64(double x) { return fabs(x); } +SCALAR_FUN_ATTR double fmax64(double x, double y) { return fmax(x, y); } +SCALAR_FUN_ATTR double fmin64(double x, double y) { return fmin(x, y); } +SCALAR_FUN_ATTR double fpow64(double x, double y) { return pow(x, y); } +SCALAR_FUN_ATTR double futrts_log64(double x) { return log(x); } +SCALAR_FUN_ATTR double futrts_log2_64(double x) { return log2(x); } +SCALAR_FUN_ATTR double futrts_log10_64(double x) { return log10(x); } +SCALAR_FUN_ATTR double futrts_log1p_64(double x) { return log1p(x); } +SCALAR_FUN_ATTR double futrts_sqrt64(double x) { return sqrt(x); } +SCALAR_FUN_ATTR double futrts_rsqrt64(double x) { return 1/sqrt(x); } +SCALAR_FUN_ATTR double futrts_cbrt64(double x) { return cbrt(x); } +SCALAR_FUN_ATTR double futrts_exp64(double x) { return exp(x); } +SCALAR_FUN_ATTR double futrts_cos64(double x) { return cos(x); } + +SCALAR_FUN_ATTR double futrts_cospi64(double x) { +#ifdef __OPENCL_VERSION__ + return cospi(x); +#elif defined(__CUDA_ARCH__) + return cospi(x); +#else + return cos(M_PI*x); +#endif +} + +SCALAR_FUN_ATTR double futrts_sin64(double x) { + return sin(x); +} + +SCALAR_FUN_ATTR double futrts_sinpi64(double x) { +#ifdef __OPENCL_VERSION__ + return sinpi(x); +#elif defined(__CUDA_ARCH__) + return sinpi(x); +#else + return sin(M_PI*x); +#endif +} + +SCALAR_FUN_ATTR double futrts_tan64(double x) { + return tan(x); +} + +SCALAR_FUN_ATTR double futrts_tanpi64(double x) { +#ifdef __OPENCL_VERSION__ + return tanpi(x); +#else + return tan(M_PI*x); +#endif +} + +SCALAR_FUN_ATTR double futrts_acos64(double x) { + return acos(x); +} + +SCALAR_FUN_ATTR double futrts_acospi64(double x) { +#ifdef __OPENCL_VERSION__ + return acospi(x); +#else + return acos(x) / M_PI; +#endif +} + +SCALAR_FUN_ATTR double futrts_asin64(double x) { + return asin(x); +} + +SCALAR_FUN_ATTR double futrts_asinpi64(double x) { +#ifdef __OPENCL_VERSION__ + return asinpi(x); +#else + return asin(x) / M_PI; +#endif +} + +SCALAR_FUN_ATTR double futrts_atan64(double x) { + return atan(x); +} + +SCALAR_FUN_ATTR double futrts_atanpi64(double x) { +#ifdef __OPENCL_VERSION__ + return atanpi(x); +#else + return atan(x) / M_PI; +#endif +} + +SCALAR_FUN_ATTR double futrts_cosh64(double x) { return cosh(x); } +SCALAR_FUN_ATTR double futrts_sinh64(double x) { return sinh(x); } +SCALAR_FUN_ATTR double futrts_tanh64(double x) { return tanh(x); } +SCALAR_FUN_ATTR double futrts_acosh64(double x) { return acosh(x); } +SCALAR_FUN_ATTR double futrts_asinh64(double x) { return asinh(x); } +SCALAR_FUN_ATTR double futrts_atanh64(double x) { return atanh(x); } +SCALAR_FUN_ATTR double futrts_atan2_64(double x, double y) { return atan2(x, y); } + +SCALAR_FUN_ATTR double futrts_atan2pi_64(double x, double y) { +#ifdef __OPENCL_VERSION__ + return atan2pi(x, y); +#else + return atan2(x, y) / M_PI; +#endif +} + +SCALAR_FUN_ATTR double futrts_hypot64(double x, double y) { return hypot(x, y); } +SCALAR_FUN_ATTR double futrts_gamma64(double x) { return tgamma(x); } +SCALAR_FUN_ATTR double futrts_lgamma64(double x) { return lgamma(x); } +SCALAR_FUN_ATTR double futrts_erf64(double x) { return erf(x); } +SCALAR_FUN_ATTR double futrts_erfc64(double x) { return erfc(x); } +SCALAR_FUN_ATTR double futrts_fma64(double a, double b, double c) { return fma(a, b, c); } +SCALAR_FUN_ATTR double futrts_round64(double x) { return rint(x); } +SCALAR_FUN_ATTR double futrts_ceil64(double x) { return ceil(x); } +SCALAR_FUN_ATTR float futrts_nextafter64(float x, float y) { return nextafter(x, y); } +SCALAR_FUN_ATTR double futrts_floor64(double x) { return floor(x); } +SCALAR_FUN_ATTR bool futrts_isnan64(double x) { return isnan(x); } +SCALAR_FUN_ATTR bool futrts_isinf64(double x) { return isinf(x); } + +SCALAR_FUN_ATTR int8_t fptosi_f64_i8(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int8_t) x; + } +} + +SCALAR_FUN_ATTR int16_t fptosi_f64_i16(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int16_t) x; + } +} + +SCALAR_FUN_ATTR int32_t fptosi_f64_i32(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int32_t) x; + } +} + +SCALAR_FUN_ATTR int64_t fptosi_f64_i64(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (int64_t) x; + } +} + +SCALAR_FUN_ATTR uint8_t fptoui_f64_i8(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint8_t) (int8_t) x; + } +} + +SCALAR_FUN_ATTR uint16_t fptoui_f64_i16(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint16_t) (int16_t) x; + } +} + +SCALAR_FUN_ATTR uint32_t fptoui_f64_i32(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint32_t) (int32_t) x; + } +} + +SCALAR_FUN_ATTR uint64_t fptoui_f64_i64(double x) { + if (futrts_isnan64(x) || futrts_isinf64(x)) { + return 0; + } else { + return (uint64_t) (int64_t) x; + } +} + +SCALAR_FUN_ATTR bool ftob_f64_bool(double x) { return x != 0; } +SCALAR_FUN_ATTR double btof_bool_f64(bool x) { return x ? 1 : 0; } + +SCALAR_FUN_ATTR int64_t futrts_to_bits64(double x) { + union { + double f; + int64_t t; + } p; + + p.f = x; + return p.t; +} + +SCALAR_FUN_ATTR double futrts_from_bits64(int64_t x) { + union { + int64_t f; + double t; + } p; + + p.f = x; + return p.t; +} + +SCALAR_FUN_ATTR double fmod64(double x, double y) { + return fmod(x, y); +} + +SCALAR_FUN_ATTR double fsignum64(double x) { + return futrts_isnan64(x) ? x : (x > 0) - (x < 0); +} + +SCALAR_FUN_ATTR double futrts_lerp64(double v0, double v1, double t) { +#ifdef __OPENCL_VERSION__ + return mix(v0, v1, t); +#else + return v0 + (v1 - v0) * t; +#endif +} + +SCALAR_FUN_ATTR double futrts_ldexp64(double x, int32_t y) { + return ldexp(x, y); +} + +SCALAR_FUN_ATTR float futrts_copysign64(double x, double y) { + return copysign(x, y); +} + +SCALAR_FUN_ATTR double futrts_mad64(double a, double b, double c) { +#ifdef __OPENCL_VERSION__ + return mad(a, b, c); +#else + return a * b + c; +#endif +} + +SCALAR_FUN_ATTR float fpconv_f32_f32(float x) { return (float) x; } +SCALAR_FUN_ATTR double fpconv_f32_f64(float x) { return (double) x; } +SCALAR_FUN_ATTR float fpconv_f64_f32(double x) { return (float) x; } +SCALAR_FUN_ATTR double fpconv_f64_f64(double x) { return (double) x; } #endif diff --git a/rts/c/scalar_f16.h b/rts/c/scalar_f16.h --- a/rts/c/scalar_f16.h +++ b/rts/c/scalar_f16.h @@ -38,403 +38,156 @@ // Some of these functions convert to single precision because half // precision versions are not available. - -SCALAR_FUN_ATTR f16 fadd16(f16 x, f16 y) { - return x + y; -} - -SCALAR_FUN_ATTR f16 fsub16(f16 x, f16 y) { - return x - y; -} - -SCALAR_FUN_ATTR f16 fmul16(f16 x, f16 y) { - return x * y; -} - -SCALAR_FUN_ATTR bool cmplt16(f16 x, f16 y) { - return x < y; -} - -SCALAR_FUN_ATTR bool cmple16(f16 x, f16 y) { - return x <= y; -} - -SCALAR_FUN_ATTR f16 sitofp_i8_f16(int8_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 sitofp_i16_f16(int16_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 sitofp_i32_f16(int32_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 sitofp_i64_f16(int64_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 uitofp_i8_f16(uint8_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 uitofp_i16_f16(uint16_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 uitofp_i32_f16(uint32_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR f16 uitofp_i64_f16(uint64_t x) { - return (f16) x; -} - -SCALAR_FUN_ATTR int8_t fptosi_f16_i8(f16 x) { - return (int8_t) (float) x; -} - -SCALAR_FUN_ATTR int16_t fptosi_f16_i16(f16 x) { - return (int16_t) x; -} - -SCALAR_FUN_ATTR int32_t fptosi_f16_i32(f16 x) { - return (int32_t) x; -} - -SCALAR_FUN_ATTR int64_t fptosi_f16_i64(f16 x) { - return (int64_t) x; -} - -SCALAR_FUN_ATTR uint8_t fptoui_f16_i8(f16 x) { - return (uint8_t) (float) x; -} - -SCALAR_FUN_ATTR uint16_t fptoui_f16_i16(f16 x) { - return (uint16_t) x; -} - -SCALAR_FUN_ATTR uint32_t fptoui_f16_i32(f16 x) { - return (uint32_t) x; -} - -SCALAR_FUN_ATTR uint64_t fptoui_f16_i64(f16 x) { - return (uint64_t) x; -} - -SCALAR_FUN_ATTR bool ftob_f16_bool(f16 x) { - return x != (f16)0; -} - -SCALAR_FUN_ATTR f16 btof_bool_f16(bool x) { - return x ? 1 : 0; -} +SCALAR_FUN_ATTR f16 fadd16(f16 x, f16 y) { return x + y; } +SCALAR_FUN_ATTR f16 fsub16(f16 x, f16 y) { return x - y; } +SCALAR_FUN_ATTR f16 fmul16(f16 x, f16 y) { return x * y; } +SCALAR_FUN_ATTR bool cmplt16(f16 x, f16 y) { return x < y; } +SCALAR_FUN_ATTR bool cmple16(f16 x, f16 y) { return x <= y; } +SCALAR_FUN_ATTR f16 sitofp_i8_f16(int8_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 sitofp_i16_f16(int16_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 sitofp_i32_f16(int32_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 sitofp_i64_f16(int64_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 uitofp_i8_f16(uint8_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 uitofp_i16_f16(uint16_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 uitofp_i32_f16(uint32_t x) { return (f16) x; } +SCALAR_FUN_ATTR f16 uitofp_i64_f16(uint64_t x) { return (f16) x; } +SCALAR_FUN_ATTR int8_t fptosi_f16_i8(f16 x) { return (int8_t) (float) x; } +SCALAR_FUN_ATTR int16_t fptosi_f16_i16(f16 x) { return (int16_t) x; } +SCALAR_FUN_ATTR int32_t fptosi_f16_i32(f16 x) { return (int32_t) x; } +SCALAR_FUN_ATTR int64_t fptosi_f16_i64(f16 x) { return (int64_t) x; } +SCALAR_FUN_ATTR uint8_t fptoui_f16_i8(f16 x) { return (uint8_t) (float) x; } +SCALAR_FUN_ATTR uint16_t fptoui_f16_i16(f16 x) { return (uint16_t) x; } +SCALAR_FUN_ATTR uint32_t fptoui_f16_i32(f16 x) { return (uint32_t) x; } +SCALAR_FUN_ATTR uint64_t fptoui_f16_i64(f16 x) { return (uint64_t) x; } +SCALAR_FUN_ATTR bool ftob_f16_bool(f16 x) { return x != (f16)0; } +SCALAR_FUN_ATTR f16 btof_bool_f16(bool x) { return x ? 1 : 0; } #ifndef EMULATE_F16 -SCALAR_FUN_ATTR bool futrts_isnan16(f16 x) { - return isnan((float)x); -} -#ifdef __OPENCL_VERSION__ - -SCALAR_FUN_ATTR f16 fabs16(f16 x) { - return fabs(x); -} - -SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { - return fmax(x, y); -} +SCALAR_FUN_ATTR bool futrts_isnan16(f16 x) { return isnan((float)x); } -SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { - return fmin(x, y); -} +#ifdef __OPENCL_VERSION__ -SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { - return pow(x, y); -} +SCALAR_FUN_ATTR f16 fabs16(f16 x) { return fabs(x); } +SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { return fmax(x, y); } +SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { return fmin(x, y); } +SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { return pow(x, y); } #elif defined(ISPC) -SCALAR_FUN_ATTR f16 fabs16(f16 x) { - return abs(x); -} -SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { - return futrts_isnan16(x) ? y : futrts_isnan16(y) ? x : max(x, y); -} - -SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { - return futrts_isnan16(x) ? y : futrts_isnan16(y) ? x : min(x, y); -} - -SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { - return pow(x, y); -} +SCALAR_FUN_ATTR f16 fabs16(f16 x) { return abs(x); } +SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { return futrts_isnan16(x) ? y : futrts_isnan16(y) ? x : max(x, y); } +SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { return futrts_isnan16(x) ? y : futrts_isnan16(y) ? x : min(x, y); } +SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { return pow(x, y); } #else // Assuming CUDA. -SCALAR_FUN_ATTR f16 fabs16(f16 x) { - return fabsf(x); -} - -SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { - return fmaxf(x, y); -} - -SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { - return fminf(x, y); -} +SCALAR_FUN_ATTR f16 fabs16(f16 x) { return fabsf(x); } +SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { return fmaxf(x, y); } +SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { return fminf(x, y); } +SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { return powf(x, y); } -SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { - return powf(x, y); -} #endif #if defined(ISPC) -SCALAR_FUN_ATTR bool futrts_isinf16(float x) { - return !futrts_isnan16(x) && futrts_isnan16(x - x); -} -SCALAR_FUN_ATTR bool futrts_isfinite16(float x) { - return !futrts_isnan16(x) && !futrts_isinf16(x); -} - +SCALAR_FUN_ATTR bool futrts_isinf16(float x) { return !futrts_isnan16(x) && futrts_isnan16(x - x); } +SCALAR_FUN_ATTR bool futrts_isfinite16(float x) { return !futrts_isnan16(x) && !futrts_isinf16(x); } #else - -SCALAR_FUN_ATTR bool futrts_isinf16(f16 x) { - return isinf((float)x); -} +SCALAR_FUN_ATTR bool futrts_isinf16(f16 x) { return isinf((float)x); } #endif #ifdef __OPENCL_VERSION__ -SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { - return log(x); -} - -SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { - return log2(x); -} - -SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { - return log10(x); -} - -SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { - return log1p(x); -} - -SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { - return sqrt(x); -} - -SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { - return cbrt(x); -} - -SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { - return exp(x); -} - -SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { - return cos(x); -} - -SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { - return sin(x); -} - -SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { - return tan(x); -} - -SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { - return acos(x); -} - -SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { - return asin(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { - return atan(x); -} - -SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { - return cosh(x); -} - -SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { - return sinh(x); -} - -SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { - return tanh(x); -} - -SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { - return acosh(x); -} - -SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { - return asinh(x); -} - -SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { - return atanh(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { - return atan2(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { - return hypot(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { - return tgamma(x); -} - -SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { - return lgamma(x); -} - -SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { - return erf(x); -} - -SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { - return erfc(x); -} - -SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { - return fmod(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { - return rint(x); -} - -SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { - return floor(x); -} - -SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { - return ceil(x); -} - -SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { - return nextafter(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { - return mix(v0, v1, t); -} - -SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { - return ldexp(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { - return copysign(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { - return mad(a, b, c); -} +SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { return log(x); } +SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { return log2(x); } +SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { return log10(x); } +SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { return log1p(x); } +SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { return sqrt(x); } +SCALAR_FUN_ATTR f16 futrts_rsqrt16(f16 x) { return rsqrt(x); } +SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { return cbrt(x); } +SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { return exp(x); } +SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { return cos(x); } +SCALAR_FUN_ATTR f16 futrts_cospi16(f16 x) { return cospi(x); } +SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { return sin(x); } +SCALAR_FUN_ATTR f16 futrts_sinpi16(f16 x) { return sinpi(x); } +SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { return tan(x); } +SCALAR_FUN_ATTR f16 futrts_tanpi16(f16 x) { return tanpi(x); } +SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { return acos(x); } +SCALAR_FUN_ATTR f16 futrts_acospi16(f16 x) { return acospi(x); } +SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { return asin(x); } +SCALAR_FUN_ATTR f16 futrts_asinpi16(f16 x) { return asinpi(x); } +SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { return atan(x); } +SCALAR_FUN_ATTR f16 futrts_atanpi16(f16 x) { return atanpi(x); } +SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { return cosh(x); } +SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { return sinh(x); } +SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { return tanh(x); } +SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { return acosh(x); } +SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { return asinh(x); } +SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { return atanh(x); } +SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { return atan2(x, y); } +SCALAR_FUN_ATTR f16 futrts_atan2pi_16(f16 x, f16 y) { return atan2pi(x, y); } +SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { return hypot(x, y); } +SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { return tgamma(x); } +SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { return lgamma(x); } +SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { return erf(x); } +SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { return erfc(x); } +SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { return fmod(x, y); } +SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { return rint(x); } +SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { return floor(x); } +SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { return ceil(x); } +SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { return nextafter(x, y); } +SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { return mix(v0, v1, t); } +SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { return ldexp(x, y); } +SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { return copysign(x, y); } +SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { return mad(a, b, c); } +SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { return fma(a, b, c); } -SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { - return fma(a, b, c); -} #elif defined(ISPC) -SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { - return futrts_isfinite16(x) || (futrts_isinf16(x) && x < 0) ? log(x) : x; -} - -SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { - return futrts_log16(x) / log(2.0f16); -} - -SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { - return futrts_log16(x) / log(10.0f16); -} - +SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { return futrts_isfinite16(x) || (futrts_isinf16(x) && x < 0) ? log(x) : x; } +SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { return futrts_log16(x) / log(2.0f16); } +SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { return futrts_log16(x) / log(10.0f16); } SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { if(x == -1.0f16 || (futrts_isinf16(x) && x > 0.0f16)) return x / 0.0f16; f16 y = 1.0f16 + x; f16 z = y - 1.0f16; return log(y) - (z-x)/y; } - -SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { - return (float16)sqrt((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { - return exp(x); -} - -SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { - return (float16)cos((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { - return (float16)sin((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { - return (float16)tan((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { - return (float16)acos((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { - return (float16)asin((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { - return (float16)atan((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { - return (exp(x)+exp(-x)) / 2.0f16; -} - -SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { - return (exp(x)-exp(-x)) / 2.0f16; -} - -SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { - return futrts_sinh16(x)/futrts_cosh16(x); -} - +SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { return (float16)sqrt((float)x); } +SCALAR_FUN_ATTR f16 futrts_rsqrt16(f16 x) { return (float16)1/sqrt((float)x); } +SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { return exp(x); } +SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { return (float16)cos((float)x); } +SCALAR_FUN_ATTR f16 futrts_cospi16(f16 x) { return (float16)cos((float)M_PI*(float)x); } +SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { return (float16)sin((float)x); } +SCALAR_FUN_ATTR f16 futrts_sinpi16(f16 x) { return (float16)sin((float)M_PI*(float)x); } +SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { return (float16)tan((float)x); } +SCALAR_FUN_ATTR f16 futrts_tanpi16(f16 x) { return (float16)(tan((float)M_PI*(float)x)); } +SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { return (float16)acos((float)x); } +SCALAR_FUN_ATTR f16 futrts_acospi16(f16 x) { return (float16)(acos((float)x)/(float)M_PI); } +SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { return (float16)asin((float)x); } +SCALAR_FUN_ATTR f16 futrts_asinpi16(f16 x) { return (float16)(asin((float)x)/(float)M_PI); } +SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { return (float16)atan((float)x); } +SCALAR_FUN_ATTR f16 futrts_atanpi16(f16 x) { return (float16)(atan((float)x)/(float)M_PI); } +SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { return (exp(x)+exp(-x)) / 2.0f16; } +SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { return (exp(x)-exp(-x)) / 2.0f16; } +SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { return futrts_sinh16(x)/futrts_cosh16(x); } SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { float16 f = x+(float16)sqrt((float)(x*x-1)); if(futrts_isfinite16(f)) return log(f); return f; } - SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { float16 f = x+(float16)sqrt((float)(x*x+1)); if(futrts_isfinite16(f)) return log(f); return f; } - SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { float16 f = (1+x)/(1-x); if(futrts_isfinite16(f)) return log(f)/2.0f16; return f; } - -SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { - return (float16)atan2((float)x, (float)y); -} - -SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { - return (float16)futrts_hypot32((float)x, (float)y); -} +SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { return (float16)atan2((float)x, (float)y); } +SCALAR_FUN_ATTR f16 futrts_atan2pi_16(f16 x, f16 y) { return (float16)(atan2((float)x, (float)y)/(float)M_PI); } +SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { return (float16)futrts_hypot32((float)x, (float)y); } extern "C" unmasked uniform float tgammaf(uniform float x); SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { @@ -455,226 +208,77 @@ } return res; } - -SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { - f16 res = (f16)futrts_cbrt32((float)x); - return res; -} - -SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { - f16 res = (f16)futrts_erf32((float)x); - return res; -} - -SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { - f16 res = (f16)futrts_erfc32((float)x); - return res; -} - -SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { - return x - y * (float16)trunc((float) (x/y)); -} - -SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { - return (float16)round((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { - return (float16)floor((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { - return (float16)ceil((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { - return (float16)futrts_nextafter32((float)x, (float) y); -} - -SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { - return v0 + (v1 - v0) * t; -} - -SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { - return futrts_ldexp32((float)x, y); -} - -SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { - return futrts_copysign32((float)x, y); -} - -SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { - return a * b + c; -} - -SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { - return a * b + c; -} +SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { return (f16)futrts_cbrt32((float)x); } +SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { return (f16)futrts_erf32((float)x); } +SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { return (f16)futrts_erfc32((float)x); } +SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { return x - y * (float16)trunc((float) (x/y)); } +SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { return (float16)round((float)x); } +SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { return (float16)floor((float)x); } +SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { return (float16)ceil((float)x); } +SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { return (float16)futrts_nextafter32((float)x, (float) y); } +SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { return v0 + (v1 - v0) * t; } +SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { return futrts_ldexp32((float)x, y); } +SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { return futrts_copysign32((float)x, y); } +SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { return a * b + c; } +SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { return a * b + c; } #else // Assume CUDA. -SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { - return hlog(x); -} - -SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { - return hlog2(x); -} - -SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { - return hlog10(x); -} - -SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { - return (f16)log1pf((float)x); -} - -SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { - return hsqrt(x); -} - -SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { - return cbrtf(x); -} - -SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { - return hexp(x); -} - -SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { - return hcos(x); -} - -SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { - return hsin(x); -} - -SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { - return tanf(x); -} - -SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { - return acosf(x); -} - -SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { - return asinf(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { - return atanf(x); -} - -SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { - return coshf(x); -} - -SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { - return sinhf(x); -} - -SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { - return tanhf(x); -} - -SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { - return acoshf(x); -} - -SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { - return asinhf(x); -} - -SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { - return atanhf(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { - return atan2f(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { - return hypotf(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { - return tgammaf(x); -} - -SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { - return lgammaf(x); -} - -SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { - return erff(x); -} - -SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { - return erfcf(x); -} - -SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { - return fmodf(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { - return rintf(x); -} - -SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { - return hfloor(x); -} - -SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { - return hceil(x); -} - -SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { - return __ushort_as_half(halfbitsnextafter(__half_as_ushort(x), __half_as_ushort(y))); -} - -SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { - return v0 + (v1 - v0) * t; -} - -SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { - return futrts_ldexp32((float)x, y); -} - -SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { - return futrts_copysign32((float)x, y); -} - -SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { - return a * b + c; -} - -SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { - return fmaf(a, b, c); -} +SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { return hlog(x); } +SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { return hlog2(x); } +SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { return hlog10(x); } +SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { return (f16)log1pf((float)x); } +SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { return hsqrt(x); } +SCALAR_FUN_ATTR f16 futrts_rsqrt16(f16 x) { return hrsqrt(x); } +SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { return cbrtf(x); } +SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { return hexp(x); } +SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { return hcos(x); } +SCALAR_FUN_ATTR f16 futrts_cospi16(f16 x) { return hcos((f16)M_PI*x); } +SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { return hsin(x); } +SCALAR_FUN_ATTR f16 futrts_sinpi16(f16 x) { return hsin((f16)M_PI*x); } +SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { return tanf(x); } +SCALAR_FUN_ATTR f16 futrts_tanpi16(f16 x) { return tanf((f16)M_PI*x); } +SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { return acosf(x); } +SCALAR_FUN_ATTR f16 futrts_acospi16(f16 x) { return (f16)acosf(x)/(f16)M_PI; } +SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { return asinf(x); } +SCALAR_FUN_ATTR f16 futrts_asinpi16(f16 x) { return (f16)asinf(x)/(f16)M_PI; } +SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { return (f16)atanf(x); } +SCALAR_FUN_ATTR f16 futrts_atanpi16(f16 x) { return (f16)atanf(x)/(f16)M_PI; } +SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { return coshf(x); } +SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { return sinhf(x); } +SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { return tanhf(x); } +SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { return acoshf(x); } +SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { return asinhf(x); } +SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { return atanhf(x); } +SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { return (f16)atan2f(x, y); } +SCALAR_FUN_ATTR f16 futrts_atan2pi_16(f16 x, f16 y) { return (f16)atan2f(x, y)/(f16)M_PI; } +SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { return hypotf(x, y); } +SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { return tgammaf(x); } +SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { return lgammaf(x); } +SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { return erff(x); } +SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { return erfcf(x); } +SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { return fmodf(x, y); } +SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { return rintf(x); } +SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { return hfloor(x); } +SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { return hceil(x); } +SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { return __ushort_as_half(halfbitsnextafter(__half_as_ushort(x), __half_as_ushort(y))); } +SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { return v0 + (v1 - v0) * t; } +SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { return futrts_ldexp32((float)x, y); } +SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { return futrts_copysign32((float)x, y); } +SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { return a * b + c; } +SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { return fmaf(a, b, c); } #endif // The CUDA __half type cannot be put in unions for some reason, so we // use bespoke conversion functions instead. #ifdef __CUDA_ARCH__ -SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { - return __half_as_ushort(x); -} -SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { - return __ushort_as_half(x); -} +SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { return __half_as_ushort(x); } +SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { return __ushort_as_half(x); } #elif defined(ISPC) - -SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { - varying int16_t y = *((varying int16_t * uniform)&x); - return y; -} - -SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { - varying f16 y = *((varying f16 * uniform)&x); - return y; +SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { varying int16_t y = *((varying int16_t * uniform)&x); return y; } +SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { varying f16 y = *((varying f16 * uniform)&x); return y; } #else SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { union { @@ -699,169 +303,55 @@ #else // No native f16 - emulate. -SCALAR_FUN_ATTR f16 fabs16(f16 x) { - return fabs32(x); -} - -SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { - return fmax32(x, y); -} - -SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { - return fmin32(x, y); -} - -SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { - return fpow32(x, y); -} - -SCALAR_FUN_ATTR bool futrts_isnan16(f16 x) { - return futrts_isnan32(x); -} - -SCALAR_FUN_ATTR bool futrts_isinf16(f16 x) { - return futrts_isinf32(x); -} - -SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { - return futrts_log32(x); -} - -SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { - return futrts_log2_32(x); -} - -SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { - return futrts_log10_32(x); -} - -SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { - return futrts_log1p_32(x); -} - -SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { - return futrts_sqrt32(x); -} - -SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { - return futrts_cbrt32(x); -} - -SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { - return futrts_exp32(x); -} - -SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { - return futrts_cos32(x); -} - -SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { - return futrts_sin32(x); -} - -SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { - return futrts_tan32(x); -} - -SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { - return futrts_acos32(x); -} - -SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { - return futrts_asin32(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { - return futrts_atan32(x); -} - -SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { - return futrts_cosh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { - return futrts_sinh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { - return futrts_tanh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { - return futrts_acosh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { - return futrts_asinh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { - return futrts_atanh32(x); -} - -SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { - return futrts_atan2_32(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { - return futrts_hypot32(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { - return futrts_gamma32(x); -} - -SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { - return futrts_lgamma32(x); -} - -SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { - return futrts_erf32(x); -} - -SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { - return futrts_erfc32(x); -} - -SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { - return fmod32(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { - return futrts_round32(x); -} - -SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { - return futrts_floor32(x); -} - -SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { - return futrts_ceil32(x); -} - -SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { - return halfbits2float(halfbitsnextafter(float2halfbits(x), float2halfbits(y))); -} - -SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { - return futrts_lerp32(v0, v1, t); -} - -SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { - return futrts_ldexp32(x, y); -} - -SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { - return futrts_copysign32((float)x, y); -} - -SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { - return futrts_mad32(a, b, c); -} - -SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { - return futrts_fma32(a, b, c); -} +SCALAR_FUN_ATTR f16 fabs16(f16 x) { return fabs32(x); } +SCALAR_FUN_ATTR f16 fmax16(f16 x, f16 y) { return fmax32(x, y); } +SCALAR_FUN_ATTR f16 fmin16(f16 x, f16 y) { return fmin32(x, y); } +SCALAR_FUN_ATTR f16 fpow16(f16 x, f16 y) { return fpow32(x, y); } +SCALAR_FUN_ATTR bool futrts_isnan16(f16 x) { return futrts_isnan32(x); } +SCALAR_FUN_ATTR bool futrts_isinf16(f16 x) { return futrts_isinf32(x); } +SCALAR_FUN_ATTR f16 futrts_log16(f16 x) { return futrts_log32(x); } +SCALAR_FUN_ATTR f16 futrts_log2_16(f16 x) { return futrts_log2_32(x); } +SCALAR_FUN_ATTR f16 futrts_log10_16(f16 x) { return futrts_log10_32(x); } +SCALAR_FUN_ATTR f16 futrts_log1p_16(f16 x) { return futrts_log1p_32(x); } +SCALAR_FUN_ATTR f16 futrts_sqrt16(f16 x) { return futrts_sqrt32(x); } +SCALAR_FUN_ATTR f16 futrts_rsqrt16(f16 x) { return futrts_rsqrt32(x); } +SCALAR_FUN_ATTR f16 futrts_cbrt16(f16 x) { return futrts_cbrt32(x); } +SCALAR_FUN_ATTR f16 futrts_exp16(f16 x) { return futrts_exp32(x); } +SCALAR_FUN_ATTR f16 futrts_cos16(f16 x) { return futrts_cos32(x); } +SCALAR_FUN_ATTR f16 futrts_cospi16(f16 x) { return futrts_cospi32(x); } +SCALAR_FUN_ATTR f16 futrts_sin16(f16 x) { return futrts_sin32(x); } +SCALAR_FUN_ATTR f16 futrts_sinpi16(f16 x) { return futrts_sinpi32(x); } +SCALAR_FUN_ATTR f16 futrts_tan16(f16 x) { return futrts_tan32(x); } +SCALAR_FUN_ATTR f16 futrts_tanpi16(f16 x) { return futrts_tanpi32(x); } +SCALAR_FUN_ATTR f16 futrts_acos16(f16 x) { return futrts_acos32(x); } +SCALAR_FUN_ATTR f16 futrts_acospi16(f16 x) { return futrts_acospi32(x); } +SCALAR_FUN_ATTR f16 futrts_asin16(f16 x) { return futrts_asin32(x); } +SCALAR_FUN_ATTR f16 futrts_asinpi16(f16 x) { return futrts_asinpi32(x); } +SCALAR_FUN_ATTR f16 futrts_atan16(f16 x) { return futrts_atan32(x); } +SCALAR_FUN_ATTR f16 futrts_atanpi16(f16 x) { return futrts_atanpi32(x); } +SCALAR_FUN_ATTR f16 futrts_cosh16(f16 x) { return futrts_cosh32(x); } +SCALAR_FUN_ATTR f16 futrts_sinh16(f16 x) { return futrts_sinh32(x); } +SCALAR_FUN_ATTR f16 futrts_tanh16(f16 x) { return futrts_tanh32(x); } +SCALAR_FUN_ATTR f16 futrts_acosh16(f16 x) { return futrts_acosh32(x); } +SCALAR_FUN_ATTR f16 futrts_asinh16(f16 x) { return futrts_asinh32(x); } +SCALAR_FUN_ATTR f16 futrts_atanh16(f16 x) { return futrts_atanh32(x); } +SCALAR_FUN_ATTR f16 futrts_atan2_16(f16 x, f16 y) { return futrts_atan2_32(x, y); } +SCALAR_FUN_ATTR f16 futrts_atan2pi_16(f16 x, f16 y) { return futrts_atan2pi_32(x, y); } +SCALAR_FUN_ATTR f16 futrts_hypot16(f16 x, f16 y) { return futrts_hypot32(x, y); } +SCALAR_FUN_ATTR f16 futrts_gamma16(f16 x) { return futrts_gamma32(x); } +SCALAR_FUN_ATTR f16 futrts_lgamma16(f16 x) { return futrts_lgamma32(x); } +SCALAR_FUN_ATTR f16 futrts_erf16(f16 x) { return futrts_erf32(x); } +SCALAR_FUN_ATTR f16 futrts_erfc16(f16 x) { return futrts_erfc32(x); } +SCALAR_FUN_ATTR f16 fmod16(f16 x, f16 y) { return fmod32(x, y); } +SCALAR_FUN_ATTR f16 futrts_round16(f16 x) { return futrts_round32(x); } +SCALAR_FUN_ATTR f16 futrts_floor16(f16 x) { return futrts_floor32(x); } +SCALAR_FUN_ATTR f16 futrts_ceil16(f16 x) { return futrts_ceil32(x); } +SCALAR_FUN_ATTR f16 futrts_nextafter16(f16 x, f16 y) { return halfbits2float(halfbitsnextafter(float2halfbits(x), float2halfbits(y))); } +SCALAR_FUN_ATTR f16 futrts_lerp16(f16 v0, f16 v1, f16 t) { return futrts_lerp32(v0, v1, t); } +SCALAR_FUN_ATTR f16 futrts_ldexp16(f16 x, int32_t y) { return futrts_ldexp32(x, y); } +SCALAR_FUN_ATTR f16 futrts_copysign16(f16 x, f16 y) { return futrts_copysign32((float)x, y); } +SCALAR_FUN_ATTR f16 futrts_mad16(f16 a, f16 b, f16 c) { return futrts_mad32(a, b, c); } +SCALAR_FUN_ATTR f16 futrts_fma16(f16 a, f16 b, f16 c) { return futrts_fma32(a, b, c); } // Even when we are using an OpenCL that does not support cl_khr_fp16, // it must still support vload_half for actually creating a @@ -881,51 +371,25 @@ } #else - -SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { - return (int16_t)float2halfbits(x); -} - -SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { - return halfbits2float((uint16_t)x); -} - -SCALAR_FUN_ATTR f16 fsignum16(f16 x) { - return futrts_isnan16(x) ? x : (x > 0 ? 1 : 0) - (x < 0 ? 1 : 0); -} +SCALAR_FUN_ATTR int16_t futrts_to_bits16(f16 x) { return (int16_t)float2halfbits(x); } +SCALAR_FUN_ATTR f16 futrts_from_bits16(int16_t x) { return halfbits2float((uint16_t)x); } +SCALAR_FUN_ATTR f16 fsignum16(f16 x) { return futrts_isnan16(x) ? x : (x > 0 ? 1 : 0) - (x < 0 ? 1 : 0); } #endif #endif -SCALAR_FUN_ATTR float fpconv_f16_f16(f16 x) { - return x; -} - -SCALAR_FUN_ATTR float fpconv_f16_f32(f16 x) { - return x; -} - -SCALAR_FUN_ATTR f16 fpconv_f32_f16(float x) { - return (f16) x; -} +SCALAR_FUN_ATTR float fpconv_f16_f16(f16 x) { return x; } +SCALAR_FUN_ATTR float fpconv_f16_f32(f16 x) { return x; } +SCALAR_FUN_ATTR f16 fpconv_f32_f16(float x) { return (f16) x; } #ifdef FUTHARK_F64_ENABLED - -SCALAR_FUN_ATTR double fpconv_f16_f64(f16 x) { - return (double) x; -} - +SCALAR_FUN_ATTR double fpconv_f16_f64(f16 x) { return (double) x; } #if defined(ISPC) -SCALAR_FUN_ATTR f16 fpconv_f64_f16(double x) { - return (f16) ((float)x); -} +SCALAR_FUN_ATTR f16 fpconv_f64_f16(double x) { return (f16) ((float)x); } #else -SCALAR_FUN_ATTR f16 fpconv_f64_f16(double x) { - return (f16) x; -} +SCALAR_FUN_ATTR f16 fpconv_f64_f16(double x) { return (f16) x; } #endif #endif - // End of scalar_f16.h. diff --git a/rts/cuda/prelude.cu b/rts/cuda/prelude.cu --- a/rts/cuda/prelude.cu +++ b/rts/cuda/prelude.cu @@ -4,6 +4,7 @@ #define FUTHARK_FUN_ATTR __device__ static #define FUTHARK_F64_ENABLED +#if defined(__CUDACC_RTC__) || defined(__HIPCC_RTC__) typedef char int8_t; typedef short int16_t; typedef int int32_t; @@ -12,6 +13,17 @@ typedef unsigned short uint16_t; typedef unsigned int uint32_t; typedef unsigned long long uint64_t; +#else +// This is for the benefit of offline compilation with clang. +typedef signed char int8_t; +typedef short int16_t; +typedef int int32_t; +typedef long int64_t; +typedef unsigned char uint8_t; +typedef unsigned short uint16_t; +typedef unsigned int uint32_t; +typedef unsigned long uint64_t; +#endif #define __global #define __local @@ -91,8 +103,11 @@ __syncthreads(); } +#if defined(__CUDACC_RTC__) || defined(__HIPCC_RTC__) #define NAN (0.0/0.0) #define INFINITY (1.0/0.0) +#endif + extern volatile __shared__ unsigned char shared_mem[]; #define SHARED_MEM_PARAM diff --git a/rts/opencl/transpose.cl b/rts/opencl/transpose.cl --- a/rts/opencl/transpose.cl +++ b/rts/opencl/transpose.cl @@ -24,6 +24,7 @@ int tblock_id_2 = get_tblock_id(2); \ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ + if (tblock_id_2 >= num_arrays) { break; } \ int32_t our_array_offset = tblock_id_2 * x_elems * y_elems; \ int32_t odata_offset = dst_offset + our_array_offset; \ int32_t idata_offset = src_offset + our_array_offset; \ @@ -82,6 +83,7 @@ int tblock_id_2 = get_tblock_id(2); \ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ + if (tblock_id_2 >= num_arrays) { break; } \ int32_t our_array_offset = tblock_id_2 * x_elems * y_elems; \ int32_t odata_offset = dst_offset + our_array_offset; \ int32_t idata_offset = src_offset + our_array_offset; \ @@ -136,6 +138,7 @@ int tblock_id_2 = get_tblock_id(2); \ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ + if (tblock_id_2 >= num_arrays) { break; } \ int32_t our_array_offset = tblock_id_2 * x_elems * y_elems; \ int32_t odata_offset = dst_offset + our_array_offset; \ int32_t idata_offset = src_offset + our_array_offset; \ @@ -229,6 +232,7 @@ int tblock_id_2 = get_tblock_id(2); \ int global_id_2 = get_global_id(2); \ for (int i2 = 0; i2 <= repeat_2; i2++) { \ + if (tblock_id_2 >= num_arrays) { break; } \ int64_t our_array_offset = tblock_id_2 * x_elems * y_elems; \ int64_t odata_offset = dst_offset + our_array_offset; \ int64_t idata_offset = src_offset + our_array_offset; \ diff --git a/rts/python/scalar.py b/rts/python/scalar.py --- a/rts/python/scalar.py +++ b/rts/python/scalar.py @@ -4,7 +4,11 @@ import math import struct +pi16 = np.float16(np.pi) +pi32 = np.float32(np.pi) +pi64 = np.float64(np.pi) + def intlit(t, x): if t == np.int8: return np.int8(x) @@ -596,6 +600,10 @@ return np.sqrt(x) +def futhark_rsqrt64(x): + return 1 / np.sqrt(x) + + def futhark_cbrt64(x): return np.cbrt(x) @@ -608,26 +616,50 @@ return np.cos(x) +def futhark_cospi64(x): + return np.cos(pi64 * x) + + def futhark_sin64(x): return np.sin(x) +def futhark_sinpi64(x): + return np.sin(pi64 * x) + + def futhark_tan64(x): return np.tan(x) +def futhark_tanpi64(x): + return np.tan(pi64 * x) + + def futhark_acos64(x): return np.arccos(x) +def futhark_acospi64(x): + return np.arccos(x) / pi64 + + def futhark_asin64(x): return np.arcsin(x) +def futhark_asinpi64(x): + return np.arcsin(x) / pi64 + + def futhark_atan64(x): return np.arctan(x) +def futhark_atanpi64(x): + return np.arctan(x) / pi64 + + def futhark_cosh64(x): return np.cosh(x) @@ -656,6 +688,10 @@ return np.arctan2(x, y) +def futhark_atan2pi_64(x, y): + return np.arctan2(x, y) / pi64 + + def futhark_hypot64(x, y): return np.hypot(x, y) @@ -730,6 +766,10 @@ return np.float32(np.sqrt(x)) +def futhark_rsqrt32(x): + return np.float32(1 / np.sqrt(x)) + + def futhark_cbrt32(x): return np.float32(np.cbrt(x)) @@ -742,26 +782,50 @@ return np.cos(x) +def futhark_cospi32(x): + return np.cos(pi32 * x) + + def futhark_sin32(x): return np.sin(x) +def futhark_sinpi32(x): + return np.sin(pi32 * x) + + def futhark_tan32(x): return np.tan(x) +def futhark_tanpi32(x): + return np.tan(pi32 * x) + + def futhark_acos32(x): return np.arccos(x) +def futhark_acospi32(x): + return np.arccos(x) / pi32 + + def futhark_asin32(x): return np.arcsin(x) +def futhark_asinpi32(x): + return np.arcsin(x) / pi32 + + def futhark_atan32(x): return np.arctan(x) +def futhark_atanpi32(x): + return np.arctan(x) / pi32 + + def futhark_cosh32(x): return np.cosh(x) @@ -790,6 +854,10 @@ return np.arctan2(x, y) +def futhark_atan2pi_32(x, y): + return np.arctan2(x, y) / pi32 + + def futhark_hypot32(x, y): return np.hypot(x, y) @@ -864,6 +932,10 @@ return np.float16(np.sqrt(x)) +def futhark_rsqrt16(x): + return np.float16(1 / np.sqrt(x)) + + def futhark_cbrt16(x): return np.float16(np.cbrt(x)) @@ -876,26 +948,50 @@ return np.cos(x) +def futhark_cospi16(x): + return np.cos(pi16 * x) + + def futhark_sin16(x): return np.sin(x) +def futhark_sinpi16(x): + return np.sin(pi16 * x) + + def futhark_tan16(x): return np.tan(x) +def futhark_tanpi16(x): + return np.tan(pi16 * x) + + def futhark_acos16(x): return np.arccos(x) +def futhark_acospi16(x): + return np.arccos(x) / pi16 + + def futhark_asin16(x): return np.arcsin(x) +def futhark_asinpi16(x): + return np.arcsin(x) / pi16 + + def futhark_atan16(x): return np.arctan(x) +def futhark_atanpi16(x): + return np.arctan(x) / pi16 + + def futhark_cosh16(x): return np.cosh(x) @@ -922,6 +1018,10 @@ def futhark_atan2_16(x, y): return np.arctan2(x, y) + + +def futhark_atan2pi_16(x, y): + return np.arctan2(x, y) / pi16 def futhark_hypot16(x, y): diff --git a/src/Futhark/AD/Derivatives.hs b/src/Futhark/AD/Derivatives.hs --- a/src/Futhark/AD/Derivatives.hs +++ b/src/Futhark/AD/Derivatives.hs @@ -159,6 +159,12 @@ Just [untyped $ 1 / (2 * sqrt (isF32 x))] pdBuiltin "sqrt64" [x] = Just [untyped $ 1 / (2 * sqrt (isF64 x))] +pdBuiltin "rsqrt16" [x] = + Just [untyped $ -1 / (2 * (isF16 x ** (3 / 2)))] +pdBuiltin "rsqrt32" [x] = + Just [untyped $ -1 / (2 * (isF32 x ** (3 / 2)))] +pdBuiltin "rsqrt64" [x] = + Just [untyped $ -1 / (2 * (isF64 x ** (3 / 2)))] pdBuiltin "cbrt16" [x] = Just [untyped $ 1 / (3 * cbrt16 (isF16 x) * cbrt16 (isF16 x))] where @@ -207,6 +213,12 @@ Just [untyped $ cos (isF32 x)] pdBuiltin "sin64" [x] = Just [untyped $ cos (isF64 x)] +pdBuiltin "sinpi16" [x] = + Just [untyped $ pi * cos (pi * isF16 x)] +pdBuiltin "sinpi32" [x] = + Just [untyped $ pi * cos (pi * isF32 x)] +pdBuiltin "sinpi64" [x] = + Just [untyped $ pi * cos (pi * isF64 x)] pdBuiltin "sinh16" [x] = Just [untyped $ cosh (isF16 x)] pdBuiltin "sinh32" [x] = @@ -219,6 +231,12 @@ Just [untyped $ -sin (isF32 x)] pdBuiltin "cos64" [x] = Just [untyped $ -sin (isF64 x)] +pdBuiltin "cospi16" [x] = + Just [untyped $ -pi * sin (pi * isF16 x)] +pdBuiltin "cospi32" [x] = + Just [untyped $ -pi * sin (pi * isF32 x)] +pdBuiltin "cospi64" [x] = + Just [untyped $ -pi * sin (pi * isF64 x)] pdBuiltin "cosh16" [x] = Just [untyped $ sinh (isF16 x)] pdBuiltin "cosh32" [x] = @@ -231,12 +249,24 @@ Just [untyped $ 1 / (cos (isF32 x) * cos (isF32 x))] pdBuiltin "tan64" [x] = Just [untyped $ 1 / (cos (isF64 x) * cos (isF64 x))] +pdBuiltin "tanpi16" [x] = + Just [untyped $ pi * (1 / (cos (pi * isF16 x) * cos (pi * isF16 x)))] +pdBuiltin "tanpi32" [x] = + Just [untyped $ pi * (1 / (cos (pi * isF32 x) * cos (pi * isF32 x)))] +pdBuiltin "tanpi64" [x] = + Just [untyped $ pi * (1 / (cos (pi * isF64 x) * cos (pi * isF64 x)))] pdBuiltin "asin16" [x] = Just [untyped $ 1 / sqrt (1 - isF16 x * isF16 x)] pdBuiltin "asin32" [x] = Just [untyped $ 1 / sqrt (1 - isF32 x * isF32 x)] pdBuiltin "asin64" [x] = Just [untyped $ 1 / sqrt (1 - isF64 x * isF64 x)] +pdBuiltin "asinpi16" [x] = + Just [untyped $ 1 / (pi * sqrt (1 - isF16 x * isF16 x))] +pdBuiltin "asinpi32" [x] = + Just [untyped $ 1 / (pi * sqrt (1 - isF32 x * isF32 x))] +pdBuiltin "asinpi64" [x] = + Just [untyped $ 1 / (pi * sqrt (1 - isF64 x * isF64 x))] pdBuiltin "asinh16" [x] = Just [untyped $ 1 / sqrt (1 + isF16 x * isF16 x)] pdBuiltin "asinh32" [x] = @@ -249,6 +279,12 @@ Just [untyped $ -1 / sqrt (1 - isF32 x * isF32 x)] pdBuiltin "acos64" [x] = Just [untyped $ -1 / sqrt (1 - isF64 x * isF64 x)] +pdBuiltin "acospi16" [x] = + Just [untyped $ -1 / (pi * sqrt (1 - isF16 x * isF16 x))] +pdBuiltin "acospi32" [x] = + Just [untyped $ -1 / (pi * sqrt (1 - isF32 x * isF32 x))] +pdBuiltin "acospi64" [x] = + Just [untyped $ -1 / (pi * sqrt (1 - isF64 x * isF64 x))] pdBuiltin "acosh16" [x] = Just [untyped $ 1 / sqrt (isF16 x * isF16 x - 1)] pdBuiltin "acosh32" [x] = @@ -260,6 +296,12 @@ pdBuiltin "atan32" [x] = Just [untyped $ 1 / (1 + isF32 x * isF32 x)] pdBuiltin "atan64" [x] = + Just [untyped $ 1 / (pi * (1 + isF64 x * isF64 x))] +pdBuiltin "atanpi16" [x] = + Just [untyped $ 1 / (pi * (1 + isF16 x * isF16 x))] +pdBuiltin "atanpi32" [x] = + Just [untyped $ 1 / (pi * (1 + isF32 x * isF32 x))] +pdBuiltin "atanpi64" [x] = Just [untyped $ 1 / (1 + isF64 x * isF64 x)] pdBuiltin "atanh16" [x] = Just [untyped $ cosh (isF16 x) * cosh (isF16 x)] @@ -281,6 +323,21 @@ Just [ untyped $ -isF64 y / (isF64 x * isF64 x + isF64 y * isF64 y), untyped $ -isF64 x / (isF64 x * isF64 x + isF64 y * isF64 y) + ] +pdBuiltin "atan2pi_16" [x, y] = + Just + [ untyped $ -isF16 y / (pi * (isF16 x * isF16 x + isF16 y * isF16 y)), + untyped $ -isF16 x / (pi * (isF16 x * isF16 x + isF16 y * isF16 y)) + ] +pdBuiltin "atan2pi_32" [x, y] = + Just + [ untyped $ -isF32 y / (pi * (isF32 x * isF32 x + isF32 y * isF32 y)), + untyped $ -isF32 x / (pi * (isF32 x * isF32 x + isF32 y * isF32 y)) + ] +pdBuiltin "atan2pi_64" [x, y] = + Just + [ untyped $ -isF64 y / (pi * (isF64 x * isF64 x + isF64 y * isF64 y)), + untyped $ -isF64 x / (pi * (isF64 x * isF64 x + isF64 y * isF64 y)) ] pdBuiltin "tanh16" [x] = Just [untyped $ 1 - tanh (isF16 x) * tanh (isF16 x)] diff --git a/src/Futhark/AD/Fwd.hs b/src/Futhark/AD/Fwd.hs --- a/src/Futhark/AD/Fwd.hs +++ b/src/Futhark/AD/Fwd.hs @@ -210,7 +210,7 @@ letBindNames (patNames pat_tan) <=< toExp $ x_tan ~*~ wrt_x ~+~ y_tan ~*~ wrt_y CmpOp {} -> - addStm $ Let pat_tan aux $ BasicOp op + addStm $ Let pat_tan aux $ zeroExp $ Prim Bool ConvOp cop x -> do x_tan <- tangent x addStm $ Let pat_tan aux $ BasicOp $ ConvOp cop x_tan diff --git a/src/Futhark/AD/Rev.hs b/src/Futhark/AD/Rev.hs --- a/src/Futhark/AD/Rev.hs +++ b/src/Futhark/AD/Rev.hs @@ -51,28 +51,8 @@ diffBasicOp :: Pat Type -> StmAux () -> BasicOp -> ADM () -> ADM () diffBasicOp pat aux e m = case e of - CmpOp cmp x y -> do - (_pat_v, pat_adj) <- commonBasicOp pat aux e m - returnSweepCode $ do - let t = cmpOpType cmp - update contrib = do - void $ updateSubExpAdj x contrib - void $ updateSubExpAdj y contrib - - case t of - FloatType ft -> - update <=< letExp "contrib" $ - Match - [Var pat_adj] - [Case [Just $ BoolValue True] $ resultBody [constant (floatValue ft (1 :: Int))]] - (resultBody [constant (floatValue ft (0 :: Int))]) - (MatchDec [Prim (FloatType ft)] MatchNormal) - IntType it -> - update <=< letExp "contrib" $ BasicOp $ ConvOp (BToI it) (Var pat_adj) - Bool -> - update pat_adj - Unit -> - pure () + CmpOp {} -> + void $ commonBasicOp pat aux e m -- ConvOp op x -> do (_pat_v, pat_adj) <- commonBasicOp pat aux e m @@ -129,8 +109,7 @@ -- Index arr slice -> do (_pat_v, pat_adj) <- commonBasicOp pat aux e m - returnSweepCode $ do - void $ updateAdjSlice slice arr pat_adj + returnSweepCode $ void $ updateAdjSlice slice arr pat_adj FlatIndex {} -> error "FlatIndex not handled by AD yet." FlatUpdate {} -> error "FlatUpdate not handled by AD yet." -- diff --git a/src/Futhark/AD/Rev/Monad.hs b/src/Futhark/AD/Rev/Monad.hs --- a/src/Futhark/AD/Rev/Monad.hs +++ b/src/Futhark/AD/Rev/Monad.hs @@ -93,7 +93,8 @@ = -- | If a SubExp is provided, it references a boolean that is true -- when in-bounds. CheckBounds (Maybe SubExp) - | AssumeBounds + | -- | Assume that these are always in-bounds. + AssumeBounds | -- | Dynamically these will always fail, so don't bother -- generating code for the update. This is only needed to ensure -- a consistent representation of sparse Jacobians. @@ -379,10 +380,54 @@ lookupAdjVal :: VName -> ADM VName lookupAdjVal v = adjVal =<< lookupAdj v -updateAdj :: VName -> VName -> ADM () -updateAdj v d = do +updateAdjIndex :: VName -> (InBounds, SubExp) -> SubExp -> ADM () +updateAdjIndex v (check, i) se = do maybeAdj <- gets $ M.lookup v . stateAdjs + t <- lookupType v + let iv = (check, i, se) case maybeAdj of + Nothing -> do + setAdj v $ AdjSparse $ Sparse (arrayShape t) (elemType t) [iv] + Just AdjZero {} -> + setAdj v $ AdjSparse $ Sparse (arrayShape t) (elemType t) [iv] + Just (AdjSparse (Sparse shape pt ivs)) -> + setAdj v $ AdjSparse $ Sparse shape pt $ iv : ivs + Just adj@AdjVal {} -> do + v_adj <- adjVal adj + v_adj_t <- lookupType v_adj + se_v <- letExp "se_v" $ BasicOp $ SubExp se + insAdj v + =<< case v_adj_t of + Acc {} -> do + let stms s = do + dims <- arrayDims <$> lookupType se_v + ~[v_adj'] <- + tabNest (length dims) [se_v, v_adj] $ \is [se_v', v_adj'] -> + letTupExp "acc" . BasicOp $ + UpdateAcc s v_adj' (i : map Var is) [Var se_v'] + pure v_adj' + case check of + CheckBounds _ -> stms Safe + AssumeBounds -> stms Unsafe + OutOfBounds -> pure v_adj + _ -> do + let stms s = do + v_adj_i <- + letExp (baseString v_adj <> "_i") . BasicOp $ + Index v_adj $ + fullSlice v_adj_t [DimFix i] + se_update <- letSubExp "updated_adj_i" =<< addExp se_v v_adj_i + letExp (baseString v_adj) . BasicOp $ + Update s v_adj (fullSlice v_adj_t [DimFix i]) se_update + case check of + CheckBounds _ -> stms Safe + AssumeBounds -> stms Unsafe + OutOfBounds -> pure v_adj + +updateAdjWithSafety :: VName -> VName -> Safety -> ADM () +updateAdjWithSafety v d safety = do + maybeAdj <- gets $ M.lookup v . stateAdjs + case maybeAdj of Nothing -> insAdj v d Just adj -> do @@ -394,16 +439,20 @@ ~[v_adj'] <- tabNest (length dims) [d, v_adj] $ \is [d', v_adj'] -> letTupExp "acc" . BasicOp $ - UpdateAcc Safe v_adj' (map Var is) [Var d'] + UpdateAcc safety v_adj' (map Var is) [Var d'] insAdj v v_adj' _ -> do v_adj' <- letExp (baseString v <> "_adj") =<< addExp v_adj d insAdj v v_adj' -updateAdjSlice :: Slice SubExp -> VName -> VName -> ADM () -updateAdjSlice (Slice [DimFix i]) v d = - updateAdjIndex v (AssumeBounds, i) (Var d) -updateAdjSlice slice v d = do +updateAdjSliceWithSafety :: Slice SubExp -> VName -> VName -> Safety -> ADM () +updateAdjSliceWithSafety (Slice [DimFix i]) v d safety = + updateAdjIndex v (bounds, i) (Var d) + where + bounds = case safety of + Safe -> CheckBounds Nothing + Unsafe -> AssumeBounds +updateAdjSliceWithSafety slice v d safety = do t <- lookupType v v_adj <- lookupAdjVal v v_adj_t <- lookupType v_adj @@ -417,7 +466,7 @@ fixSlice (fmap pe64 slice) $ map le64 is letTupExp (baseString v_adj') . BasicOp $ - UpdateAcc Safe v_adj' slice' [Var d'] + UpdateAcc safety v_adj' slice' [Var d'] pure v_adj' _ -> do v_adjslice <- @@ -427,52 +476,15 @@ letInPlace "updated_adj" v_adj slice =<< addExp v_adjslice d insAdj v v_adj' +updateAdj :: VName -> VName -> ADM () +updateAdj v d = updateAdjWithSafety v d Unsafe + +updateAdjSlice :: Slice SubExp -> VName -> VName -> ADM () +updateAdjSlice slice v d = updateAdjSliceWithSafety slice v d Unsafe + updateSubExpAdj :: SubExp -> VName -> ADM () updateSubExpAdj Constant {} _ = pure () updateSubExpAdj (Var v) d = void $ updateAdj v d - --- The index may be negative, in which case the update has no effect. -updateAdjIndex :: VName -> (InBounds, SubExp) -> SubExp -> ADM () -updateAdjIndex v (check, i) se = do - maybeAdj <- gets $ M.lookup v . stateAdjs - t <- lookupType v - let iv = (check, i, se) - case maybeAdj of - Nothing -> do - setAdj v $ AdjSparse $ Sparse (arrayShape t) (elemType t) [iv] - Just AdjZero {} -> - setAdj v $ AdjSparse $ Sparse (arrayShape t) (elemType t) [iv] - Just (AdjSparse (Sparse shape pt ivs)) -> - setAdj v $ AdjSparse $ Sparse shape pt $ iv : ivs - Just adj@AdjVal {} -> do - v_adj <- adjVal adj - v_adj_t <- lookupType v_adj - se_v <- letExp "se_v" $ BasicOp $ SubExp se - insAdj v - =<< case v_adj_t of - Acc {} - | check == OutOfBounds -> - pure v_adj - | otherwise -> do - dims <- arrayDims <$> lookupType se_v - ~[v_adj'] <- - tabNest (length dims) [se_v, v_adj] $ \is [se_v', v_adj'] -> - letTupExp "acc" . BasicOp $ - UpdateAcc Safe v_adj' (i : map Var is) [Var se_v'] - pure v_adj' - _ -> do - let stms s = do - v_adj_i <- - letExp (baseString v_adj <> "_i") . BasicOp $ - Index v_adj $ - fullSlice v_adj_t [DimFix i] - se_update <- letSubExp "updated_adj_i" =<< addExp se_v v_adj_i - letExp (baseString v_adj) . BasicOp $ - Update s v_adj (fullSlice v_adj_t [DimFix i]) se_update - case check of - CheckBounds _ -> stms Safe - AssumeBounds -> stms Unsafe - OutOfBounds -> pure v_adj -- | Is this primal variable active in the AD sense? FIXME: this is -- (obviously) much too conservative. diff --git a/src/Futhark/AD/Rev/SOAC.hs b/src/Futhark/AD/Rev/SOAC.hs --- a/src/Futhark/AD/Rev/SOAC.hs +++ b/src/Futhark/AD/Rev/SOAC.hs @@ -192,6 +192,9 @@ (mapstm, redstm) <- histomapToMapAndHist pat (n, histops, f, as) vjpStm ops mapstm $ vjpStm ops redstm m +vjpSOAC ops pat aux (Stream w as accs lam) m = do + stms <- collectStms_ $ auxing aux $ sequentialStreamWholeArray pat w accs lam as + foldr (vjpStm ops) m stms vjpSOAC _ _ _ soac _ = error $ "vjpSOAC unhandled:\n" ++ prettyString soac diff --git a/src/Futhark/Analysis/HORep/MapNest.hs b/src/Futhark/Analysis/HORep/MapNest.hs --- a/src/Futhark/Analysis/HORep/MapNest.hs +++ b/src/Futhark/Analysis/HORep/MapNest.hs @@ -3,15 +3,18 @@ module Futhark.Analysis.HORep.MapNest ( Nesting (..), MapNest (..), + depth, typeOf, params, inputs, setInputs, fromSOAC, toSOAC, + reshape, ) where +import Control.Monad (replicateM) import Data.List (find) import Data.Map.Strict qualified as M import Data.Maybe @@ -19,10 +22,11 @@ import Futhark.Analysis.HORep.SOAC qualified as SOAC import Futhark.Construct import Futhark.IR hiding (typeOf) +import Futhark.IR.SOACS (SOACS) import Futhark.IR.SOACS.SOAC qualified as Futhark import Futhark.Transform.Substitute -data Nesting rep = Nesting +data Nesting = Nesting { nestingParamNames :: [VName], nestingResult :: [VName], nestingReturnType :: [Type], @@ -30,25 +34,33 @@ } deriving (Eq, Ord, Show) -data MapNest rep = MapNest SubExp (Lambda rep) [Nesting rep] [SOAC.Input] +data MapNest = MapNest + { mapNestWidth :: SubExp, + mapNestLambda :: Lambda SOACS, + mapNestNestings :: [Nesting], + mapNestInput :: [SOAC.Input] + } deriving (Show) -typeOf :: MapNest rep -> [Type] +depth :: MapNest -> Int +depth (MapNest _ _ nests _) = 1 + length nests + +typeOf :: MapNest -> [Type] typeOf (MapNest w lam [] _) = map (`arrayOfRow` w) $ lambdaReturnType lam typeOf (MapNest w _ (nest : _) _) = map (`arrayOfRow` w) $ nestingReturnType nest -params :: MapNest rep -> [VName] +params :: MapNest -> [VName] params (MapNest _ lam [] _) = map paramName $ lambdaParams lam params (MapNest _ _ (nest : _) _) = nestingParamNames nest -inputs :: MapNest rep -> [SOAC.Input] +inputs :: MapNest -> [SOAC.Input] inputs (MapNest _ _ _ inps) = inps -setInputs :: [SOAC.Input] -> MapNest rep -> MapNest rep +setInputs :: [SOAC.Input] -> MapNest -> MapNest setInputs [] (MapNest w body ns _) = MapNest w body ns [] setInputs (inp : inps) (MapNest _ body ns _) = MapNest w body ns' (inp : inps) where @@ -57,94 +69,112 @@ ns' = zipWith setDepth ns ws setDepth n nw = n {nestingWidth = nw} -fromSOAC :: - ( Buildable rep, - MonadFreshNames m, - LocalScope rep m, - Op rep ~ Futhark.SOAC rep - ) => - SOAC rep -> - m (Maybe (MapNest rep)) -fromSOAC = fromSOAC' mempty +pushIntoMapLambda :: + Stms SOACS -> + Stm SOACS -> + Maybe (Stm SOACS) +pushIntoMapLambda stms (Let pat aux (Op (Futhark.Screma w inps form))) + | Just map_lam <- Futhark.isMapSOAC form, + not $ any ((`namesIntersect` bound_by_stms) . freeIn) inps = + let lam_body = lambdaBody map_lam + map_lam' = + map_lam {lambdaBody = lam_body {bodyStms = stms <> bodyStms lam_body}} + form' = Futhark.mapSOAC map_lam' + in Just $ Let pat aux (Op (Futhark.Screma w inps form')) + where + bound_by_stms = namesFromList $ foldMap (patNames . stmPat) stms +pushIntoMapLambda _ _ = Nothing +massage :: SOAC SOACS -> SOAC SOACS +massage (SOAC.Screma w inps form) + | Just lam <- Futhark.isMapSOAC form, + Just (init_stms, last_stm) <- stmsLast $ bodyStms $ lambdaBody lam, + all (cheap . stmExp) init_stms, + all (`notNameIn` freeIn (bodyResult (lambdaBody lam))) $ + foldMap (patNames . stmPat) init_stms, + Just last_stm' <- pushIntoMapLambda init_stms last_stm = + let lam' = + lam {lambdaBody = (lambdaBody lam) {bodyStms = oneStm last_stm'}} + in SOAC.Screma w inps (Futhark.mapSOAC lam') + where + cheap (BasicOp BinOp {}) = True + cheap (BasicOp SubExp {}) = True + cheap (BasicOp CmpOp {}) = True + cheap (BasicOp ConvOp {}) = True + cheap (BasicOp UnOp {}) = True + cheap _ = False +massage soac = soac + fromSOAC' :: - ( Buildable rep, - MonadFreshNames m, - LocalScope rep m, - Op rep ~ Futhark.SOAC rep - ) => + (MonadFreshNames m, LocalScope SOACS m) => [Ident] -> - SOAC rep -> - m (Maybe (MapNest rep)) -fromSOAC' bound (SOAC.Screma w inps (SOAC.ScremaForm lam [] [])) = do - maybenest <- case ( stmsToList $ bodyStms $ lambdaBody lam, - bodyResult $ lambdaBody lam - ) of - ([Let pat _ e], res) - | map resSubExp res == map Var (patNames pat) -> - localScope (scopeOfLParams $ lambdaParams lam) $ - SOAC.fromExp e - >>= either (pure . Left) (fmap (Right . fmap (pat,)) . fromSOAC' bound') - _ -> - pure $ Right Nothing + SOAC SOACS -> + m (Maybe MapNest) +fromSOAC' bound soac + | SOAC.Screma w inps (SOAC.ScremaForm lam [] []) <- massage soac = do + let bound' = bound <> map paramIdent (lambdaParams lam) - case maybenest of - -- Do we have a nested MapNest? - Right (Just (pat, mn@(MapNest inner_w body' ns' inps'))) -> do - (ps, inps'') <- - unzip - <$> fixInputs - w - (zip (map paramName $ lambdaParams lam) inps) - (zip (params mn) inps') - let n' = - Nesting - { nestingParamNames = ps, - nestingResult = patNames pat, - nestingReturnType = typeOf mn, - nestingWidth = inner_w - } - pure $ Just $ MapNest w body' (n' : ns') inps'' - -- No nested MapNest it seems. - _ -> do - let isBound name - | Just param <- find ((name ==) . identName) bound = - Just param - | otherwise = - Nothing - boundUsedInBody = - mapMaybe isBound $ namesToList $ freeIn lam - newParams <- mapM (newIdent' (++ "_wasfree")) boundUsedInBody - let subst = - M.fromList $ - zip (map identName boundUsedInBody) (map identName newParams) - inps' = - inps - ++ map - (SOAC.addTransform (SOAC.Replicate mempty $ Shape [w]) . SOAC.identInput) - boundUsedInBody - lam' = - lam - { lambdaBody = - substituteNames subst $ lambdaBody lam, - lambdaParams = - lambdaParams lam - ++ [Param mempty name t | Ident name t <- newParams] - } - pure $ Just $ MapNest w lam' [] inps' - where - bound' = bound <> map paramIdent (lambdaParams lam) + maybenest <- case ( stmsToList $ bodyStms $ lambdaBody lam, + bodyResult $ lambdaBody lam + ) of + ([Let pat _ e], res) + | map resSubExp res == map Var (patNames pat) -> + localScope (scopeOfLParams $ lambdaParams lam) $ + SOAC.fromExp e + >>= either (pure . Left) (fmap (Right . fmap (pat,)) . fromSOAC' bound') + _ -> + pure $ Right Nothing + + case maybenest of + -- Do we have a nested MapNest? + Right (Just (pat, mn@(MapNest inner_w body' ns' inps'))) -> do + (ps, inps'') <- + unzip + <$> fixInputs + w + (zip (map paramName $ lambdaParams lam) inps) + (zip (params mn) inps') + let n' = + Nesting + { nestingParamNames = ps, + nestingResult = patNames pat, + nestingReturnType = typeOf mn, + nestingWidth = inner_w + } + pure $ Just $ MapNest w body' (n' : ns') inps'' + -- No nested MapNest it seems. + _ -> do + let isBound name + | Just param <- find ((name ==) . identName) bound = + Just param + | otherwise = + Nothing + boundUsedInBody = + mapMaybe isBound $ namesToList $ freeIn lam + newParams <- mapM (newIdent' (++ "_wasfree")) boundUsedInBody + let subst = + M.fromList $ + zip (map identName boundUsedInBody) (map identName newParams) + inps' = + inps + ++ map + (SOAC.addTransform (SOAC.Replicate mempty $ Shape [w]) . SOAC.identInput) + boundUsedInBody + lam' = + lam + { lambdaBody = + substituteNames subst $ lambdaBody lam, + lambdaParams = + lambdaParams lam + ++ [Param mempty name t | Ident name t <- newParams] + } + pure $ Just $ MapNest w lam' [] inps' fromSOAC' _ _ = pure Nothing -toSOAC :: - ( MonadFreshNames m, - HasScope rep m, - Buildable rep, - BuilderOps rep, - Op rep ~ Futhark.SOAC rep - ) => - MapNest rep -> - m (SOAC rep) +fromSOAC :: (MonadFreshNames m, LocalScope SOACS m) => SOAC SOACS -> m (Maybe MapNest) +fromSOAC = fromSOAC' mempty + +toSOAC :: (MonadFreshNames m, HasScope SOACS m) => MapNest -> m (SOAC SOACS) toSOAC (MapNest w lam [] inps) = pure $ SOAC.Screma w inps (Futhark.mapSOAC lam) toSOAC (MapNest w lam (Nesting npnames nres nrettype nw : ns) inps) = do @@ -181,3 +211,33 @@ inspect (param, SOAC.Input ts a t) = do param' <- newNameFromString (baseString param ++ "_rep") pure (param', SOAC.Input (ts SOAC.|> SOAC.Replicate mempty (Shape [w])) a t) + +-- | Reshape a map nest. It is assumed that any validity tests have +-- already been done. Will automatically reshape the inputs +-- appropriately. +reshape :: (MonadFreshNames m) => Certs -> Shape -> MapNest -> m MapNest +reshape cs shape (MapNest _ map_lam _ inps) = + descend [] $ stripDims 1 shape + where + w = shapeSize 0 shape + transform p inp = + let shape' = shape <> arrayShape p + tr = SOAC.Reshape cs ReshapeArbitrary shape' + in SOAC.addTransform tr inp + inps' = zipWith transform (map paramType $ lambdaParams map_lam) inps + + descend nests nest_shape + | shapeRank nest_shape == 0 = + pure $ MapNest w map_lam nests inps' + | otherwise = do + nest_params <- + mapM (newVName . baseString . paramName) $ + lambdaParams map_lam + res <- + replicateM + (length $ lambdaReturnType map_lam) + (newVName "mapnest_res") + let types = + map (`arrayOfShape` nest_shape) $ lambdaReturnType map_lam + nest = Nesting nest_params res types (shapeSize 0 nest_shape) + descend (nests ++ [nest]) $ stripDims 1 nest_shape diff --git a/src/Futhark/Analysis/HORep/SOAC.hs b/src/Futhark/Analysis/HORep/SOAC.hs --- a/src/Futhark/Analysis/HORep/SOAC.hs +++ b/src/Futhark/Analysis/HORep/SOAC.hs @@ -113,6 +113,14 @@ Index Certs (Slice SubExp) deriving (Show, Eq, Ord) +instance FreeIn ArrayTransform where + freeIn' (Rearrange cs _) = freeIn' cs + freeIn' (Reshape cs _ shape) = freeIn' cs <> freeIn' shape + freeIn' (ReshapeOuter cs _ shape) = freeIn' cs <> freeIn' shape + freeIn' (ReshapeInner cs _ shape) = freeIn' cs <> freeIn' shape + freeIn' (Replicate cs shape) = freeIn' cs <> freeIn' shape + freeIn' (Index cs slice) = freeIn' cs <> freeIn' slice + instance Substitute ArrayTransform where substituteNames substs (Rearrange cs xs) = Rearrange (substituteNames substs cs) xs @@ -153,6 +161,9 @@ substituteNames substs (ArrayTransforms ts) = ArrayTransforms $ substituteNames substs <$> ts +instance FreeIn ArrayTransforms where + freeIn' (ArrayTransforms trs) = foldMap freeIn' trs + -- | The empty transformation list. noTransforms :: ArrayTransforms noTransforms = ArrayTransforms Seq.empty @@ -267,6 +278,9 @@ -- the first element of the 'ArrayTransform' list is applied first. data Input = Input ArrayTransforms VName Type deriving (Show, Eq, Ord) + +instance FreeIn Input where + freeIn' (Input trs v t) = freeIn' trs <> freeIn' v <> freeIn' t instance Substitute Input where substituteNames substs (Input ts v t) = diff --git a/src/Futhark/CLI/Run.hs b/src/Futhark/CLI/Run.hs --- a/src/Futhark/CLI/Run.hs +++ b/src/Futhark/CLI/Run.hs @@ -35,7 +35,7 @@ pr <- newFutharkiState config fp (tenv, ienv) <- case pr of Left err -> do - hPutDoc stderr err + hPutDocLn stderr err exitFailure Right env -> pure env diff --git a/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs b/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs --- a/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs +++ b/src/Futhark/CodeGen/ImpGen/GPU/SegHist.hs @@ -535,7 +535,7 @@ let lock_shape = Shape [tvSize num_subhistos_per_block, hist_H_chk] - let dims = map pe64 $ shapeDims lock_shape + let dims = [sExt64 (tvExp num_subhistos_per_block), pe64 hist_H_chk] locks <- sAllocArray "locks" int32 lock_shape $ Space "shared" diff --git a/src/Futhark/IR/SOACS/Simplify.hs b/src/Futhark/IR/SOACS/Simplify.hs --- a/src/Futhark/IR/SOACS/Simplify.hs +++ b/src/Futhark/IR/SOACS/Simplify.hs @@ -26,7 +26,7 @@ import Control.Monad.Writer import Data.Either import Data.Foldable -import Data.List (partition, transpose, unzip6, zip6) +import Data.List (partition, transpose, unzip4, unzip6, zip6) import Data.List.NonEmpty (NonEmpty (..)) import Data.Map.Strict qualified as M import Data.Maybe @@ -206,7 +206,8 @@ RuleOp removeDuplicateMapOutput, RuleOp fuseConcatScatter, RuleOp simplifyMapIota, - RuleOp moveTransformToInput + RuleOp moveTransformToInput, + RuleOp moveTransformToOutput ] bottomUpRules :: [BottomUpRule (Wise SOACS)] @@ -989,4 +990,77 @@ ) mapOverArr _ = pure Nothing moveTransformToInput _ _ _ _ = + Skip + +-- The idea behidn this rule is to tak cases such as +-- +-- let ...A... = +-- map (\x -> ... +-- let x = ... +-- ... +-- let y = f(x) +-- ... +-- in ...y ...) +-- +-- where 'f' is some transformation like a reshape, and move it out +-- such that we get +-- +-- let ...A'... = +-- map (\x -> ... +-- let x = ... +-- ... +-- in ...x ...) +-- let A' = f'(A') +-- +-- This can improve simplification in case A' fuses or simplifies with +-- something else. +-- +-- TODO: currently we only handle reshapes here, but the principle +-- should actually hold for any ArrayTransform. +moveTransformToOutput :: TopDownRuleOp (Wise SOACS) +moveTransformToOutput vtable screma_pat screma_aux (Screma w arrs (ScremaForm map_lam scan reduce)) + | (transformed, map_infos, stms') <- + foldl' onStm ([], zip3 map_res map_rets map_pes, mempty) $ bodyStms $ lambdaBody map_lam, + (map_res', map_rets', map_pes') <- unzip3 map_infos, + not $ null transformed = Simplify $ do + (tr_res, tr_rets, tr_names, post) <- unzip4 <$> mapM mkTransformed transformed + let map_lam' = + map_lam + { lambdaBody = mkBody stms' $ nonmap_res <> map_res' <> tr_res, + lambdaReturnType = nonmap_rets <> map_rets' <> tr_rets + } + pat_names = map patElemName (nonmap_pes <> map_pes') <> tr_names + auxing screma_aux . letBindNames pat_names . Op $ + Screma w arrs (ScremaForm map_lam' scan reduce) + sequence_ post + where + num_nonmap_res = scanResults scan + redResults reduce + (nonmap_pes, map_pes) = + splitAt num_nonmap_res $ patElems screma_pat + (nonmap_rets, map_rets) = + splitAt num_nonmap_res $ lambdaReturnType map_lam + (nonmap_res, map_res) = + splitAt num_nonmap_res $ bodyResult $ lambdaBody map_lam + + scope = scopeOf $ bodyStms $ lambdaBody map_lam + + invariantToMap = all (`ST.elem` vtable) . namesToList . freeIn + + onStm (transformed, map_infos, stms) (Let (Pat [pe]) aux (BasicOp (Reshape k new_shape arr))) + | ([(res, _, screma_pe)], map_pesres') <- partition matches map_infos, + Just t <- typeOf <$> M.lookup arr scope, + invariantToMap t = + let cs = stmAuxCerts aux <> resCerts res + transform = (arr, cs, BasicOp . Reshape k (Shape [w] <> new_shape)) + in ((t, screma_pe, transform) : transformed, map_pesres', stms) + where + matches (r, _, _) = resSubExp r == Var (patElemName pe) + onStm (transformed, map_infos, stms) stm = + (transformed, map_infos, stms <> oneStm stm) + + mkTransformed (t, pe, (arr, cs, f)) = do + v <- newVName (baseString (patElemName pe) <> "_pretr") + let bind = letBindNames [patElemName pe] $ f v + pure (SubExpRes cs (Var arr), t, v, bind) +moveTransformToOutput _ _ _ _ = Skip diff --git a/src/Futhark/Internalise/Entry.hs b/src/Futhark/Internalise/Entry.hs --- a/src/Futhark/Internalise/Entry.hs +++ b/src/Futhark/Internalise/Entry.hs @@ -191,6 +191,10 @@ elemTypeExp (E.TEParens te _) = elemTypeExp te elemTypeExp _ = Nothing +rowTypeExp :: Int -> E.TypeExp E.Exp VName -> Maybe (E.TypeExp E.Exp VName) +rowTypeExp 0 te = Just te +rowTypeExp r te = rowTypeExp (r - 1) =<< elemTypeExp te + entryPointType :: VisibleTypes -> E.EntryType -> @@ -227,13 +231,14 @@ rank = E.shapeRank shape ts' = map (strip rank) ts record_t = E.Scalar (E.Record fs) - record_te = elemTypeExp =<< E.entryAscribed t + record_te = rowTypeExp rank =<< E.entryAscribed t ept <- snd <$> entryPointType types (E.EntryType record_t record_te) ts' addType desc . I.OpaqueRecordArray rank (entryPointTypeName ept) =<< opaqueRecordArray types rank fs' ts E.Array _ shape et -> do let ts' = map (strip (E.shapeRank shape)) ts - elem_te = elemTypeExp =<< E.entryAscribed t + rank = E.shapeRank shape + elem_te = rowTypeExp rank =<< E.entryAscribed t ept <- snd <$> entryPointType types (E.EntryType (E.Scalar et) elem_te) ts' addType desc . I.OpaqueArray (E.shapeRank shape) (entryPointTypeName ept) $ map valueType ts diff --git a/src/Futhark/Internalise/LiftLambdas.hs b/src/Futhark/Internalise/LiftLambdas.hs --- a/src/Futhark/Internalise/LiftLambdas.hs +++ b/src/Futhark/Internalise/LiftLambdas.hs @@ -120,7 +120,7 @@ valBindRetType = Info (RetType dims ret), valBindBody = funbody, valBindDoc = Nothing, - valBindAttrs = mempty, + valBindAttrs = [AttrAtom (AtomName "inline") mempty], valBindLocation = mempty, valBindEntryPoint = Nothing } diff --git a/src/Futhark/Internalise/Monomorphise.hs b/src/Futhark/Internalise/Monomorphise.hs --- a/src/Futhark/Internalise/Monomorphise.hs +++ b/src/Futhark/Internalise/Monomorphise.hs @@ -865,9 +865,14 @@ where tparamArg dinst tp = case M.lookup (typeParamName tp) dinst of - Just e -> - replaceExp e - Nothing -> + Just e + -- In some cases we infer anySizes for size arguments. This + -- only occurs when those sizes don't actually matter (knock + -- on wood...), but we should never actually insert anySize + -- as a concrete argument. + | e /= anySize -> + replaceExp e + _ -> pure $ sizeFromInteger 0 mempty -- Monomorphising higher-order functions can result in function types diff --git a/src/Futhark/Optimise/Fusion/TryFusion.hs b/src/Futhark/Optimise/Fusion/TryFusion.hs --- a/src/Futhark/Optimise/Fusion/TryFusion.hs +++ b/src/Futhark/Optimise/Fusion/TryFusion.hs @@ -18,9 +18,10 @@ import Control.Monad import Control.Monad.Reader import Control.Monad.State -import Data.List (find, tails, (\\)) +import Data.List (find, (\\)) import Data.Map.Strict qualified as M import Data.Maybe +import Futhark.Analysis.HORep.MapNest (MapNest) import Futhark.Analysis.HORep.MapNest qualified as MapNest import Futhark.Analysis.HORep.SOAC qualified as SOAC import Futhark.Construct @@ -66,8 +67,6 @@ type SOAC = SOAC.SOAC SOACS -type MapNest = MapNest.MapNest SOACS - inputToOutput :: SOAC.Input -> Maybe (SOAC.ArrayTransform, SOAC.Input) inputToOutput (SOAC.Input ts ia iat) = case SOAC.viewf ts of @@ -767,52 +766,19 @@ | otherwise = Nothing pullReshape :: SOAC -> SOAC.ArrayTransforms -> TryFusion (SOAC, SOAC.ArrayTransforms) -pullReshape (SOAC.Screma _ inps form) ots - | Just maplam <- Futhark.isMapSOAC form, - SOAC.Reshape cs k shape SOAC.:< ots' <- SOAC.viewf ots, - all primType $ lambdaReturnType maplam = do - let mapw' = case reverse $ shapeDims shape of - [] -> intConst Int64 0 - d : _ -> d - trInput inp - | arrayRank (SOAC.inputType inp) == 1 = - SOAC.addTransform (SOAC.Reshape cs k shape) inp - | otherwise = - SOAC.addTransform (SOAC.ReshapeOuter cs k shape) inp - inputs' = map trInput inps - inputTypes = map SOAC.inputType inputs' - - let outersoac :: - ([SOAC.Input] -> SOAC) -> - (SubExp, [SubExp]) -> - TryFusion ([SOAC.Input] -> SOAC) - outersoac inner (w, outershape) = do - let addDims t = arrayOf t (Shape outershape) NoUniqueness - retTypes = map addDims $ lambdaReturnType maplam - - ps <- forM inputTypes $ \inpt -> - newParam "pullReshape_param" $ - stripArray (length shape - length outershape) inpt - - inner_body <- - runBodyBuilder $ - varsRes - <$> (letTupExp "x" <=< SOAC.toExp $ inner $ map (SOAC.identInput . paramIdent) ps) - let inner_fun = - Lambda - { lambdaParams = ps, - lambdaReturnType = retTypes, - lambdaBody = inner_body - } - pure $ flip (SOAC.Screma w) $ Futhark.mapSOAC inner_fun - - op' <- - foldM outersoac (flip (SOAC.Screma mapw') $ Futhark.mapSOAC maplam) $ - zip (drop 1 $ reverse $ shapeDims shape) $ - drop 1 . reverse . drop 1 . tails $ - shapeDims shape - pure (op' inputs', ots') -pullReshape _ _ = fail "Cannot pull reshape" +pullReshape soac ots = do + Just mapnest <- MapNest.fromSOAC soac + SOAC.Reshape cs _kind newshape SOAC.:< ots' <- pure $ SOAC.viewf ots + -- This handles only the easy case where the underlying lambda is + -- scalar. The more complicated cases could also be handled, but + -- requires more tricky checks. + guard $ + all + ((== MapNest.depth mapnest) . arrayRank) + (MapNest.typeOf mapnest) + mapnest' <- MapNest.reshape cs newshape mapnest + soac' <- MapNest.toSOAC mapnest' + pure (soac', ots') -- Tie it all together in exposeInputs (for making inputs to a -- consumer available) and pullOutputTransforms (for moving @@ -825,6 +791,7 @@ exposeInputs inpIds ker = (exposeInputs' =<< pushRearrange') <|> (exposeInputs' =<< pullRearrange') + <|> (exposeInputs' =<< pullReshape') <|> (exposeInputs' =<< pullIndex') <|> exposeInputs' ker where @@ -842,6 +809,16 @@ (soac', ot') <- pullRearrange (fsSOAC ker) ot unless (SOAC.nullTransforms ot') $ fail "pullRearrange was not enough" + pure + ker + { fsSOAC = soac', + fsOutputTransform = SOAC.noTransforms + } + + pullReshape' = do + (soac', ot') <- pullReshape (fsSOAC ker) ot + unless (SOAC.nullTransforms ot') $ + fail "pullReshape was not enough" pure ker { fsSOAC = soac', diff --git a/src/Futhark/Optimise/Simplify/Engine.hs b/src/Futhark/Optimise/Simplify/Engine.hs --- a/src/Futhark/Optimise/Simplify/Engine.hs +++ b/src/Futhark/Optimise/Simplify/Engine.hs @@ -589,8 +589,15 @@ isConsumed :: BlockPred rep isConsumed _ utable = any (`UT.isConsumed` utable) . patNames . stmPat +-- The main purpose of this rule is to avoid hoisting 'inblock' SegOps +-- out of their enclosing SegOp, *including* when those are present in +-- nested Bodies. isOp :: BlockPred rep isOp _ _ (Let _ _ Op {}) = True +isOp vtable utable (Let _ _ (Match _ cs def_body _)) = + any (any (isOp vtable utable) . bodyStms) $ def_body : map caseBody cs +isOp vtable utable (Let _ _ (Loop _ _ body)) = + any (isOp vtable utable) $ bodyStms body isOp _ _ _ = False constructBody :: diff --git a/src/Futhark/Optimise/Simplify/Rules/Simple.hs b/src/Futhark/Optimise/Simplify/Rules/Simple.hs --- a/src/Futhark/Optimise/Simplify/Rules/Simple.hs +++ b/src/Futhark/Optimise/Simplify/Rules/Simple.hs @@ -173,6 +173,7 @@ | isCt0 e2 = constRes $ BoolValue False | isCt1 e1 = resIsSubExp e2 | isCt1 e2 = resIsSubExp e1 + | e1 == e2 = resIsSubExp e1 | Var v <- e1, Just (BasicOp (UnOp (Neg Bool) e1'), v_cs) <- defOf v, e1' == e2 = diff --git a/src/Futhark/Pass/ExplicitAllocations/GPU.hs b/src/Futhark/Pass/ExplicitAllocations/GPU.hs --- a/src/Futhark/Pass/ExplicitAllocations/GPU.hs +++ b/src/Futhark/Pass/ExplicitAllocations/GPU.hs @@ -122,8 +122,10 @@ where -- Heuristic: do not rearrange for returned arrays that are -- sufficiently small. + thresholdBytes = 8 + coalesceReturnOfShape _ [] = False - coalesceReturnOfShape bs [Constant (IntValue (Int64Value d))] = bs * d > 4 + coalesceReturnOfShape bs [Constant (IntValue (Int64Value d))] = bs * d > thresholdBytes coalesceReturnOfShape _ _ = True hint t Returns {} diff --git a/src/Futhark/Pkg/Types.hs b/src/Futhark/Pkg/Types.hs --- a/src/Futhark/Pkg/Types.hs +++ b/src/Futhark/Pkg/Types.hs @@ -282,7 +282,7 @@ pPkgPath = T.pack - <$> some (alphaNumChar <|> oneOf ("@-/.:" :: String)) + <$> some (alphaNumChar <|> oneOf ("@-/.:_" :: String)) "package path" pRequired = diff --git a/src/Language/Futhark/Interpreter.hs b/src/Language/Futhark/Interpreter.hs --- a/src/Language/Futhark/Interpreter.hs +++ b/src/Language/Futhark/Interpreter.hs @@ -35,7 +35,7 @@ import Data.Array import Data.Bifunctor import Data.Bitraversable -import Data.Either (fromRight) +import Data.Functor (($>), (<&>)) import Data.List ( find, foldl', @@ -100,7 +100,7 @@ = EvalM ( ReaderT (Stack, M.Map ImportName Env) - (StateT Exts (F ExtOp)) + (StateT (Exts, AD.Counter) (F ExtOp)) a ) deriving @@ -109,11 +109,11 @@ Functor, MonadFree ExtOp, MonadReader (Stack, M.Map ImportName Env), - MonadState Exts + MonadState (Exts, AD.Counter) ) runEvalM :: M.Map ImportName Env -> EvalM a -> F ExtOp a -runEvalM imports (EvalM m) = evalStateT (runReaderT m (mempty, imports)) mempty +runEvalM imports (EvalM m) = evalStateT (runReaderT m (mempty, imports)) (mempty, AD.Counter 0) stacking :: SrcLoc -> Env -> EvalM a -> EvalM a stacking loc env = local $ \(ss, imports) -> @@ -129,23 +129,35 @@ stacktrace :: EvalM [Loc] stacktrace = asks $ map stackFrameLoc . fst +-- | Instead of tracking the actual depth of AD, we just use the size +-- of the stack as a proxy. +adDepth :: EvalM AD.Depth +adDepth = AD.Depth . length <$> stacktrace + lookupImport :: ImportName -> EvalM (Maybe Env) lookupImport f = asks $ M.lookup f . snd putExtSize :: VName -> Value -> EvalM () -putExtSize v x = modify $ M.insert v x +putExtSize v x = modify $ first $ M.insert v x getExts :: EvalM Exts -getExts = get +getExts = gets fst +putCounter :: AD.Counter -> EvalM () +putCounter i = modify $ second $ const i + +getCounter :: EvalM AD.Counter +getCounter = gets snd + -- | Disregard any existential sizes computed during this action. -- This is used so that existentials computed during one iteration of -- a loop or a function call are not remembered the next time around. localExts :: EvalM a -> EvalM a localExts m = do - s <- get + e <- getExts x <- m - put s + i <- getCounter + put (e, i) pure x extEnv :: EvalM Env @@ -1166,13 +1178,8 @@ ] Just m -> pure (mempty, Module m) evalModExp env (ModDecs ds _) = do - Env terms types <- foldM evalDec env ds - -- Remove everything that was present in the original Env. - let env' = - Env - (terms `M.difference` envTerm env) - (types `M.difference` envType env) - pure (env', Module env') + (_, built_env) <- evalDecs env ds + pure (built_env, Module built_env) evalModExp env (ModVar qv _) = (mempty,) <$> evalModuleVar env qv evalModExp env (ModAscript me _ (Info substs) _) = @@ -1207,22 +1214,22 @@ evalDec env (ValDec (ValBind _ v _ (Info ret) tparams ps fbody _ _ _)) = localExts $ do binding <- evalFunctionBinding env tparams ps ret fbody sizes <- extEnv - pure $ env {envTerm = M.insert v binding $ envTerm env} <> sizes + pure $ mempty {envTerm = M.singleton v binding} <> sizes evalDec env (OpenDec me _) = do (me_env, me') <- evalModExp env me case me' of - Module me'' -> pure $ me'' <> me_env <> env + Module me'' -> pure $ me'' <> me_env _ -> error "Expected Module" evalDec env (ImportDec name name' loc) = evalDec env $ LocalDec (OpenDec (ModImport name name' loc) loc) loc evalDec env (LocalDec d _) = evalDec env d -evalDec env ModTypeDec {} = pure env +evalDec _env ModTypeDec {} = pure mempty evalDec env (TypeDec (TypeBind v _ ps _ (Info (RetType dims t)) _ _)) = do let abbr = TypeBinding env ps $ RetType dims t - pure env {envType = M.insert v abbr $ envType env} + pure mempty {envType = M.singleton v abbr} evalDec env (ModDec (ModBind v ps ret body _ loc)) = do (mod_env, mod) <- evalModExp env $ wrapInLambda ps - pure $ modEnv (M.singleton v mod) <> mod_env <> env + pure $ modEnv (M.singleton v mod) <> mod_env where wrapInLambda [] = case ret of Just (se, substs) -> ModAscript body se substs loc @@ -1230,6 +1237,13 @@ wrapInLambda [p] = ModLambda p ret body loc wrapInLambda (p : ps') = ModLambda p Nothing (wrapInLambda ps') loc +evalDecs :: Env -> [Dec] -> EvalM (Env, Env) +evalDecs env = foldM evalDec' (env, mempty) + where + evalDec' (env', built_env) dec = do + dec_env <- evalDec env' dec + pure (dec_env <> env', dec_env <> built_env) + -- | The interpreter context. All evaluation takes place with respect -- to a context, and it can be extended with more definitions, which -- is how the REPL works. @@ -1352,9 +1366,9 @@ adToPrim v = putV $ AD.primitive v - adBinOp op x y = - either (const Nothing) Just $ AD.doOp op [x, y] - adUnOp op x = either (const Nothing) Just $ AD.doOp op [x] + adBinOp op x y i = + either (const Nothing) Just $ AD.doOp op [x, y] i + adUnOp op x i = either (const Nothing) Just $ AD.doOp op [x] i fun1 f = TermValue Nothing $ ValueFun $ \x -> f x @@ -1414,7 +1428,8 @@ pure . ValueFun $ \g -> pure . ValueFun $ \h -> f x y z a b c d e g h - bopDef fs = fun2 $ \x y -> + bopDef fs = fun2 $ \x y -> do + i <- getCounter case (x, y) of (ValuePrim x', ValuePrim y') | Just z <- msum $ map (`bopDef'` (x', y')) fs -> do @@ -1423,7 +1438,8 @@ _ | Just x' <- getAD x, Just y' <- getAD y, - Just z <- msum $ map (`bopDefAD` (x', y')) fs -> do + Just (z, i') <- msum $ map (`bopDefAD` (x', y', i)) fs -> do + putCounter i' breakOnNaN [adToPrim x', adToPrim y'] $ adToPrim z pure $ putAD z _ -> @@ -1438,9 +1454,10 @@ x' <- valf x y' <- valf y retf =<< op x' y' - bopDefAD (_, _, _, dop) (x, y) = dop x y + bopDefAD (_, _, _, dop) (x, y, i) = dop x y i - unopDef fs = fun1 $ \x -> + unopDef fs = fun1 $ \x -> do + i <- getCounter case x of (ValuePrim x') | Just r <- msum $ map (`unopDef'` x') fs -> do @@ -1448,7 +1465,8 @@ pure $ ValuePrim r _ | Just x' <- getAD x, - Just r <- msum $ map (`unopDefAD'` x') fs -> do + Just (r, i') <- msum $ map (`unopDefAD'` (x', i)) fs -> do + putCounter i' breakOnNaN [adToPrim x'] $ adToPrim r pure $ putAD r _ -> @@ -1460,9 +1478,10 @@ unopDef' (valf, retf, op, _) x = do x' <- valf x retf =<< op x' - unopDefAD' (_, _, _, dop) = dop + unopDefAD' (_, _, _, dop) (x, i) = dop x i - tbopDef op f = fun1 $ \v -> + tbopDef op f = fun1 $ \v -> do + i <- getCounter case fromTuple v of Just [ValuePrim x, ValuePrim y] | Just x' <- getV x, @@ -1473,7 +1492,8 @@ Just [x, y] | Just x' <- getAD x, Just y' <- getAD y, - Right z <- AD.doOp op [x', y'] -> do + Right (z, i') <- AD.doOp op [x', y'] i -> do + putCounter i' breakOnNaN [adToPrim x', adToPrim y'] $ adToPrim z pure $ putAD z _ -> @@ -1978,7 +1998,7 @@ -- exposed by the AD module? fun3 $ \f v s -> do -- Get the depth - depth <- length <$> stacktrace + depth <- adDepth -- Augment the values let v' = @@ -1996,17 +2016,27 @@ let o' = fst $ valueAccum (\a b -> (b : a, b)) [] o -- For each output.. - let m = flip map (zip o' s') $ \(on, sn) -> case on of - -- If it is a VJP variable of the correct depth, run - -- deriveTapqe on it- and its corresponding seed - (ValueAD d (AD.VJP (AD.VJPValue t))) - | d == depth -> - (putAD $ AD.tapePrimal t, AD.deriveTape t sn) - -- Otherwise, its partial derivatives are all 0 - _ -> (on, M.empty) + m <- + forM + (zip o' s') + ( \(on, sn) -> case on of + -- If it is a VJP variable of the correct depth, run + -- deriveTape on it- and its corresponding seed + (ValueAD d (AD.VJP (AD.VJPValue t))) + | d == depth -> + getCounter + >>= either + (pure . Left) + (\(m', i) -> putCounter i $> Right (putAD $ AD.tapePrimal t, m')) + . AD.deriveTape t sn + -- Otherwise, its partial derivatives are all 0 + _ -> pure $ Right (on, M.empty) + ) + <&> either (error . show) id . sequence -- Add together every derivative - let drvs = M.map (Just . putAD) $ M.unionsWith add $ map snd m + drvs' <- AD.unionsWithM add (map snd m) + let drvs = M.map (Just . putAD) drvs' -- Extract the output values, and the partial derivatives let ov = modifyValue (\i _ -> fst $ m !! (length m - 1 - i)) o @@ -2031,15 +2061,17 @@ -- TODO: Perhaps this could be fully abstracted by AD? -- Making addFor private would be nice.. add x y = - fromRight (error "jvp: illtyped add") $ - AD.doOp (AD.OpBin $ AD.addFor $ P.primValueType $ AD.primitive x) [x, y] + getCounter + >>= either + (error . show) + (\(a, b) -> putCounter b >> pure a) + . AD.doOp (AD.OpBin $ AD.addFor $ P.primValueType $ AD.primitive x) [x, y] def "jvp2" = Just $ -- TODO: This could be much better. Currently, it is very inefficient -- Perhaps creating JVPValues could be abstracted into a function -- exposed by the AD module? fun3 $ \f v s -> do - -- Get the depth - depth <- length <$> stacktrace + depth <- adDepth -- Turn the seeds into a list of ADValues let s' = @@ -2067,7 +2099,8 @@ mapM ( \on -> case on of -- If it is a JVP variable of the correct depth, return its primal and derivative - (ValueAD d (AD.JVP (AD.JVPValue pv dv))) | d == depth -> Just (putAD pv, putAD dv) + (ValueAD d (AD.JVP (AD.JVPValue pv dv))) + | d == depth -> Just (putAD pv, putAD dv) -- Otherwise, its partial derivatives are all 0 _ -> (on,) . ValuePrim . putV . P.blankPrimValue . P.primValueType . AD.primitive <$> getAD on ) @@ -2120,7 +2153,7 @@ interpretDecs :: Ctx -> [Dec] -> F ExtOp Env interpretDecs ctx decs = runEvalM (ctxImports ctx) $ do - env <- foldM evalDec (ctxEnv ctx) decs + (env, _) <- evalDecs (ctxEnv ctx) decs -- We need to extract any new existential sizes and add them as -- ordinary bindings to the context, or we will not be able to -- look up their values later. diff --git a/src/Language/Futhark/Interpreter/AD.hs b/src/Language/Futhark/Interpreter/AD.hs --- a/src/Language/Futhark/Interpreter/AD.hs +++ b/src/Language/Futhark/Interpreter/AD.hs @@ -5,26 +5,63 @@ Tape (..), VJPValue (..), JVPValue (..), + Counter (..), + Depth (..), doOp, addFor, tapePrimal, primitive, varPrimal, deriveTape, + unionWithM, + unionsWithM, ) where import Control.Monad (foldM, zipWithM) -import Data.Either (fromRight, isRight) -import Data.List (find, foldl') +import Control.Monad.Trans.Class (lift) +import Control.Monad.Trans.Except (ExceptT, catchE, runExceptT, throwE) +import Control.Monad.Trans.State (State, get, modify, runState) +import Data.Either (isRight) +import Data.Foldable (find, foldlM) +import Data.Functor ((<&>)) import Data.Map qualified as M -import Data.Maybe (fromMaybe) +import Data.Maybe (fromJust, fromMaybe) import Data.Text qualified as T import Futhark.AD.Derivatives (pdBinOp, pdBuiltin, pdUnOp) import Futhark.Analysis.PrimExp (PrimExp (..)) import Language.Futhark.Core (VName (..), nameFromString, nameFromText) import Language.Futhark.Primitive + ( BinOp (Add, FAdd, FMul, LogAnd, LogOr, Mul), + CmpOp, + ConvOp, + Overflow (OverflowWrap), + PrimType (Bool, FloatType, IntType), + PrimValue (BoolValue), + UnOp, + binOpType, + blankPrimValue, + cmpOpType, + convOpType, + doBinOp, + doCmpOp, + doConvOp, + doUnOp, + flipConvOp, + primFuns, + primValueType, + unOpType, + ) +-- | Used to uniquely identify values. +newtype Counter = Counter Int + deriving (Eq, Ord, Num, Show) + +type ADMonad = ExceptT String (State Counter) + +incCounter :: ADMonad () +incCounter = lift $ modify $ \i -> i + 1 + -- Mathematical operations subject to AD. data Op = OpBin BinOp @@ -70,7 +107,10 @@ mulFor Bool = LogAnd mulFor t = error $ "mulFor: " ++ show t -type Depth = Int +-- | An indication of the nesting depth of AD. This is used to avoid +-- pertubation confusion. +newtype Depth = Depth Int + deriving (Ord, Eq, Show) -- Types and utility functions-- -- When taking the partial derivative of a function, we @@ -92,7 +132,7 @@ depth :: ADValue -> Depth depth (Variable d _) = d -depth (Constant _) = 0 +depth (Constant _) = Depth 0 primal :: ADValue -> ADValue primal (Variable _ (VJP (VJPValue t))) = tapePrimal t @@ -113,29 +153,29 @@ varPrimal (VJP (VJPValue t)) = primitive $ tapePrimal t varPrimal (JVP (JVPValue v _)) = primitive $ primal v --- Evaluates a PrimExp using doOp -evalPrimExp :: M.Map VName ADValue -> PrimExp VName -> Either String ADValue +-- Evaluates a PrimExp using doOp' +evalPrimExp :: M.Map VName ADValue -> PrimExp VName -> ADMonad ADValue evalPrimExp m (LeafExp n _) = - maybe (Left $ "Unknown variable " <> show n) Right $ M.lookup n m + maybe (throwE $ "Unknown variable " <> show n) pure $ M.lookup n m evalPrimExp _ (ValueExp pv) = - Right $ Constant pv + pure $ Constant pv evalPrimExp m (BinOpExp op x y) = do x' <- evalPrimExp m x y' <- evalPrimExp m y - doOp (OpBin op) [x', y'] + doOp' (OpBin op) [x', y'] evalPrimExp m (CmpOpExp op x y) = do x' <- evalPrimExp m x y' <- evalPrimExp m y - doOp (OpCmp op) [x', y'] + doOp' (OpCmp op) [x', y'] evalPrimExp m (UnOpExp op x) = do x' <- evalPrimExp m x - doOp (OpUn op) [x'] + doOp' (OpUn op) [x'] evalPrimExp m (ConvOpExp op x) = do x' <- evalPrimExp m x - doOp (OpConv op) [x'] + doOp' (OpConv op) [x'] evalPrimExp m (FunExp fn p _) = do p' <- mapM (evalPrimExp m) p - doOp (OpFn fn) p' + doOp' (OpFn fn) p' -- Returns a list of PrimExps calculating the partial -- derivative of each operands of a given operation @@ -151,20 +191,25 @@ -- This function performs a mathematical operation on a -- list of operands, performing automatic differentiation -- if one or more operands is a Variable (of depth > 0) -doOp :: Op -> [ADValue] -> Either String ADValue -doOp op o +doOp :: Op -> [ADValue] -> Counter -> Either String (ADValue, Counter) +doOp op o uid = case runState (runExceptT $ doOp' op o) uid of + (Left s, _) -> Left s + (Right v, uid') -> Right (v, uid') + +doOp' :: Op -> [ADValue] -> ADMonad ADValue +doOp' op o | not $ opTypeMatch op (map primValueType pv) = -- This function may be called with arguments of invalid types, -- because it is used as part of an overloaded operator. - Left $ unwords ["invalid types for op", show op, "and operands", show o] + throwE $ unwords ["invalid types for op", show op, "and operands", show o] | otherwise = do let dep = case op of - OpCmp _ -> 0 -- AD is not well-defined for comparason operations + OpCmp _ -> Depth 0 -- AD is not well-defined for comparason operations -- There are no derivatives for those written in -- PrimExp (check lookupPDs) _ -> maximum (map depth o) - if dep == 0 - then maybe (Left "failed to evaluate const") Right constCase + if dep == Depth 0 + then maybe (throwE "failed to evaluate const") pure constCase <* incCounter else nonconstCase dep where pv = map primitive o @@ -203,7 +248,7 @@ (OpFn fn, _) -> do (_, _, f) <- M.lookup fn primFuns f pv - _ -> error "doOp: opTypeMatch" + _ -> error "doOp': opTypeMatch" nonconstCase dep = do -- In this case, some values are variables. We therefore @@ -211,7 +256,7 @@ -- First, we calculate the value for the previous depth let oprev = map (primalFor dep) o - vprev <- doOp op oprev + vprev <- doOp' op oprev -- Then we separate the values of the maximum depth from -- those of a lower depth @@ -221,8 +266,8 @@ -- Finally, we perform the necessary steps for the given -- type of AD Just (Right (VJP {})) -> - Right . Variable dep . VJP . VJPValue $ - vjpHandleOp op (map extractVJP o') vprev + Variable dep . VJP . VJPValue + <$> vjpHandleOp op (map extractVJP o') vprev Just (Right (JVP {})) -> Variable dep . JVP . JVPValue vprev <$> jvpHandleOp op (map extractJVP o') @@ -231,7 +276,7 @@ -- least one variable of depth > 0 error "find isRight" -calculatePDs :: Op -> [ADValue] -> [ADValue] +calculatePDs :: Op -> [ADValue] -> ADMonad [ADValue] calculatePDs op args = -- Create a unique VName for each operand let n = map (\i -> VName (nameFromString $ "x" ++ show i) i) [1 .. length args] @@ -244,7 +289,7 @@ fromMaybe (error "lookupPDs failed") $ lookupPDs op $ zipWith (\v val -> LeafExp v $ primValueType $ primitive val) n args - res = map (either (error . ("evalPrimExp failed: " <>)) id . evalPrimExp m) pde + res = mapM (\x -> catchE (evalPrimExp m x) $ error . ("evalPrimExp failed: " <>)) pde in res -- VJP / Reverse mode automatic differentiation-- @@ -255,63 +300,107 @@ deriving (Show) -- | Represents a computation tree, as well as every intermediate --- value in its evaluation. TODO: make this a graph. +-- value in its evaluation. data Tape = -- | This represents a variable. Each variable is given a unique ID, -- and has an initial value - TapeID Depth ADValue + TapeID Counter ADValue | -- | This represents a constant. TapeConst ADValue | -- | This represents the application of a mathematical operation. -- Each parameter is given by its Tape, and the return value of -- the operation is saved - TapeOp Op [Tape] ADValue + TapeOp Op [Tape] Counter ADValue deriving (Show) -- | Returns the primal value of a Tape. tapePrimal :: Tape -> ADValue tapePrimal (TapeID _ v) = v tapePrimal (TapeConst v) = v -tapePrimal (TapeOp _ _ v) = v +tapePrimal (TapeOp _ _ _ v) = v -- This updates Tape of a VJPValue with a new operation, -- treating all operands of a lower depth as constants -vjpHandleOp :: Op -> [Either ADValue VJPValue] -> ADValue -> Tape +vjpHandleOp :: Op -> [Either ADValue VJPValue] -> ADValue -> ADMonad Tape vjpHandleOp op p v = do - TapeOp op (map toTape p) v + i <- lift get + pure $ TapeOp op (map toTape p) i v where toTape (Left v') = TapeConst v' toTape (Right (VJPValue t)) = t +unionWithM :: (Monad m, Ord k) => (a -> a -> m a) -> M.Map k a -> M.Map k a -> m (M.Map k a) +unionWithM f m1 m2 = do + let m = M.union (M.difference m1 m2) (M.difference m2 m1) + let k = M.keys $ M.intersection m1 m2 + v <- mapM (\k' -> f (fromJust $ M.lookup k' m1) (fromJust $ M.lookup k' m2)) k + pure $ foldl (\m' (k', v') -> M.insert k' v' m') m (zip k v) + +unionsWithM :: (Foldable f, Monad m, Ord k) => (a -> a -> m a) -> f (M.Map k a) -> m (M.Map k a) +unionsWithM f = foldM (unionWithM f) M.empty + -- | This calculates every partial derivative of a 'Tape'. The result -- is a map of the partial derivatives, each key corresponding to the -- ID of a free variable (see TapeID). -deriveTape :: Tape -> ADValue -> M.Map Int ADValue -deriveTape (TapeID i _) s = M.fromList [(i, s)] -deriveTape (TapeConst _) _ = M.empty -deriveTape (TapeOp op p _) s = - -- Calculate the new sensitivities - let s'' = case op of - OpConv op' -> - -- In case of type conversion, simply convert the sensitivity - [ fromRight (error "deriveTape: doOp failed") $ - doOp (OpConv $ flipConvOp op') [s] - ] - _ -> - map (mul s) $ calculatePDs op $ map tapePrimal p +deriveTape :: Tape -> ADValue -> Counter -> Either String (M.Map Counter ADValue, Counter) +deriveTape tp s uid = case runState (runExceptT $ deriveTape' tp s) uid of + (Left e, _) -> Left e + (Right v, uid') -> Right (v, uid') - -- Propagate the new sensitivities - pd = zipWith deriveTape p s'' - in -- Add up the results - foldl' (M.unionWith add) M.empty pd +deriveTape' :: Tape -> ADValue -> ADMonad (M.Map Counter ADValue) +deriveTape' (TapeID i _) s = pure $ M.singleton i s +deriveTape' (TapeConst _) _ = pure M.empty +deriveTape' tp@(TapeOp op p uid _) s = + fst <$> derive tp s M.empty (countReferences p $ M.singleton (-uid - 1) 1) where - add x y = - fromRight (error "deriveTape: add failed") $ - doOp (OpBin $ addFor $ opReturnType op) [x, y] - mul x y = - fromRight (error "deriveTape: mul failed") $ - doOp (OpBin $ mulFor $ opReturnType op) [x, y] + add x y = doOp' (OpBin $ addFor $ opReturnType op) [x, y] + mul x y = doOp' (OpBin $ mulFor $ opReturnType op) [x, y] + madd :: Counter -> ADValue -> M.Map Counter ADValue -> ADMonad (M.Map Counter ADValue) + madd i a m = case M.lookup i m of + Just b -> add a b <&> (\x -> M.insert i x m) + Nothing -> pure $ M.insert i a m + derive :: + Tape -> + ADValue -> + M.Map Counter ADValue -> + M.Map Counter Int -> + ADMonad (M.Map Counter ADValue, M.Map Counter Int) + derive (TapeID i _) s' ss rs = madd i s' ss <&> (,rs) + derive (TapeConst _) _ ss rs = pure (ss, rs) + derive (TapeOp op' p' uid' _) s' ss rs = do + -- Decrease the reference counter + let r = fromJust (M.lookup (-uid' - 1) rs) - 1 + rs' = M.insert (-uid' - 1) r rs + -- Add the sensitivity + ss' <- madd (-uid' - 1) s' ss + -- If there are still more references left, do nothing + if r > 0 + then pure (ss', rs') + else -- Otherwise, derive the tape + if r == 0 + then do + let s'' = fromJust (M.lookup (-uid' - 1) ss') + + -- Calculate the new sensitivities + s''' <- case op' of + OpConv op'' -> + -- In case of type conversion, simply convert the sensitivity + sequence [doOp' (OpConv $ flipConvOp op'') [s'']] + _ -> calculatePDs op' (map tapePrimal p') >>= mapM (mul s'') + + -- Propagate the new sensitivities + foldlM (\(ss'', rs'') (p'', s'''') -> derive p'' s'''' ss'' rs'') (ss', rs') $ zip p' s''' + else error "TODO: This branch is unreachable unless `countReferences` undercounts" + countReferences :: [Tape] -> M.Map Counter Int -> M.Map Counter Int + countReferences p' d' = foldl f d' p' + f d'' x = + case x of + (TapeOp _ p'' uid'' _) -> case M.lookup (-uid'' - 1) d'' of + Just v -> M.insert (-uid'' - 1) (v + 1) d'' + Nothing -> countReferences p'' $ M.insert (-uid'' - 1) 1 d'' + _ -> d'' + -- JVP / Forward mode automatic differentiation-- -- | In JVP, the derivative of the variable must be saved. This is @@ -322,16 +411,16 @@ -- | This calculates the tangent part of the JVPValue resulting -- from the application of a mathematical operation on one or more -- JVPValues. -jvpHandleOp :: Op -> [Either ADValue JVPValue] -> Either String ADValue +jvpHandleOp :: Op -> [Either ADValue JVPValue] -> ADMonad ADValue jvpHandleOp op p = do case op of OpConv _ -> -- In case of type conversion, simply convert -- the old tangent - doOp op [tangent $ head p] + doOp' op [tangent $ head p] _ -> do -- Calculate the new tangent using the chain rule - let pds = calculatePDs op $ map primal' p + pds <- calculatePDs op $ map primal' p vs <- zipWithM mul pds $ map tangent p foldM add (Constant $ blankPrimValue op_t) vs where @@ -340,5 +429,5 @@ primal' (Right (JVPValue v _)) = v tangent (Left _) = Constant $ blankPrimValue $ opReturnType op tangent (Right (JVPValue _ d)) = d - add x y = doOp (OpBin $ addFor $ opReturnType op) [x, y] - mul x y = doOp (OpBin $ mulFor $ opReturnType op) [x, y] + add x y = doOp' (OpBin $ addFor $ opReturnType op) [x, y] + mul x y = doOp' (OpBin $ mulFor $ opReturnType op) [x, y] diff --git a/src/Language/Futhark/Interpreter/Values.hs b/src/Language/Futhark/Interpreter/Values.hs --- a/src/Language/Futhark/Interpreter/Values.hs +++ b/src/Language/Futhark/Interpreter/Values.hs @@ -29,8 +29,8 @@ ) where +import Control.Monad.Identity import Data.Array -import Data.Bifunctor (Bifunctor (second)) import Data.List (genericLength) import Data.Map qualified as M import Data.Maybe @@ -111,7 +111,7 @@ | -- The shape, the update function, and the array. ValueAcc ValueShape (Value m -> Value m -> m (Value m)) !(Array Int (Value m)) | -- A primitive value with added information used in automatic differentiation - ValueAD Int AD.ADVariable + ValueAD AD.Depth AD.ADVariable instance Show (Value m) where show (ValuePrim v) = "ValuePrim " <> show v <> "" @@ -198,20 +198,9 @@ -- TODO: Perhaps there is some clever way to reuse the code between -- valueAccum and valueAccumLM valueAccum :: (a -> Value m -> (a, Value m)) -> a -> Value m -> (a, Value m) -valueAccum f i v@(ValuePrim {}) = f i v -valueAccum f i v@(ValueAD {}) = f i v -valueAccum f i (ValueRecord m) = - second ValueRecord $ M.mapAccum (valueAccum f) i m -valueAccum f i (ValueArray s a) = do - -- TODO: This could probably be better - -- Transform into a map - let m = M.fromList $ assocs a - -- Accumulate over the map - let (i', m') = M.mapAccum (valueAccum f) i m - -- Transform back into an array and return - let a' = array (bounds a) (M.toList m') - (i', ValueArray s a') -valueAccum _ _ v = error $ "valueAccum not implemented for " ++ show v +valueAccum f i = runIdentity . valueAccumLM f' i + where + f' acc v = pure $ f acc v valueAccumLM :: (Monad f) => (a -> Value m -> f (a, Value m)) -> a -> Value m -> f (a, Value m) valueAccumLM f i v@(ValuePrim {}) = f i v @@ -228,6 +217,9 @@ -- Transform back into an array and return let a' = array (bounds a) (M.toList m') pure (i', ValueArray s a') +valueAccumLM f i (ValueSum shape c vs) = do + (a, vs') <- mapAccumLM (valueAccumLM f) i vs + pure (a, ValueSum shape c vs') valueAccumLM _ _ v = error $ "valueAccum not implemented for " ++ show v -- | Does the value correspond to an empty array? diff --git a/src/Language/Futhark/Primitive.hs b/src/Language/Futhark/Primitive.hs --- a/src/Language/Futhark/Primitive.hs +++ b/src/Language/Futhark/Primitive.hs @@ -1210,6 +1210,10 @@ f32 "sqrt32" sqrt, f64 "sqrt64" sqrt, -- + f16 "rsqrt16" $ recip . sqrt, + f32 "rsqrt32" $ recip . sqrt, + f64 "rsqrt64" $ recip . sqrt, + -- f16 "cbrt16" $ convFloat . cbrtf . convFloat, f32 "cbrt32" cbrtf, f64 "cbrt64" cbrt, @@ -1238,6 +1242,10 @@ f32 "sin32" sin, f64 "sin64" sin, -- + f16 "sinpi16" $ sin . (pi *), + f32 "sinpi32" $ sin . (pi *), + f64 "sinpi64" $ sin . (pi *), + -- f16 "sinh16" sinh, f32 "sinh32" sinh, f64 "sinh64" sinh, @@ -1246,6 +1254,10 @@ f32 "cos32" cos, f64 "cos64" cos, -- + f16 "cospi16" $ cos . (pi *), + f32 "cospi32" $ cos . (pi *), + f64 "cospi64" $ cos . (pi *), + -- f16 "cosh16" cosh, f32 "cosh32" cosh, f64 "cosh64" cosh, @@ -1254,6 +1266,10 @@ f32 "tan32" tan, f64 "tan64" tan, -- + f16 "tanpi16" $ tan . (pi *), + f32 "tanpi32" $ tan . (pi *), + f64 "tanpi64" $ tan . (pi *), + -- f16 "tanh16" tanh, f32 "tanh32" tanh, f64 "tanh64" tanh, @@ -1262,6 +1278,10 @@ f32 "asin32" asin, f64 "asin64" asin, -- + f16 "asinpi16" $ (/ pi) . asin, + f32 "asinpi32" $ (/ pi) . asin, + f64 "asinpi64" $ (/ pi) . asin, + -- f16 "asinh16" asinh, f32 "asinh32" asinh, f64 "asinh64" asinh, @@ -1270,6 +1290,10 @@ f32 "acos32" acos, f64 "acos64" acos, -- + f16 "acospi16" $ (/ pi) . acos, + f32 "acospi32" $ (/ pi) . acos, + f64 "acospi64" $ (/ pi) . acos, + -- f16 "acosh16" acosh, f32 "acosh32" acosh, f64 "acosh64" acosh, @@ -1278,6 +1302,10 @@ f32 "atan32" atan, f64 "atan64" atan, -- + f16 "atanpi16" $ (/ pi) . atan, + f32 "atanpi32" $ (/ pi) . atan, + f64 "atanpi64" $ (/ pi) . atan, + -- f16 "atanh16" atanh, f32 "atanh32" atanh, f64 "atanh64" atanh, @@ -1400,6 +1428,33 @@ \case [FloatValue (Float64Value x), FloatValue (Float64Value y)] -> Just $ FloatValue $ Float64Value $ atan2 x y + _ -> Nothing + ) + ), + ( "atan2pi_16", + ( [FloatType Float16, FloatType Float16], + FloatType Float16, + \case + [FloatValue (Float16Value x), FloatValue (Float16Value y)] -> + Just $ FloatValue $ Float16Value $ atan2 x y / pi + _ -> Nothing + ) + ), + ( "atan2pi_32", + ( [FloatType Float32, FloatType Float32], + FloatType Float32, + \case + [FloatValue (Float32Value x), FloatValue (Float32Value y)] -> + Just $ FloatValue $ Float32Value $ atan2 x y / pi + _ -> Nothing + ) + ), + ( "atan2pi_64", + ( [FloatType Float64, FloatType Float64], + FloatType Float64, + \case + [FloatValue (Float64Value x), FloatValue (Float64Value y)] -> + Just $ FloatValue $ Float64Value $ atan2 x y / pi _ -> Nothing ) ), diff --git a/src/Language/Futhark/TypeChecker/Consumption.hs b/src/Language/Futhark/TypeChecker/Consumption.hs --- a/src/Language/Futhark/TypeChecker/Consumption.hs +++ b/src/Language/Futhark/TypeChecker/Consumption.hs @@ -616,6 +616,7 @@ && not (S.null (aliases t `S.intersection` (cons <> obs))) ) $ lift . addError loop_loc mempty + $ withIndexLink "aliases-previously-returned" $ "Return value for consuming loop parameter" <+> dquotes (prettyName pat_v) <+> "aliases previously returned value."