diff --git a/docs/c-api.rst b/docs/c-api.rst --- a/docs/c-api.rst +++ b/docs/c-api.rst @@ -81,6 +81,13 @@ you must not pass values between them. They have the same C type, so this is an easy mistake to make. + After you have created a context object, you must immediately call + :c:func:`futhark_context_get_error`, which will return non-``NULL`` + if initialisation failed. If initialisation has failed, then you + still need to call :c:func:`futhark_context_free` to release + resources used for the context object, but you may not use the + context object for anything else. + .. c:function:: void futhark_context_free(struct futhark_context *ctx) Free the context object. It must not be used again. The @@ -207,6 +214,17 @@ sure to call :c:func:`futhark_context_sync` before using the value of ``out0``. +The exact behaviour of the exit code depends on the backend. For the +sequential C backend, errors will always be available when the entry +point returns, and :c:func:`futhark_context_sync` will always return +success. When using a GPU backend such as ``cuda`` or ``opencl``, the +entry point may still be running asynchronous operations when it +returns, in which case the entry point may return zero successfully, +even though execution has already (or will) fail. These problems will +be reported when :c:func:`futhark_context_sync` is called. When using +GPU backends, be careful to check the return code of *both* the entry +point itself, and :c:func:`futhark_context_sync`. + GPU --- @@ -325,3 +343,30 @@ During :c:func:`futhark_context_new`, read PTX code from the given file instead of using the embedded program. + +General guarantees +------------------ + +Calling an entry point, or interacting with Futhark values through the +functions listed above, has no system-wide side effects, such as +writing to the file system, launching processes, or performing network +connections. Defects in the program or Futhark compiler itself can +with high probability result only in the consumption of CPU or GPU +resources, or a process crash. + +Using the ``#[unsafe]`` attribute with in-place updates can result in +writes to arbitrary memory locations. A malicious program can likely +exploit this to obtain arbitrary code execution, just as with any +insecure C program. If you must run untrusted code, consider using +the ``--safe`` command line option to instruct the compiler to disable +``#[unsafe]``. + +Initialising a Futhark context likewise has no side effects, except if +explicitly configured differently, such as by using +:c:func:`futhark_context_config_dump_program_to`. In its default +configuration, Futhark will not access the file system. + +Note that for the GPU backends, the underlying API (such as CUDA or +OpenCL) may perform file system operations during startup, and perhaps +for caching GPU kernels in some cases. This is beyond Futhark's +control. diff --git a/docs/language-reference.rst b/docs/language-reference.rst --- a/docs/language-reference.rst +++ b/docs/language-reference.rst @@ -728,12 +728,14 @@ ``^``, ``&``, ``|``, ``>>``, ``<<`` Bitwise operators, respectively bitwise xor, and, or, arithmetic - shift right and left, and logical shift right. Shift amounts - must be non-negative and the operands must be integers. Note - that, unlike in C, bitwise operators have *higher* priority than - arithmetic operators. This means that ``x & y == z`` is - understood as ``(x & y) == z``, rather than ``x & (y == z)`` as - it would in C. Note that the latter is a type error in Futhark + shift right and left, and logical shift right. **Shifting is + undefined if the right operand is negative, or greater than or + equal to the length in bits of the left operand.** + + Note that, unlike in C, bitwise operators have *higher* priority + than arithmetic operators. This means that ``x & y == z`` is + understood as ``(x & y) == z``, rather than ``x & (y == z)`` as it + would in C. Note that the latter is a type error in Futhark anyhow. ``==``, ``!=`` diff --git a/docs/usage.rst b/docs/usage.rst --- a/docs/usage.rst +++ b/docs/usage.rst @@ -19,6 +19,8 @@ useful mostly for testing and benchmarking. Libraries can be called from non-Futhark code. +.. _executable: + Compiling to Executable ----------------------- @@ -450,3 +452,18 @@ Futhark arrays are mapped to either the Numpy ``ndarray`` type or the `pyopencl.array `_ type. Scalars are mapped to Numpy scalar types. + +Reproducibility +--------------- + +The Futhark compiler is deterministic by design, meaning that +repeatedly compiling the *same program* with the *same compilation +flags* and using the *same version* of the compiler will produce +identical output every time. + +Note that this only applies to the code generated by the Futhark +compiler itself. When compiling to an executable with one of the C +backends (see :ref:`executable`), Futhark will invoke a C compiler +that may not be perfectly reproducible. In such cases the generated +``.c`` and ``.h`` files will be reproducible, but the final executable +may not. diff --git a/futhark.cabal b/futhark.cabal --- a/futhark.cabal +++ b/futhark.cabal @@ -1,7 +1,7 @@ cabal-version: 2.4 name: futhark -version: 0.16.1 +version: 0.16.2 synopsis: An optimising compiler for a functional, array-oriented language. description: Futhark is a small programming language designed to be compiled to diff --git a/rts/c/util.h b/rts/c/util.h --- a/rts/c/util.h +++ b/rts/c/util.h @@ -2,7 +2,7 @@ // // Various helper functions that are useful in all generated C code. -static const char *fut_progname = "(some Futhark code)"; +static const char *fut_progname = "(embedded Futhark)"; static void futhark_panic(int eval, const char *fmt, ...) { va_list ap; diff --git a/rts/python/opencl.py b/rts/python/opencl.py --- a/rts/python/opencl.py +++ b/rts/python/opencl.py @@ -204,7 +204,7 @@ if (len(program_src) >= 0): return cl.Program(self.ctx, program_src).build( ["-DLOCKSTEP_WIDTH={}".format(lockstep_width)] - + ["-D{}={}".format(s.replace('z', 'zz').replace('.', 'zi'),v) for (s,v) in self.sizes.items()]) + + ["-D{}={}".format(s.replace('z', 'zz').replace('.', 'zi').replace('#', 'zh'),v) for (s,v) in self.sizes.items()]) def opencl_alloc(self, min_size, tag): min_size = 1 if min_size == 0 else min_size @@ -219,6 +219,11 @@ cl.enqueue_copy(self.queue, failure, self.global_failure, is_blocking=True) self.failure_is_an_option = np.int32(0) if failure[0] >= 0: + # Reset failure information. + cl.enqueue_fill_buffer(self.queue, self.global_failure, np.int32(-1), 0, np.int32().itemsize) + + # Read failure args. failure_args = np.empty(self.global_failure_args_max+1, dtype=np.int32) cl.enqueue_copy(self.queue, failure_args, self.global_failure_args, is_blocking=True) + raise Exception(self.failure_msgs[failure[0]].format(*failure_args)) diff --git a/src/Futhark/CLI/Autotune.hs b/src/Futhark/CLI/Autotune.hs --- a/src/Futhark/CLI/Autotune.hs +++ b/src/Futhark/CLI/Autotune.hs @@ -348,6 +348,12 @@ (ReqArg (\prog -> Right $ \config -> config { optFuthark = Just prog }) "PROGRAM") "The binary used for operations (defaults to 'futhark')." + , Option [] ["pass-option"] + (ReqArg (\opt -> + Right $ \config -> + config { optExtraOptions = opt : optExtraOptions config }) + "OPT") + "Pass this option to programs being run." , Option [] ["tuning"] (ReqArg (\s -> Right $ \config -> config { optTuning = Just s }) "EXTENSION") diff --git a/src/Futhark/CLI/Bench.hs b/src/Futhark/CLI/Bench.hs --- a/src/Futhark/CLI/Bench.hs +++ b/src/Futhark/CLI/Bench.hs @@ -165,15 +165,19 @@ progressBar :: Int -> Int -> Int -> String progressBar cur bound steps = "[" ++ map cell [1..steps] ++ "] " ++ show cur ++ "/" ++ show bound - where step_size = bound `div` steps + where step_size :: Double + step_size = fromIntegral bound / fromIntegral steps + cur' = fromIntegral cur chars = " ▏▎▍▍▌▋▊▉█" char i = fromMaybe ' ' $ maybeNth (i::Int) chars + num_chars = fromIntegral $ length chars cell :: Int -> Char cell i - | i * step_size <= cur = char 9 - | otherwise = char (((cur - (i-1) * step_size) * length chars) - `div` step_size) + | i' * step_size <= cur' = char 9 + | otherwise = char (floor (((cur' - (i'-1) * step_size) * num_chars) + / step_size)) + where i' = fromIntegral i descString :: String -> Int -> String descString desc pad_to = desc ++ ": " ++ replicate (pad_to - length desc) ' ' diff --git a/src/Futhark/CodeGen/Backends/CCUDA/Boilerplate.hs b/src/Futhark/CodeGen/Backends/CCUDA/Boilerplate.hs --- a/src/Futhark/CodeGen/Backends/CCUDA/Boilerplate.hs +++ b/src/Futhark/CodeGen/Backends/CCUDA/Boilerplate.hs @@ -386,6 +386,14 @@ ctx->failure_is_an_option = 0; if (failure_idx >= 0) { + // We have to clear global_failure so that the next entry point + // is not considered a failure from the start. + typename int32_t no_failure = -1; + CUDA_SUCCEED( + cuMemcpyHtoD(ctx->global_failure, + &no_failure, + sizeof(int32_t))); + typename int32_t args[$int:max_failure_args+1]; CUDA_SUCCEED( cuMemcpyDtoH(&args, diff --git a/src/Futhark/CodeGen/Backends/COpenCL/Boilerplate.hs b/src/Futhark/CodeGen/Backends/COpenCL/Boilerplate.hs --- a/src/Futhark/CodeGen/Backends/COpenCL/Boilerplate.hs +++ b/src/Futhark/CodeGen/Backends/COpenCL/Boilerplate.hs @@ -416,6 +416,14 @@ OPENCL_SUCCEED_OR_RETURN(clFinish(ctx->opencl.queue)); if (failure_idx >= 0) { + // We have to clear global_failure so that the next entry point + // is not considered a failure from the start. + typename cl_int no_failure = -1; + OPENCL_SUCCEED_OR_RETURN( + clEnqueueWriteBuffer(ctx->opencl.queue, ctx->global_failure, CL_TRUE, + 0, sizeof(cl_int), &no_failure, + 0, NULL, NULL)); + typename cl_int args[$int:max_failure_args+1]; OPENCL_SUCCEED_OR_RETURN( clEnqueueReadBuffer(ctx->opencl.queue, diff --git a/src/Futhark/CodeGen/Backends/GenericC.hs b/src/Futhark/CodeGen/Backends/GenericC.hs --- a/src/Futhark/CodeGen/Backends/GenericC.hs +++ b/src/Futhark/CodeGen/Backends/GenericC.hs @@ -1656,6 +1656,7 @@ mapM_ resetMemConst ps compileCode init_consts libDecl [C.cedecl|int init_constants($ty:ctx_ty *ctx) { + (void)ctx; int err = 0; $items:defs $items:init_consts' @@ -1666,6 +1667,7 @@ free_consts <- collect $ mapM_ freeConst ps libDecl [C.cedecl|int free_constants($ty:ctx_ty *ctx) { + (void)ctx; $items:free_consts return 0; }|] @@ -1988,7 +1990,7 @@ cached <- cacheMem name case cached of Just cur_size -> - stm [C.cstm|if ($exp:cur_size < $exp:size) { + stm [C.cstm|if ($exp:cur_size < (size_t)$exp:size) { $exp:name = realloc($exp:name, $exp:size); $exp:cur_size = $exp:size; }|] diff --git a/src/Futhark/CodeGen/Backends/GenericPython.hs b/src/Futhark/CodeGen/Backends/GenericPython.hs --- a/src/Futhark/CodeGen/Backends/GenericPython.hs +++ b/src/Futhark/CodeGen/Backends/GenericPython.hs @@ -294,7 +294,7 @@ -> [Option] -> Imp.Definitions op -> m String -compileProg module_name constructor imports defines ops userstate pre_timing options prog = do +compileProg module_name constructor imports defines ops userstate sync options prog = do src <- getNameSource let prog' = runCompilerM ops src userstate compileProg' maybe_shebang = @@ -321,14 +321,14 @@ case module_name of Just name -> do (entry_points, entry_point_types) <- - unzip <$> mapM compileEntryFun (filter (Imp.functionEntry . snd) funs) + unzip <$> mapM (compileEntryFun sync) (filter (Imp.functionEntry . snd) funs) return [ClassDef $ Class name $ Assign (Var "entry_points") (Dict entry_point_types) : map FunDef (constructor' : definitions ++ entry_points)] Nothing -> do let classinst = Assign (Var "self") $ simpleCall "internal" [] (entry_point_defs, entry_point_names, entry_points) <- - unzip3 <$> mapM (callEntryFun pre_timing) + unzip3 <$> mapM (callEntryFun sync) (filter (Imp.functionEntry . snd) funs) return (parse_options ++ ClassDef (Class "internal" $ map FunDef $ @@ -604,14 +604,14 @@ [srcmem, srcidx, Var "ct.c_byte"] stm $ Exp $ simpleCall "ct.memmove" [offset_call1, offset_call2, nbytes] -compileEntryFun :: (Name, Imp.Function op) +compileEntryFun :: [PyStmt] -> (Name, Imp.Function op) -> CompilerM op s (PyFunDef, (PyExp, PyExp)) -compileEntryFun entry = do +compileEntryFun sync entry = do (fname', params, prepareIn, body_lib, _, prepareOut, res, _) <- prepareEntry entry let ret = Return $ tupleOrSingle $ map snd res (pts, rts) = entryTypes $ snd entry return (Def fname' ("self" : params) $ - prepareIn ++ body_lib ++ prepareOut ++ [ret], + prepareIn ++ body_lib ++ prepareOut ++ sync ++ [ret], (String fname', Tuple [List (map String pts), List (map String rts)])) entryTypes :: Imp.Function op -> ([String], [String]) diff --git a/src/Futhark/CodeGen/ImpGen/Kernels/Base.hs b/src/Futhark/CodeGen/ImpGen/Kernels/Base.hs --- a/src/Futhark/CodeGen/ImpGen/Kernels/Base.hs +++ b/src/Futhark/CodeGen/ImpGen/Kernels/Base.hs @@ -1425,7 +1425,8 @@ -- Avoid loop for the common case where each thread is statically -- known to write at most one element. - if toExp' int32 per_group_elems == kernelGroupSize constants + localOps threadOperations $ + if toExp' int32 per_group_elems == kernelGroupSize constants then sWhen (offset + ltid .<. toExp' int32 w) $ copyDWIMFix (patElemName pe) [ltid + offset] (Var what) [ltid] else @@ -1441,7 +1442,8 @@ local_is <- localThreadIDs $ map snd dims is_for_thread <- mapM (dPrimV "thread_out_index") $ zipWith (+) group_is local_is - sWhen (isActive $ zip is_for_thread $ map fst dims) $ + localOps threadOperations $ + sWhen (isActive $ zip is_for_thread $ map fst dims) $ copyDWIMFix (patElemName pe) (map Imp.vi32 is_for_thread) (Var what) local_is compileGroupResult space pe (Returns _ what) = do diff --git a/src/Futhark/IR/Mem/IxFun.hs b/src/Futhark/IR/Mem/IxFun.hs --- a/src/Futhark/IR/Mem/IxFun.hs +++ b/src/Futhark/IR/Mem/IxFun.hs @@ -421,8 +421,8 @@ mid_dims = take (length dims - num_coercions) $ drop hd_len dims' num_rshps = length reshapes guard (num_rshps == 0 || (num_rshps == 1 && length mid_dims == 1)) - let dims'' = map snd $ sortBy (compare `on` fst) $ - zipWith (\ld n -> (ldPerm ld, ld { ldShape = n })) + let dims'' = permuteInv perm $ + zipWith (\ld n -> ld { ldShape = n }) dims' (newDims newshape) lmad' = LMAD off dims'' return $ IxFun (lmad' :| lmads) (newDims newshape) cg diff --git a/src/Futhark/IR/Prop/Types.hs b/src/Futhark/IR/Prop/Types.hs --- a/src/Futhark/IR/Prop/Types.hs +++ b/src/Futhark/IR/Prop/Types.hs @@ -5,7 +5,6 @@ rankShaped , arrayRank , arrayShape - , modifyArrayShape , setArrayShape , existential , uniqueness diff --git a/src/Language/Futhark/TypeChecker/Terms.hs b/src/Language/Futhark/TypeChecker/Terms.hs --- a/src/Language/Futhark/TypeChecker/Terms.hs +++ b/src/Language/Futhark/TypeChecker/Terms.hs @@ -2193,7 +2193,6 @@ occur [observation as loc] checkOccurences dflow - occurs <- consumeArg argloc argtype' (diet tp1') case anyConsumption dflow of Just c -> @@ -2201,7 +2200,11 @@ in zeroOrderType (mkUsage argloc "potential consumption in expression") msg tp1 _ -> return () - occur $ dflow `seqOccurences` occurs + occurs <- (dflow `seqOccurences`) <$> consumeArg argloc argtype' (diet tp1') + + checkIfConsumable loc $ S.map AliasBound $ allConsumed occurs + occur occurs + (argext, parsubst) <- case pname of Named pname' -> do @@ -2928,9 +2931,8 @@ let als = AliasBound nm `S.insert` aliases t in occur [observation als loc] --- | Proclaim that we have written to the given variable. -consume :: SrcLoc -> Aliasing -> TermTypeM () -consume loc als = do +checkIfConsumable :: SrcLoc -> Aliasing -> TermTypeM () +checkIfConsumable loc als = do vtable <- asks $ scopeVtable . termScope let consumable v = case M.lookup v vtable of Just (BoundV Local _ t) @@ -2940,9 +2942,15 @@ _ -> False case filter (not . consumable) $ map aliasVar $ S.toList als of v:_ -> typeError loc mempty $ - "Attempt to consume variable" <+> pquote (pprName v) + "Would consume variable" <+> pquote (pprName v) <> ", which is not allowed." - [] -> occur [consumption als loc] + [] -> return () + +-- | Proclaim that we have written to the given variable. +consume :: SrcLoc -> Aliasing -> TermTypeM () +consume loc als = do + checkIfConsumable loc als + occur [consumption als loc] -- | Proclaim that we have written to the given variable, and mark -- accesses to it and all of its aliases as invalid inside the given