diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,11 @@ # ChangeLog for wai +## 3.2.3 + +* Add documentation recommending streaming request bodies. [#818](https://github.com/yesodweb/wai/pull/818) +* Add two functions, `consumeRequestBodyStrict` and `consumeRequestBodyLazy`, + that are synonyms for `strictRequestBody` and `lazyRequestBody`. [#818](https://github.com/yesodweb/wai/pull/818) + ## 3.2.2.1 * Fix missing reexport of `getRequestBodyChunk` [#753](https://github.com/yesodweb/wai/issues/753) diff --git a/Network/Wai.hs b/Network/Wai.hs --- a/Network/Wai.hs +++ b/Network/Wai.hs @@ -61,8 +61,11 @@ , requestHeaderRange , requestHeaderReferer , requestHeaderUserAgent + -- $streamingRequestBodies , strictRequestBody + , consumeRequestBodyStrict , lazyRequestBody + , consumeRequestBodyLazy -- * Response , Response , StreamingBody @@ -315,11 +318,62 @@ ifRequest rpred middle app req | rpred req = middle app req | otherwise = app req - +-- $streamingRequestBodies +-- +-- == Streaming Request Bodies +-- +-- WAI is designed for streaming in request bodies, which allows you to process them incrementally. +-- You can stream in the request body using functions like 'getRequestBodyChunk', +-- the @wai-conduit@ package, or Yesod's @rawRequestBody@. +-- +-- In the normal case, incremental processing is more efficient, since it +-- reduces maximum total memory usage. +-- In the worst case, it helps protect your server against denial-of-service (DOS) attacks, in which +-- an attacker sends huge request bodies to your server. +-- +-- Consider these tips to avoid reading the entire request body into memory: +-- +-- * Look for library functions that support incremental processing. Sometimes these will use streaming +-- libraries like @conduit@, @pipes@, or @streaming@. +-- * Any attoparsec parser supports streaming input. For an example of this, see the +-- "Data.Conduit.Attoparsec" module in @conduit-extra@. +-- * Consider streaming directly to a file on disk. For an example of this, see the +-- "Data.Conduit.Binary" module in @conduit-extra@. +-- * If you need to direct the request body to multiple destinations, you can stream to both those +-- destinations at the same time. +-- For example, if you wanted to run an HMAC on the request body as well as parse it into JSON, +-- you could use Conduit's @zipSinks@ to send the data to @cryptonite-conduit@'s 'sinkHMAC' and +-- @aeson@'s Attoparsec parser. +-- * If possible, avoid processing large data on your server at all. +-- For example, instead of uploading a file to your server and then to AWS S3, +-- you can have the browser upload directly to S3. +-- +-- That said, sometimes it is convenient, or even necessary to read the whole request body into memory. +-- For these purposes, functions like 'strictRequestBody' or 'lazyRequestBody' can be used. +-- When this is the case, consider these strategies to mitigating potential DOS attacks: +-- +-- * Set a limit on the request body size you allow. +-- If certain endpoints need larger bodies, whitelist just those endpoints for the large size. +-- Be especially cautious about endpoints that don't require authentication, since these are easier to DOS. +-- You can accomplish this with @wai-extra@'s @requestSizeLimitMiddleware@ or Yesod's @maximumContentLength@. +-- * Consider rate limiting not just on total requests, but also on total bytes sent in. +-- * Consider using services that allow you to identify and blacklist attackers. +-- * Minimize the amount of time the request body stays in memory. +-- * If you need to share request bodies across middleware and your application, you can do so using Wai's 'vault'. +-- If you do this, remove the request body from the vault as soon as possible. +-- +-- Warning: Incremental processing will not always be sufficient to prevent a DOS attack. +-- For example, if an attacker sends you a JSON body with a 2MB long string inside, +-- even if you process the body incrementally, you'll still end up with a 2MB-sized 'Text'. +-- +-- To mitigate this, employ some of the countermeasures listed above, +-- and try to reject such payloads as early as possible in your codebase. -- | Get the request body as a lazy ByteString. However, do /not/ use any lazy -- I\/O, instead reading the entire body into memory strictly. -- +-- Note: Since this function consumes the request body, future calls to it will return the empty string. +-- -- Since 3.0.1 strictRequestBody :: Request -> IO L.ByteString strictRequestBody req = @@ -331,9 +385,18 @@ then return $ front LI.Empty else loop (front . LI.Chunk bs) +-- | Synonym for 'strictRequestBody'. +-- This function name is meant to signal the non-idempotent nature of 'strictRequestBody'. +-- +-- @since 3.2.3 +consumeRequestBodyStrict :: Request -> IO L.ByteString +consumeRequestBodyStrict = strictRequestBody + -- | Get the request body as a lazy ByteString. This uses lazy I\/O under the -- surface, and therefore all typical warnings regarding lazy I/O apply. -- +-- Note: Since this function consumes the request body, future calls to it will return the empty string. +-- -- Since 1.4.1 lazyRequestBody :: Request -> IO L.ByteString lazyRequestBody req = @@ -346,3 +409,10 @@ else do bss <- loop return $ LI.Chunk bs bss + +-- | Synonym for 'lazyRequestBody'. +-- This function name is meant to signal the non-idempotent nature of 'lazyRequestBody'. +-- +-- @since 3.2.3 +consumeRequestBodyLazy :: Request -> IO L.ByteString +consumeRequestBodyLazy = lazyRequestBody diff --git a/wai.cabal b/wai.cabal --- a/wai.cabal +++ b/wai.cabal @@ -1,5 +1,6 @@ +Cabal-Version: >=1.10 Name: wai -Version: 3.2.2.1 +Version: 3.2.3 Synopsis: Web Application Interface. Description: Provides a common protocol for communication between web applications and web servers. . @@ -11,7 +12,6 @@ Homepage: https://github.com/yesodweb/wai Category: Web Build-Type: Simple -Cabal-Version: >=1.8 Stability: Stable extra-source-files: README.md ChangeLog.md @@ -20,18 +20,19 @@ location: git://github.com/yesodweb/wai.git Library - Build-Depends: base >= 4.8 && < 5 + default-language: Haskell2010 + Build-Depends: base >= 4.10 && < 5 , bytestring >= 0.10.4 , network >= 2.2.1.5 , http-types >= 0.7 , text >= 0.7 - , transformers >= 0.0 , vault >= 0.3 && < 0.4 Exposed-modules: Network.Wai Network.Wai.Internal ghc-options: -Wall test-suite test + default-language: Haskell2010 hs-source-dirs: test main-is: Spec.hs type: exitcode-stdio-1.0