memcache 0.4.2.1 → 0.5.0.0
raw patch · 24 files changed
+8343/−1413 lines, 24 filesdep +cryptohash-md5dep +unixdep ~hashabledep ~timedep ~unliftio
Dependencies added: cryptohash-md5, unix
Dependency ranges changed: hashable, time, unliftio
Files
- CHANGELOG.md +455/−0
- Database/Memcache/Auth.hs +11/−11
- Database/Memcache/Client.hs +337/−85
- Database/Memcache/Cluster.hs +641/−96
- Database/Memcache/ElastiCacheClient.hs +248/−40
- Database/Memcache/Errors.hs +60/−3
- Database/Memcache/Ketama.hs +185/−0
- Database/Memcache/Server.hs +162/−50
- Database/Memcache/Socket.hs +171/−31
- Database/Memcache/Types.hs +5/−5
- Database/Memcache/Types/Authentication.hs +27/−2
- Database/Memcache/Types/Server.hs +70/−0
- Database/Memcache/Types/ServerSpec.hs +125/−23
- Database/Memcache/Util.hs +19/−1
- README.md +128/−12
- TODO.md +1/−1
- bench/Parser.hs +20/−4
- memcache.cabal +32/−9
- test-integration/Integration.hs +479/−60
- test/FileLimit.hs +30/−0
- test/KetamaVectors.hs +297/−0
- test/MockServer.hs +281/−46
- test/RoutingSpec.hs +437/−0
- test/Spec.hs +4122/−934
CHANGELOG.md view
@@ -1,3 +1,458 @@+# 0.5.0.0 - November 18th, 2026++This release changes how a cluster routes keys to its servers, so upgrading+moves keys to other servers, once; see "Upgrading from 0.4" in the README. It+also changes what `optsServerRetries` counts, the types of two lower-level+functions, and some return values and exceptions, so read the breaking+changes first. Two security fixes stop memcached from running the bytes of a+value being stored as commands, and memcached 1.6.10 or later is now+required; see Security.++## Security++Both fixes are also in 0.4.2.1.++- `set`, `add`, `replace`, `cas`, `append`, `prepend` and `setMany` reject a+ value larger than 1GiB, memcached's largest possible item size, with+ `OpError ErrValueTooLarge` before sending anything. A value of about 2GiB+ or more (longer than 2147483645 bytes) previously made memcached run the+ value's bytes as commands, so storing untrusted data could run commands on+ the server, and a value over 1GiB could time out, be sent again and mark+ the server dead. Such a value no longer reaches the server, so it no+ longer makes memcached delete the key's existing value, as memcached does+ whenever it rejects a store as too large; a value within 1GiB but over the+ server's own limit (`-I`, 1MiB by default) still does. Every operation+ that sends a value now evaluates it first, before its timeout starts: an+ exception it throws is no longer retried and no longer marks the server+ dead, and time spent computing it no longer counts towards+ `optsServerTimeout`. `setMany` evaluates every value before it sends+ anything, so a list of lazily computed values is all in memory at once,+ and one value over the limit stores none of the pairs. The request+ encoders in `Database.Memcache.Socket` still encode a value of any length;+ don't send one longer than 1GiB.+- memcached 1.6.10 or later is now required (previously documented as+ 1.6.0). memcached before 1.6.10 rejects the meta set command this library+ stores values with, without reading the value that follows it, and then+ runs the value's bytes as commands, so storing untrusted data could run+ commands on the server. Each new connection now asks the server for its+ version before it is used: until that check passes, the only requests sent+ are the authentication exchange, when credentials are set, and `version`.+ A server older than 1.6.10, or any other reply that is neither an error+ nor a readable version, is refused with `ProtocolError (UnknownOp ...)`,+ whose message quotes the reply; the connection is closed, and the refusal+ is not retried and doesn't mark the server dead. An error reply, such as+ `CLIENT_ERROR unauthenticated`, throws what it throws for any request, and+ so does a reply that can't be parsed, such as an empty line or an overlong+ one, which counts against the server. This covers connections over TCP and+ Unix sockets, and ElastiCache discovery, which now fails for a cluster+ running a memcached older than 1.6.10. Through a proxy, the check sees+ only the proxy's version, so every memcached behind it must be 1.6.10 or+ later too. Each new connection takes one more round trip, which counts+ towards `optsServerTimeout`. Connections made directly with+ `Database.Memcache.Socket.newConnection` are not checked; send+ `storeRequest`, `setManyRequest` and `modifyRequest` only to 1.6.10 or+ later.++## Breaking changes++- memcached 1.6.10 or later is now required (see Security): a new+ connection to an older server is refused, so every operation that needs+ one fails with `ProtocolError (UnknownOp ...)`.+- Clusters route keys with ketama consistent hashing, compatible with+ libmemcached (see New): `ketamaServerForKey` is the new default+ `optsGetServerForKey`. 0.4's router sent keys to no more than two servers+ at a time, and the server it chose for a key could change with the+ `hashable` version. With n servers, upgrading moves all but about one key+ in n to a different server, once: about half the keys with 2 servers, 70%+ with 3 and 88% with 8. A single server is unaffected. While old and new+ versions run side by side, as in a rolling deploy, they read and write a+ moved key on different servers; "Upgrading from 0.4", in the README and in+ `Database.Memcache.Cluster`, gives two ways to switch. One of them uses+ `legacyServerForKey`, 0.4's router, which `Database.Memcache.Cluster` now+ exports, deprecated, for one release; it will be removed in the next major+ version. A key whose server is marked failed now goes to the next live+ server on the ring, and other keys stay where they are.+- With keys spread over every server, `getMany` sends each server its keys+ in batches of up to 256, one round trip per batch and one server after+ another, so it can take longer than before with three or more servers. It+ still returns its results grouped by server, so their order changes too.+- Servers keep the order they are configured or discovered in, instead of+ being sorted by `sid`: `getServers` and `stats` list them in that order,+ `flush` goes through them in it, and `version` asks the first live one.+- `optsServerRetries` now counts retries, as its documentation said, rather+ than attempts: an operation is tried up to `optsServerRetries + 1` times,+ and once for 0 or a negative value. So a value of 1 or more that you set+ yourself makes one more attempt than in 0.4; set it one lower to keep the+ old number of attempts. `minBound`, which used to retry practically+ without end, now gives one attempt. The default drops from 2 to 1, so by+ default an operation is still tried twice.+- `increment`, `decrement`, `append`, `prepend`, `add`, `delete`, and `cas`+ and `replace` with a version other than 0 are no longer sent again once+ their request may have reached the server, so they can't take effect twice+ or report failure because their first request succeeded. Once the client+ has begun to send the request, a failure such as a timeout, a network+ error, a dropped connection, a reply that can't be read or a+ `SERVER_ERROR` whose text names no status is thrown as it is, whatever+ retries are left, and marks the server dead. A failure before sending,+ such as a refused connection or a timeout while waiting for a pooled+ connection, is still retried. Before sending one of these requests on a+ pooled connection, the client checks, without waiting, that nothing is+ left to read on it and that the server hasn't closed or reset it, as+ memcached closes idle connections when it restarts or after+ `-o idle_timeout`. If the check fails, the client retries the request on a+ new connection when retries are left; on Windows, the check only sees what+ has already been read, so it can't see a closed connection. Other+ operations are still retried after any failure that counts against the+ server, so the server may carry out their request twice.+- When a server's host name resolves to several addresses, the client now+ tries each in turn, not just the first (see Fixes). So when the first+ address refuses and a later one doesn't answer, an attempt throws+ `ClientError Timeout` once its `optsServerTimeout` runs out, instead of+ the first address's `IOException` at once. If the addresses are separate+ memcached instances, give each its own `ServerSpec`: while the first+ refuses connections, new connections go to the next one that accepts, and+ so to another cache.+- `append` and `prepend` return `Nothing` when a non-zero version doesn't+ match, as `cas` and `replace` do, instead of throwing+ `OpError ErrKeyExists`.+- `flush` stops at the first server whose reply it rejects (see Fixes),+ leaving the servers after it unflushed, where 0.4 checked such a reply+ only after sending `flush_all` to every server.+- memcached's capitalised out-of-memory errors are thrown as+ `OpError ErrOutOfMemory`, like the lowercase ones, instead of+ `ProtocolError ServerError` (see Fixes), so the request is no longer sent+ again, and they don't count towards marking the server dead. Code that+ matched `ServerError` for them should match `ErrOutOfMemory` instead.+- Numbers in replies (value lengths, CAS versions, flags and counter values)+ must be plain decimal digits that fit their type, as memcached writes them+ (see Fixes), where 0.4 also read forms such as `(5)` and `0x10`, and+ wrapped numbers around to fit. So from a server or proxy that writes them+ otherwise, a value length now throws `BadLength`, and a CAS version, flags+ or counter value reads as 0. A `VA` reply without a value length, which+ used to read as a hit with an empty value, throws `BadLength` too.+- `displayException` for a `MemcacheError` gives a one-line message, such as+ `memcached operation failed: key not found`, where it used to give the+ same text as `show`, such as `OpError ErrKeyNotFound`; GHC 9.10 and later+ also print it for an uncaught exception. Any detail, such as text from the+ server, is cut to its first 200 characters, with backslashes and+ characters outside printable ASCII escaped. `show` is unchanged, so code+ that parses `displayException`'s text should match on the constructors, or+ use `show`, instead.+- `ElastiCacheClient.newClient` throws `ClientError Timeout` unless its+ first discovery, connecting included, finishes within `optsServerTimeout`.+ It had no time limit, so with the default of 750ms, a slow DNS lookup or+ connection can now make it fail. An `ERROR`, `CLIENT_ERROR` or+ `SERVER_ERROR` reply to the request for the cluster configuration, such as+ the `ERROR` that plain memcached gives, throws a `StringException` from+ `UnliftIO.Exception` quoting the line, escaped and cut to 200 characters,+ where `newClient` used to block forever, or throw+ `ProtocolError UnexpectedEOF` if the endpoint then closed the connection.+ Discovery now calls `optsServerSpecsToServers` with one node at a time.+- `stats` throws `BadLength` for a server's reply of more than 262,144+ entries, or of more than 16MiB counting each line as it was sent, such as+ the reply to `stats conns` on a server with more than about 65,500 open+ connections. `stats` retries it, as it does other server failures, and it+ counts towards marking the server dead. `recvRawUntil` throws `BadLength`+ once it has read more than 1MiB without its delimiter, so ElastiCache+ discovery does too for a version line or configuration reply that long. No+ option raises these limits.+- `parseServerSpec`, and so `parseConfigurationEndpoint`, percent-decodes+ the username and password of a `memcached://` URI to the bytes they+ encode, so `memcached://user:p%40ss@cache.example` has the password+ `p@ss`, and returns `Left` if the decoded credentials fail+ `validateCredentials`. `validateCredentials`, which authentication also+ uses, now allows `:` in passwords, as memcached's auth file splits each+ line at its first `:`, and rejects NUL in usernames and passwords, so+ connecting with a NUL in either now throws+ `ClientError InvalidAuthentication` before anything is sent. An IPv6+ address in brackets, as in `memcached://[::1]:11211`, now gives the host+ `::1`, without the brackets, and the `%25` that introduces a zone ID is+ decoded, so `[fe80::1%25eth0]` gives `fe80::1%eth0`. A path after an empty+ host, other than slashes alone, as in `memcached:///run/memcached.sock`,+ is now a Unix socket (see New), where 0.4 ignored it and gave `127.0.0.1`;+ with a port, or with `%00` or a raw `@` in the path, such a URI now+ returns `Left`.+- `show` for an `Authentication`, and so for a `ServerSpec`, prints the+ password as `<redacted>`, and the error that `parseServerSpec` and+ `parseConfigurationEndpoint` give for an invalid URI no longer quotes it,+ so that passwords stay out of logs.+- `getManyRequest` in `Database.Memcache.Socket`, and the request builders+ that `keyedBatchOp` in `Database.Memcache.Cluster` takes, have a new `Int`+ argument: an opaque, which the request must send unchanged, in decimal, as+ its `O` flag, so that its reply can be matched to it (see Fixes).+ `getManyRequest` sends it in place of the `k` flag, so its replies no+ longer echo the key.+- `Database.Memcache.Client` exports more names (see New), such as `Key`,+ `Version`, `Server` and `defaultOptions`, and `Database.Memcache.Cluster`,+ `Database.Memcache.Server`, `Database.Memcache.Socket` and+ `Database.Memcache.Types` export new names too. A new name can clash with+ another in a module that imports one of these modules unqualified without+ listing the names it imports, even one that hides some with `hiding`.++## New++- Unix sockets, contributed by drlkf: a `ServerSpec` whose host begins with+ `/` connects to the Unix socket at that path, for a memcached on the same+ machine. `parseServerSpec` reads one from a URI with a path, other than+ slashes alone, after an empty host, such as+ `memcached:///run/memcached.sock`, or+ `memcached://user:secret@/run/memcached.sock` with credentials. The path+ is percent-decoded to bytes. A port or `%00` is an error, and so is an `@`+ not written `%40`, so that a URI with one slash too many, such as+ `memcached:///user:secret@cache.example`, isn't taken for a socket path+ holding the password. The client doesn't use a socket's port to connect,+ but the port still names the server when keys are routed: `"11211"`, which+ a URI gives, names the socket by its path alone, as libmemcached's+ `memcached_server_add` does, and `"0"` names it `path:0`, as+ libmemcached's `memcached_server_add_unix_socket` and pylibmc do.+ ElastiCache discovery fails, rather than connect to a socket on this+ machine, if the endpoint lists a node whose host is a path.+- Keys go to the same server as with other clients that use libmemcached's+ weighted ketama distribution, given the same host strings and ports in the+ same order and equal weights: libmemcached with+ `MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED`, php-memcached with+ `OPT_LIBKETAMA_COMPATIBLE`, pylibmc with the `ketama_weighted` behavior,+ and twemproxy with `hash: md5` and `distribution: ketama` and its servers+ listed without node names. In the rare case that two servers have a point+ with the same value, libmemcached before 1.1, which php-memcached and+ pylibmc may be built with, and twemproxy may choose differently for the+ keys just below it. The new module `Database.Memcache.Ketama` (`Ring`,+ `ring`, `serverName`, `keyHash`, `owner`, `owners` and `ringPoints`) works+ out which server a key belongs to, and builds rings with weights, which+ clusters don't use. `Database.Memcache.Cluster` exports the default router+ as `ketamaServerForKey`, and `isAlive` for a custom router that skips+ servers marked failed, and `Database.Memcache.Server` exports `serverHost`+ and `serverPort`.+- `withClient` creates a client, runs an action with it, and closes the+ client's connections with `quit` when the action returns or throws.+- `setMany` and `deleteMany` store and delete many keys, pipelining the+ requests to each server as `getMany` does. `setMany` returns the new+ version of each key, grouped by server; `deleteMany` returns once each key+ has been deleted or found missing, without saying which. Their request+ encoders, `setManyRequest` and `deleteManyRequest`, are in+ `Database.Memcache.Socket`.+- `defaultOptions`, `defaultServerOptions` and `defaultServerSpec` name the+ values that `def` gives. They are exported from+ `Database.Memcache.Cluster`, `Database.Memcache.Server` and+ `Database.Memcache.Types` respectively, and all three from+ `Database.Memcache.Client`.+- `Database.Memcache.Client` also exports `parseServerSpec`, `Retries`,+ `ServerOptions(..)`, the abstract `Server` type, and `Key`, `Value`,+ `Initial`, `Delta`, `Expiration`, `Flags` and `Version`, the types its own+ functions use.+- For lower-level use: `closeAll` in `Database.Memcache.Server` closes every+ connection to a server, as `quit` does for each server of a client;+ `keyedOpExpecting`, `anyOpExpecting` and `allOpExpecting` in+ `Database.Memcache.Cluster` reject a reply whose code is none of those+ given, closing its connection, and `keyedOpExpectingOnce` also doesn't+ send its request again once it may have reached the server; and+ `checkIdle` in `Database.Memcache.Socket` checks, without waiting, that+ nothing is left to read on a connection and that the server hasn't closed+ it (on Windows, it only checks what has already been read).++## Fixes++- Fix `getMany` dropping keys, or returning one key's value as another's,+ when it couldn't match the key that memcached echoed in a reply: a key+ containing byte 0xa0, such as UTF-8 `"voilà"`; a key with a space or+ control byte once an increment has created it or made its value longer, or+ an append or prepend has rebuilt it; and a plain key that another client+ stored with the `b` flag. When such a key held a line break, the echo also+ ended the reply line inside the key, so `getMany` could return part of the+ key as another key's value, fail with `BadLength` and mark a healthy+ server dead, or end the batch early and leave the rest of its replies on+ the pooled connection, to be read as the replies to later requests.+ `getMany`'s requests no longer ask memcached to echo the key, and batch+ replies are now matched to their requests by memcached's opaque (`O`) flag+ instead. A reply that matches no request in its batch, which used to be+ dropped, or that comes out of order, now throws `UnexpectedResponse` and+ closes the connection; like any server failure, the batch is then sent+ again on a new connection if retries are left, and the failure counts+ towards marking the server dead.+- Reply lines are split into tokens on spaces alone, as memcached writes+ them, so a token that `recvResponse` returns keeps any other whitespace+ byte, such as a tab or 0xa0, and so do `stats` values.+- In single-key operations, `version`, `flush` and authentication, a reply+ with an unknown code, or one the request can't get, as when an earlier+ request's reply is still on the connection, now closes the connection+ instead of returning it to the pool out of step, where later requests+ would read each other's replies. It throws `UnknownStatus` or+ `UnexpectedResponse`, where some such replies used to be taken as an+ answer: as success by a store, `increment`, `decrement`, `delete`, `flush`+ or authentication, and as a miss by `get`, `gat` and `touch` for an `NF`,+ and by `cas`, `replace`, `append`, `prepend`, `increment`, `decrement` and+ `delete` for an `EN`. `flush` now stops at the first server whose reply it+ rejects, leaving the servers after it unflushed, where it used to check+ such a reply only after sending `flush_all` to every server (see Breaking+ changes). In `getMany`, `setMany` and `deleteMany`, a reply with an+ unknown code throws `UnknownStatus` and closes the connection; the batch+ is then sent again on a new connection if retries are left, and the+ failure counts towards marking the server dead. `getMany` used to skip+ such a reply, or throw for it without sending the batch again or counting+ the failure.+- memcached's capitalised out-of-memory errors,+ `Out of memory allocating new item` (from `increment` and `decrement`) and+ `Out of memory during read`, are reported as `OpError ErrOutOfMemory`,+ like the lowercase ones (see Breaking changes). They were thrown as+ `ProtocolError ServerError`, so the request was sent again and a full but+ healthy server was marked dead.+- An attempt that runs out of `optsServerTimeout` while still waiting for a+ free pooled connection is no longer blamed on the server: it is retried as+ before, but if it was the last attempt, the operation throws+ `ClientError Timeout` without marking the server dead. Running out of time+ later, while connecting, authenticating or using the connection, still+ counts against the server. A connection is now opened, and authenticated,+ once its request has a place in the pool, so a failed connect no longer+ leaves requests queued for the pool blocked until their own timeout, as it+ did with resource-pool 0.3.0.0 to 0.5.0.0. An `allOp'` operation must make+ its requests through the `Server` it is given for its timeouts to count+ against the server.+- A failed attempt that counts against the server now closes the server's+ idle pooled connections before the operation is retried or the server is+ marked dead, as a timeout already did. So with `soNumResources` of 2 or+ more, a memcached restart or memcached's `idle_timeout` no longer sends+ each retry to another closed connection, failing the operation and marking+ a healthy server dead. With resource-pool 0.2, an asynchronous exception,+ such as a `cancel`, that arrived while those connections were being closed+ could be lost, so that the request carried on as if it hadn't arrived, or+ cut the closing short, leaving some of them open; with later versions, it+ could still cut the closing short between two of the pool's stripes.+ `close` in `Database.Memcache.Server`, which closes them, now holds such+ an exception back until every idle connection is closed, as `closeAll`+ does, so they are all closed first, and the exception then ends the+ request.+- Retries no longer run with asynchronous exceptions masked. Like the first+ attempt, each runs in the caller's masking state, so `optsServerTimeout`+ and the caller's own asynchronous exceptions, such as a `cancel`, can+ interrupt a retry that keeps working without blocking.+- When a server's host name resolves to several addresses, the client tries+ each in turn, within the attempt's `optsServerTimeout`, and keeps the+ first connection that succeeds until it is closed; it used to try only the+ first address. If every address fails, the error is still the first+ address's. An address that doesn't answer uses up the attempt's time, so+ when the first address refuses and a later one doesn't answer, an attempt+ now throws `ClientError Timeout` where it used to throw the first+ address's error at once (see Breaking changes). A failed authentication,+ or a server version the client refuses, is not retried against another+ address. If the addresses are separate memcached instances, then while the+ first refuses connections, new connections go to the next one that+ accepts, and so to another cache, and stay with it until they are closed;+ give each instance its own `ServerSpec`.+- `quit` no longer sends memcached's `quit` command, so it never connects to+ a server, and it never throws, even when every server is marked dead. It+ closes the client's connections to every server, including those marked+ dead: idle ones at once, and each one in use as soon as its request+ finishes. An asynchronous exception, such as a `cancel`, that arrives+ while it closes them is held back until every server's idle connections+ are closed. A request that starts later, or is still waiting for a+ connection, or opening one, when `quit` runs, opens a new one, which stays+ open as usual.+- `ElastiCacheClient.newClient` no longer stops refreshing the node list+ after the first refresh that fails. A refresh that throws, gets an error+ reply or runs out of time keeps the current servers, and discovery tries+ again at the next interval. Its error is dropped, where the first refresh+ that threw used to end discovery and print its exception on stderr. Each+ refresh has the first discovery's time limit, `optsServerTimeout`, and+ connects to the endpoint afresh, looking its name up again, as a+ connection kept from an earlier refresh could lead to a node that+ ElastiCache has since replaced. A refresh also keeps the `Server`, and so+ the pooled connections and failure state, of each node still listed, where+ it used to replace every server and leave the old connections open. It+ closes the connections of a node no longer listed, or listed with another+ IP address, as when ElastiCache replaces it: idle ones at once, and one in+ use as soon as its request finishes, so that requests connect again,+ looking the node's name up afresh.+- ElastiCache discovery checks the endpoint's replies more strictly. A+ configuration that lists more than 1,000 nodes, the same host and port+ twice, or a node that isn't a plain host and port now fails discovery, as+ a reply that can't be parsed does: `ElastiCacheClient.newClient` throws a+ `StringException`, and a refresh keeps the current servers. A node isn't a+ plain host and port if its host or port is empty, if its host is an IPv6+ address in brackets, or if its host or port has a character, such as `/`,+ `?` or `@`, that would change the host or port or add credentials, as+ `h/x` on port 22122 used to give `h` on port 11211; an empty host used to+ give `127.0.0.1`, and an empty port failed only at the first request. A+ version of more than 64 bytes in the endpoint's version reply fails+ discovery too, where parsing it took time that grew faster than the square+ of its length. Discovery's errors escape the text they quote from the+ endpoint and cut it to 200 characters, as `displayException` does for a+ `MemcacheError`. The refresh thread runs with asynchronous exceptions+ unmasked, even when `newClient` is called with them masked, as in the+ acquire action of a `bracket`, so that each refresh's time limit can+ interrupt it.+- In `getMany`, `setMany` and `deleteMany`, a bare `EN` (a miss reply+ without an opaque) now counts as the reply to the next request. One after+ the reply to the batch's last request throws `UnexpectedResponse`, and so+ does a reply to a request that a bare `EN` was counted for, or to an+ earlier one, so a server that sends bare `EN`s without end can no longer+ keep a batch reading.+- `stats` now closes each connection it reads a reply from, instead of+ returning it to the pool, so a later request to each server opens a new+ one in its place. A key holding a line break, which `stats cachedump` can+ list as it is stored, could end the reply early as `stats` read it, and+ the rest of the reply was then read as the reply to a later request on the+ same connection. The bytes after such a line break are still read as lines+ of their own, so they can add entries or end the reply early, but `stats`+ no longer sends its request again or marks the server dead when they make+ an empty line, or a `SERVER_ERROR` line naming no status: after a reply's+ first line, it throws the error of any empty or error line once every+ server has answered.+- Numbers in replies (value lengths, CAS versions, flags and counter values)+ are read only as plain decimal digits that fit their type, as memcached+ writes them (see Breaking changes), where they used to be read leniently,+ as with `(5)` or `0x10`, or wrapped around to fit. A value length in any+ other form, or a `VA` reply without one, throws `BadLength`; such a `VA`+ reply used to read as a hit with an empty value. A CAS version, flags or+ counter value in any other form reads as 0, as a missing one does, so such+ a CAS skips the check when passed back to an operation that takes a+ `Version`.++## Documentation and development++- The README and `Database.Memcache.Client` explain why to create one client+ and share it, and say more about timeouts, retries and failover, including+ which requests are never sent twice. `Database.Memcache.Cluster` documents+ routing, failover and upgrading from 0.4, which the README sums up, and+ the README has a section on Unix sockets.+- The README and the docs say which error replies count against a server: a+ `SERVER_ERROR` whose text names no status, such as memcached's proxy sends+ when a backend fails, counts against the server as a timeout does, while+ `ERROR`, `CLIENT_ERROR` and a `SERVER_ERROR` that names a status are+ thrown at once without counting against it. So behind memcached's proxy, a+ request for a key on a failed backend marks the whole proxy dead: at once+ if it mustn't be sent twice, and otherwise once its retries get the same+ reply. The docs of `Timeout` and `NoServersReady` now say when each is+ thrown.+- The docs say what each resource-pool version does with `soNumResources`+ and `soNumStripes`. With 0.3, which the bounds allow, a pool has a stripe+ for each capability, whatever `soNumStripes` is, so a pool with the+ default options holds one connection for each capability, and a thread+ takes connections only from its own capability's stripe. With 0.4 or+ later, `soNumResources` is split between the stripes, and creating a+ client throws an `ErrorCall` if it is smaller than `soNumStripes`.+- New dependency: `cryptohash-md5`, for the MD5 hashes of the ketama ring.+ The `spec` test suite now also depends on `directory`, `hashable`, `time`+ and `unix` (except on Windows), and `integration` on `unliftio`.+- Expand the test suite. Routing is checked against the servers that+ libmemcached-awesome 1.1.4 chooses for the same keys, including its own+ test fixture, and `legacyServerForKey` against a copy of 0.4's router.+ Mock tests cover the new operations, replies out of step, retries,+ failover and pool timeouts, closing connections, connecting to each+ resolved address, Unix sockets, and ElastiCache discovery and refreshes.+ The integration tests also run against memcached restarted, paused with+ `SIGSTOP`, listening on a Unix socket and out of memory, and test+ ElastiCache discovery against plain memcached.+- The `integration` test suite needs memcached 1.6.13 or later. With 1.6.10+ to 1.6.12, which the library accepts, two of its tests fail, as those+ servers answer an `append` or `prepend` with the version the request+ carried, or 0, rather than the new one. The documentation of `append` and+ `prepend` says so, as passing that 0 back as a version skips the check.+- The `parser` benchmark also times hashing a key, building a ring, finding+ a key's server on it and routing a key through a cluster.+- CI runs on the `ubuntu-26.04` runner image, named in the workflows instead+ of `ubuntu-latest`, installs the gold linker that older GHCs need there,+ keeps its caches per OS release, and uses `freckle/stack-action` 5.7.33.+ # 0.4.2.1 - November 18th, 2026 No API changes from 0.4.2.0.
Database/Memcache/Auth.hs view
@@ -22,12 +22,17 @@ import Database.Memcache.Errors import Database.Memcache.Socket import Database.Memcache.Types+import Database.Memcache.Types.Authentication (validateCredentials) import Control.Monad (unless) import qualified Data.ByteString.Char8 as B8 import UnliftIO.Exception (catch, throwIO) --- | Authenticate a connection using text-protocol credentials.+-- | Authenticate a connection using text-protocol credentials. Throws+-- 'InvalidAuthentication' when the credentials can't be sent safely,+-- @'OpError' 'TextAuthFail'@ when the server rejects them, and another+-- 'MemcacheError', such as 'UnexpectedResponse', for any other reply but+-- @STORED@. authenticate :: Connection -> Authentication -> IO () authenticate _ NoAuth = return () authenticate socket (Auth user pass) = do@@ -46,17 +51,12 @@ response <- recvResponse socket result <- responseStatus response `catch` handleAuthFailure case result of- NoError -> return ()+ NoError | responseCode response == "STORED" -> return ()+ -- Any other reply could leave the connection out of step, so it must not+ -- reach the pool.+ NoError -> throwIO $ ProtocolError UnexpectedResponse+ { protocolError = "Expected AUTH, got: " <> show (responseCode response) } status -> throwIO $ OpError status where handleAuthFailure (ProtocolError BadCommand{}) = throwIO $ OpError TextAuthFail handleAuthFailure err = throwIO (err :: MemcacheError)---- | Validate that credentials contain no delimiters used by the auth command.-validateCredentials :: Username -> Password -> Either String ()-validateCredentials user pass- | B8.any forbidden user = Left "username contains a forbidden delimiter"- | B8.any forbidden pass = Left "password contains a forbidden delimiter"- | otherwise = Right ()- where- forbidden c = c == ' ' || c == '\t' || c == '\r' || c == '\n' || c == ':'
Database/Memcache/Client.hs view
@@ -11,8 +11,9 @@ A Memcached client using the Meta (text) protocol. Memcached is an in-memory key-value store typically used as a distributed and shared cache. Clients can-connect to a single Memcached server or a cluster of servers; clusters use-consistent hashing to route requests to the appropriate server.+connect to a single Memcached server or a cluster of servers; clusters route+each key to a server by ketama consistent hashing, compatible with+libmemcached (see "Database.Memcache.Cluster"). This requires memcached 1.6.10 or newer. Older servers reject the meta set command this client stores values with, without reading the value that@@ -36,16 +37,51 @@ We support the following logic for handling failure in operations: -* __Timeouts__: we timeout any operation that takes too long and consider it- failed.+* __Timeouts__: we time out any attempt at an operation that takes longer+ than 'optsServerTimeout', including time spent waiting for a+ free connection to the server, and consider it failed. -* __Retry__: on operation failure (timeout, network error) we close the- connection and retry the operation, doing this up to a- configurable maximum.+* __Retry__: on operation failure (a timeout, a network error, or a+ @SERVER_ERROR@ reply whose text names no status, such as+ memcached's proxy sends when a backend fails) we close the+ connection, if any, and retry the operation, doing this up to+ 'optsServerRetries' times. Unless the attempt only ran out of time+ waiting for a free connection, we also close the server's idle+ connections, as what made it fail, such as a restart of the+ server, may have broken them too. A reply to an operation's+ request that shows the connection is out of step also closes+ it. Only 'getMany', 'setMany' and 'deleteMany' retry then, and+ close the idle connections too; other operations throw at once,+ as the server may have carried out the request. -* __Failover__: when an operation against a server in a cluster fails all- retries, we mark that server as dead and use the remaining- servers in the cluster to handle all operations. After a+ A request that mustn't be sent twice is retried only after a+ failure before any of it was sent, such as a refused+ connection or a timeout while waiting for a free connection.+ That is 'increment', 'decrement', 'append', 'prepend', 'add',+ 'delete', and 'cas' and 'replace' with a version other than+ @0@. Once such a request may have reached the server, a+ failure is thrown, and the server is marked dead as if every+ attempt had failed: the server may have carried out the+ request, and a retry could apply it twice, or report failure+ because the first one succeeded. Before sending one on a+ connection from the pool, we check that the server hasn't+ closed or reset it, as memcached closes idle connections when+ it restarts or after @-o idle_timeout@, and that no bytes from+ the server are already waiting on it. If the check fails, the+ request is retried on a new connection if retries are left. With+ 'optsServerRetries' at @0@, that failure is thrown instead, and+ the server is marked dead. Other operations are retried after+ any server failure, so the server may carry out their request+ twice (see 'optsServerRetries'). Set 'optsServerRetries' to @0@+ for no retries.++* __Failover__: when an operation against a server in a cluster fails on every+ attempt, we mark that server as dead and use the remaining+ servers in the cluster to handle all operations: with the+ default router, its keys go to the next live server on the+ hash ring, and other keys stay where they are. That doesn't+ happen when the last attempt only ran out of time waiting for+ a free connection, as it never reached the server. After a configurable period of time has passed, we consider the server alive again and try to use it. This can lead to consistency issues (stale data), but is usually fine for caching purposes@@ -98,23 +134,33 @@ -} module Database.Memcache.Client ( -- * Client creation- newClient, Client, ServerSpec(..), Options(..),- Authentication(..), Username, Password, def,- quit,+ newClient, Client, ServerSpec(..), defaultServerSpec, parseServerSpec,+ Options(..), defaultOptions, Authentication(..), Username, Password,+ -- | 'def' from "Data.Default" gives the same values as+ -- 'defaultServerSpec', 'defaultOptions' and 'defaultServerOptions',+ -- picked by the type it is used at.+ def,+ quit, withClient, + -- ** Configuration types+ Retries, ServerOptions(..), defaultServerOptions, Server,++ -- * Fields & Values+ Key, Value, Initial, Delta, Expiration, Flags, Version,+ -- * Operations -- ** Get operations get, getMany, gat, touch, -- ** Set operations- set, cas, add, replace,+ set, setMany, cas, add, replace, -- ** Modify operations increment, decrement, append, prepend, -- ** Delete operations- delete, flush,+ delete, deleteMany, flush, -- ** Information operations StatResults, stats, version,@@ -129,43 +175,90 @@ import Database.Memcache.Socket import Database.Memcache.Types -import Control.Monad (foldM, forM_, void, when)+import Control.Exception (uninterruptibleMask_)+import Control.Monad (foldM, forM_, when) import Data.ByteString (ByteString)-import qualified Data.ByteString as B (any, empty, length, null)-import qualified Data.ByteString.Char8 as C (unwords)+import qualified Data.ByteString as B (any, empty, length, null, stripSuffix)+import qualified Data.ByteString.Char8 as C (split, unwords) import Data.Default import Data.Maybe (fromMaybe, listToMaybe) import Data.Word-import UnliftIO.Exception (SomeException, handle, throwIO)+import UnliftIO.Exception (Exception, bracket, handle, throwIO) -- | A Memcached cluster client. type Client = Cluster -- | Key/value pairs returned by 'stats'. type StatResults = [(ByteString, ByteString)] --- | Create a client for the supplied servers. An empty list uses the default--- local server specification.+-- | Create a client for the supplied servers. An empty list uses+-- 'defaultServerSpec'. --+-- Create one client and share it for the life of your program. It is safe to+-- use from many threads, and it keeps a pool of connections to each server,+-- opened on first use and reused across requests. A client created for each+-- request instead opens (and, with credentials, authenticates) a new+-- connection to every server it uses, every time. Each pool holds as many+-- connections as 'soNumResources' and 'soNumStripes' allow, one by default+-- (with resource-pool 0.3, one for each capability), so concurrent requests to+-- a server beyond that wait for a free connection. The wait counts towards+-- each attempt's 'optsServerTimeout'. If the last attempt runs out of time+-- while still waiting, the server isn't marked down; running out of time+-- afterwards is a server failure, however long the wait was.+-- -- Nothing is sent until an operation needs a connection to a server. Each new -- connection then checks the server's version, and refuses a server older than -- memcached 1.6.10: until the check passes, the only requests sent are the -- authentication exchange, when credentials are set, and @version@. newClient :: [ServerSpec] -> Options -> IO Client-newClient scs = newCluster $ if null scs then [def] else scs+newClient scs = newCluster $ if null scs then [defaultServerSpec] else scs --- | Close all client connections, ignoring individual shutdown failures.-quit :: Cluster -> IO ()-quit c = void $ allOp' c $ \server -> handle ignore $ do- withSocket server $ \socket -> sendRaw socket quitRequest- close server- where ignore :: SomeException -> IO ()- ignore = const (pure ())+-- | Close the client's connections to every server, including servers marked+-- dead. Idle connections are closed straight away. A connection that another+-- thread is using at the time is closed as soon as the request using it+-- finishes, instead of going back to the pool; 'quit' doesn't wait for those+-- requests or interrupt them. Requests that start after 'quit', including+-- later requests of an operation that was already running, never use those+-- connections: they open new ones, which stay open as usual. So does a request+-- that is still waiting for a connection, or opening one, when 'quit' runs.+--+-- This doesn't send Memcached's @quit@ command: closing the connection has the+-- same effect, while sending the command would mean first connecting to any+-- server the client has no connection to. So 'quit' never connects to a+-- server, returns promptly and doesn't throw, even when a server is down.+-- Like 'closeAll', it can't be cut short by an asynchronous exception: one+-- that arrives while it closes connections is delivered once every server's+-- idle connections are closed.+quit :: Client -> IO ()+quit c = getServers c >>= uninterruptibleMask_ . mapM_ closeAll +-- | Create a client with 'newClient', run an action with it, and close the+-- client's connections with 'quit' when the action returns or throws.+--+-- Use it to scope one long-lived client to your program, a worker or a test,+-- and share that client between threads. Don't wrap individual requests in+-- it: every new client opens (and, with credentials, authenticates) its own+-- connection to each server it uses, so a client per request pays that cost+-- on every request.+--+-- 'newClient' doesn't connect to any server (connections are opened on first+-- use), so if it throws there is nothing to close. 'quit' doesn't throw, so+-- 'withClient' returns the action's result, or rethrows its exception,+-- unchanged.+--+-- Don't use the client after 'withClient' returns, or from threads that+-- outlive the action. A request such a thread is making when the action ends+-- completes. If it already has a connection, that connection is then closed.+-- If it is still waiting for a connection, or opening one, it opens a new one,+-- which, like those of the requests the thread starts afterwards, stays open+-- until it has been idle for 'soKeepAlive'.+withClient :: [ServerSpec] -> Options -> (Client -> IO a) -> IO a+withClient specs opts = bracket (newClient specs opts) quit+ -- | Retrieve the value for the given key from Memcached, along with its -- flags and CAS version. Returns 'Nothing' on a miss.-get :: Cluster -> Key -> IO (Maybe (Value, Flags, Version))+get :: Client -> Key -> IO (Maybe (Value, Flags, Version)) get c k = do- response <- keyedOp c k getRequest+ response <- keyedOpExpecting c "GET" getReplies k getRequest status <- responseStatus response case (responseCode response, status) of ("VA", NoError) -> return $ Just (valueResult response)@@ -173,7 +266,7 @@ _ -> failResponse response "GET" -- | Fetch many keys. Missing keys are omitted from the result.-getMany :: Cluster -> [Key] -> IO [(Key, (Value, Flags, Version))]+getMany :: Client -> [Key] -> IO [(Key, (Value, Flags, Version))] getMany c keys = do replies <- keyedBatchOp c [(key, getManyRequest) | key <- keys] reverse <$> foldM collect [] replies@@ -189,9 +282,9 @@ -- | Get-and-touch: Retrieve the value for the given key from Memcached, and -- also update the stored key-value pairs expiration time at the server. Use an -- expiration value of @0@ to store forever.-gat :: Cluster -> Key -> Expiration -> IO (Maybe (Value, Flags, Version))+gat :: Client -> Key -> Expiration -> IO (Maybe (Value, Flags, Version)) gat c k e = do- response <- keyedOp c k (`gatRequest` e)+ response <- keyedOpExpecting c "GAT" getReplies k (`gatRequest` e) status <- responseStatus response case (responseCode response, status) of ("VA", NoError) -> return $ Just (valueResult response)@@ -201,9 +294,9 @@ -- | Update the expiration time of a stored key-value pair, returning its -- new CAS version or 'Nothing'. -- Use an expiration value of @0@ to store forever.-touch :: Cluster -> Key -> Expiration -> IO (Maybe Version)+touch :: Client -> Key -> Expiration -> IO (Maybe Version) touch c k e = do- response <- keyedOp c k (`touchRequest` e)+ response <- keyedOpExpecting c "TOUCH" touchReplies k (`touchRequest` e) status <- responseStatus response case status of NoError | responseCode response == "HD" -> return $ Just (responseCas response)@@ -213,13 +306,40 @@ -- | Store a new (or overwrite existing) key-value pair, returning its @Version@ -- identifier. Use an expiration value of @0@ to store forever. ----- A value larger than 1 GiB, the largest item memcached can store, is--- rejected with 'OpError' 'ErrValueTooLarge' before anything is sent, as it--- is by every operation that sends a value. memcached itself rejects a value--- too large for its item size limit (its @-I@ option) with the same status.-set :: Cluster -> Key -> Value -> Flags -> Expiration -> IO Version+-- A value larger than 1 GiB, the largest item any memcached server can store,+-- throws 'OpError' 'ErrValueTooLarge' before anything is sent, as it does in+-- every operation that stores a value.+set :: Client -> Key -> Value -> Flags -> Expiration -> IO Version set c k v f e = store c k v f e 0 Set "SET" +-- | Store many key-value pairs, as 'set' does, returning the new @Version@ of+-- each. Like 'getMany', the requests to each server are pipelined, and the+-- results are grouped by server rather than in the order given; within a+-- server they keep the input order. A key given more than once is stored+-- once per occurrence, in order, so its last entry in the result is the+-- version now stored. Use an expiration value of @0@ to store forever.+--+-- Every key, and the length of every value, is checked before anything is+-- sent, so an invalid key or a value larger than 1 GiB stores nothing. Every+-- value is therefore evaluated before the first request is sent. Each+-- server's pairs are sent in batches of up to 256 that close early once their+-- requests reach 1 MiB (a larger request goes alone), and each batch, like a+-- 'set', must finish within 'optsServerTimeout'. When a server fails+-- part-way through, the requests it was handling are sent again, storing the+-- same values; the versions returned are from the retry. If this throws,+-- some of the pairs may have been stored.+setMany :: Client -> [(Key, Value, Flags, Expiration)] -> IO [(Key, Version)]+setMany c items = do+ mapM_ (\(_, v, _, _) -> checkValue v) items+ replies <- keyedBatchOp c [(k, \wire -> setManyRequest wire v f e) | (k, v, f, e) <- items]+ mapM stored replies+ where+ stored (key, response) = do+ status <- responseStatus response+ case status of+ NoError -> return (key, responseCas response)+ _ -> failResponse response "SETMANY"+ -- | Store a key-value pair, but only if the version specified by the client -- matches the @Version@ of the key-value pair at the server. The version -- identifier of the stored key-value pair is returned, or if the key doesn't@@ -228,10 +348,10 @@ -- -- A version of @0@ skips the check, so the value is stored unconditionally, as -- with 'set'.-cas :: Cluster -> Key -> Value -> Flags -> Expiration -> Version -> IO (Maybe Version)+cas :: Client -> Key -> Value -> Flags -> Expiration -> Version -> IO (Maybe Version) cas c k v f e ver = do checkValue v- response <- keyedOp c k (\wire -> storeRequest Set wire v f e ver)+ response <- resendUnlessVersioned ver c "SET" storeReplies k (\wire -> storeRequest Set wire v f e ver) status <- responseStatus response case status of NoError -> return $ Just (responseCas response)@@ -242,10 +362,10 @@ -- | Store a new key-value pair, returning its @Version@ identifier. If the -- key-value pair already exists, then fail (return 'Nothing'). Use an -- expiration value of @0@ to store forever.-add :: Cluster -> Key -> Value -> Flags -> Expiration -> IO (Maybe Version)+add :: Client -> Key -> Value -> Flags -> Expiration -> IO (Maybe Version) add c k v f e = do checkValue v- response <- keyedOp c k (\wire -> storeRequest Add wire v f e 0)+ response <- keyedOpExpectingOnce c "ADD" storeReplies k (\wire -> storeRequest Add wire v f e 0) status <- responseStatus response case status of NoError -> return $ Just (responseCas response)@@ -258,10 +378,10 @@ -- -- A non-zero version must match the @Version@ of the key-value pair at the -- server, otherwise 'Nothing' is returned. A version of @0@ skips the check.-replace :: Cluster -> Key -> Value -> Flags -> Expiration -> Version -> IO (Maybe Version)+replace :: Client -> Key -> Value -> Flags -> Expiration -> Version -> IO (Maybe Version) replace c k v f e ver = do checkValue v- response <- keyedOp c k (\wire -> storeRequest Replace wire v f e ver)+ response <- resendUnlessVersioned ver c "REPLACE" storeReplies k (\wire -> storeRequest Replace wire v f e ver) status <- responseStatus response case status of NoError -> return $ Just (responseCas response)@@ -273,25 +393,23 @@ store :: Cluster -> Key -> Value -> Flags -> Expiration -> Version -> StoreMode -> String -> IO Version store c k v f e ver mode operation = do checkValue v- response <- keyedOp c k (\wire -> storeRequest mode wire v f e ver)+ response <- keyedOpExpecting c operation storeReplies k (\wire -> storeRequest mode wire v f e ver) status <- responseStatus response case status of NoError -> return $ responseCas response _ -> failResponse response operation --- | The largest value memcached can store: 1 GiB, its--- @ITEM_SIZE_MAX_UPPER_LIMIT@, which caps its @-I@ item size option.+-- | 1 GiB, memcached's @ITEM_SIZE_MAX_UPPER_LIMIT@. No memcached server can+-- store a larger item, so none can store a longer value. maxValueLength :: Int maxValueLength = 1073741824 --- | Throw 'OpError' 'ErrValueTooLarge', the status memcached returns for a--- value too large to store, if the value is longer than @maxValueLength@.--- Every operation that sends a value calls this first, so that nothing is--- sent: memcached mishandles a data length over 2147483645 bytes and then--- runs the rest of the value's bytes as commands. This evaluates the value+-- | Reject a value longer than @maxValueLength@ before anything is sent.+-- memcached can't store it, and one longer than 2147483645 bytes would make+-- it run the rest of the value's bytes as commands. This evaluates the value -- before a connection is taken. checkValue :: Value -> IO ()-checkValue v = when (B.length v > maxValueLength) $ throwStatus ErrValueTooLarge+checkValue value = when (B.length value > maxValueLength) $ throwStatus ErrValueTooLarge -- | Increment a numeric value stored against a key, returning the incremented -- value and the @Version@ identifier of the key-value pair. Throws@@ -306,7 +424,7 @@ -- -- A non-zero version must match the @Version@ of an existing key-value pair, -- otherwise 'Nothing' is returned. A version of @0@ skips the check.-increment :: Cluster -> Key -> Initial -> Delta -> Expiration -> Version -> IO (Maybe (Word64, Version))+increment :: Client -> Key -> Initial -> Delta -> Expiration -> Version -> IO (Maybe (Word64, Version)) increment c k i d e ver = arithmetic c k i d e ver Incr "INCREMENT" -- | Decrement a numeric value stored against a key, returning the decremented@@ -316,12 +434,12 @@ -- Otherwise this behaves like 'increment': a missing key is created with the -- 'Initial' value unless the expiration is 'maxBound', a version of @0@ skips -- the check, and a non-numeric value throws 'OpError' 'ErrValueNonNumeric'.-decrement :: Cluster -> Key -> Initial -> Delta -> Expiration -> Version -> IO (Maybe (Word64, Version))+decrement :: Client -> Key -> Initial -> Delta -> Expiration -> Version -> IO (Maybe (Word64, Version)) decrement c k i d e ver = arithmetic c k i d e ver Decr "DECREMENT" arithmetic :: Cluster -> Key -> Initial -> Delta -> Expiration -> Version -> ArithMode -> String -> IO (Maybe (Word64, Version)) arithmetic c k i d e ver mode operation = do- response <- keyedOp c k (\wire -> arithmeticRequest mode wire i d e ver)+ response <- keyedOpExpectingOnce c operation arithmeticReplies k (\wire -> arithmeticRequest mode wire i d e ver) status <- responseStatus response case status of NoError -> return $ Just (responseNumber response, responseCas response)@@ -335,9 +453,12 @@ -- doesn't exist. -- -- A non-zero version must match the @Version@ of the key-value pair at the--- server, otherwise 'OpError' 'ErrKeyExists' is thrown. A version of @0@ skips+-- server, otherwise 'Nothing' is returned. A version of @0@ skips the check.+--+-- memcached 1.6.10 to 1.6.12 return the version given instead of the new one,+-- so @0@ if the check was skipped. Passing that @0@ back as a version skips -- the check.-append :: Cluster -> Key -> Value -> Version -> IO (Maybe Version)+append :: Client -> Key -> Value -> Version -> IO (Maybe Version) append c k v ver = modify c k v ver Append "APPEND" -- | Prepend a value to an existing key-value pair, returning the new @Version@@@ -345,19 +466,23 @@ -- doesn't exist. -- -- A non-zero version must match the @Version@ of the key-value pair at the--- server, otherwise 'OpError' 'ErrKeyExists' is thrown. A version of @0@ skips+-- server, otherwise 'Nothing' is returned. A version of @0@ skips the check.+--+-- memcached 1.6.10 to 1.6.12 return the version given instead of the new one,+-- so @0@ if the check was skipped. Passing that @0@ back as a version skips -- the check.-prepend :: Cluster -> Key -> Value -> Version -> IO (Maybe Version)+prepend :: Client -> Key -> Value -> Version -> IO (Maybe Version) prepend c k v ver = modify c k v ver Prepend "PREPEND" modify :: Cluster -> Key -> Value -> Version -> StoreMode -> String -> IO (Maybe Version) modify c k v ver mode operation = do checkValue v- response <- keyedOp c k (\wire -> modifyRequest mode wire v ver)+ response <- keyedOpExpectingOnce c operation storeReplies k (\wire -> modifyRequest mode wire v ver) status <- responseStatus response case status of NoError -> return $ Just (responseCas response) ErrKeyNotFound -> return Nothing+ ErrKeyExists -> return Nothing ErrItemNotStored -> return Nothing _ -> failResponse response operation @@ -366,9 +491,9 @@ -- A non-zero version must match the @Version@ of the key-value pair at the -- server, otherwise the key isn't deleted and 'False' is returned. A version -- of @0@ deletes the key unconditionally.-delete :: Cluster -> Key -> Version -> IO Bool+delete :: Client -> Key -> Version -> IO Bool delete c k ver = do- response <- keyedOp c k (`deleteRequest` ver)+ response <- keyedOpExpectingOnce c "DELETE" deleteReplies k (`deleteRequest` ver) status <- responseStatus response case status of NoError -> return True@@ -376,12 +501,36 @@ ErrKeyExists -> return False _ -> failResponse response "DELETE" +-- | Delete many keys, pipelining the requests to each server as 'getMany'+-- does. The deletes are unconditional, like 'delete' with a version of @0@.+--+-- This returns once each key has been deleted or found missing, without+-- saying which: when a server fails part-way through, the requests it was+-- handling are sent again, and the retry finds the keys the first attempt+-- deleted missing, so a per-key answer would be wrong for them. Use 'delete'+-- to learn whether a key was present: it never sends a request again once+-- the server may have carried it out, and throws instead.+--+-- Every key is checked before anything is sent, so an invalid key deletes+-- nothing. If this throws, some of the keys may have been deleted.+deleteMany :: Client -> [Key] -> IO ()+deleteMany c keys = do+ replies <- keyedBatchOp c [(key, deleteManyRequest) | key <- keys]+ forM_ replies $ \(_, response) -> do+ status <- responseStatus response+ case status of+ NoError -> return ()+ ErrKeyNotFound -> return ()+ _ -> failResponse response "DELETEMANY"+ -- | Remove (delete) all currently stored key-value pairs from the cluster. The -- expiration value can be used to cause this flush to occur in the future--- rather than immediately.-flush :: Cluster -> Maybe Expiration -> IO ()+-- rather than immediately. The servers are flushed in turn, and one that+-- fails, or gives a reply other than @OK@, throws and leaves the servers+-- after it unflushed.+flush :: Client -> Maybe Expiration -> IO () flush c e = do- results <- allOp c (flushRequest e)+ results <- allOpExpecting c "FLUSH" ["OK"] (flushRequest e) forM_ results $ \(_, response) -> do status <- responseStatus response when (status /= NoError) $ failResponse response "FLUSH"@@ -389,14 +538,46 @@ -- | Return statistics on the stored key-value pairs at every available server -- in the cluster. The optional key can be used to select a different set of -- statistics from the server than the default. Most Memcached servers support--- @"items"@, @"slabs"@ or @"settings"@.-stats :: Cluster -> Maybe Key -> IO [(Server, StatResults)]-stats c key = validateStatsArgs key >> allOp' c (runStats key)+-- @"items"@, @"slabs"@ or @"settings"@. A server's reply of more than 262,144+-- entries, or of more than 16 MiB counting each entry's line as it was sent,+-- fails with 'BadLength'.+--+-- Some groups, such as @cachedump@, list keys as memcached stores them, and a+-- key can hold any bytes, a line break among them. 'stats' reads the reply a+-- line at a time, so the bytes after a key's line break are read as a line of+-- their own. Those bytes can split into entries the server never sent, end the+-- reply early (so the entries after them are missing), or make the request+-- fail. So that the rest of such a reply is never read as the reply to a later+-- request, 'stats' closes each connection it reads a reply from rather than+-- returning it to the pool.+--+-- memcached sends an error only as a reply's first line, and never an empty+-- line. So after the first line, an empty line, or one whose first word is+-- @ERROR@, @CLIENT_ERROR@ or @SERVER_ERROR@, can only be a listed key's bytes.+-- 'stats' throws the error such a line raises, in place of all the servers'+-- entries, once every server has answered. It doesn't send the request again+-- or mark the server dead, since a resend would read the same key and fail+-- again. If a server fails, its failure is thrown instead. The entries that+-- keys' bytes add still count towards the limit of 262,144 entries, and a+-- reply they take over it fails as any other reply that is too long does: the+-- request is sent again, and the server is marked dead if every attempt fails.+stats :: Client -> Maybe Key -> IO [(Server, StatResults)]+stats c key = validateStatsArgs key >> allOp' c (runStats key) >>= mapM rethrow where- runStats statsArgs server = withSocket server $ \socket -> do- sendRaw socket (statsRequest statsArgs)- collect socket []+ -- 'withSocket' closes the connection when the function throws, so the+ -- outcome of reading the reply leaves it as an exception, caught at once.+ -- 'retryOp' then sees a success: the error of an empty or error line from+ -- a listed key's bytes neither resends the request nor marks the server+ -- dead, and is rethrown below once every server has answered. Any other+ -- failure, such as a reply too long or an error on the first line, leaves+ -- 'withSocket' as itself, for 'retryOp' to handle as usual.+ runStats statsArgs server = handle (\(StatsRead result) -> return result) $+ withSocket server $ \socket -> do+ sendRaw socket (statsRequest statsArgs)+ collect socket 0 0 [] >>= throwIO . StatsRead + rethrow (server, result) = (,) server <$> either throwIO return result+ -- Arguments may contain spaces (@stats cachedump 1 100@), but control -- bytes would let a caller inject a second command line. validateStatsArgs Nothing = return ()@@ -407,24 +588,95 @@ throwIO $ ClientError (KeyTooLong "invalid stats argument") | otherwise = return () - collect socket values = do- response <- recvResponse socket- case (responseCode response, responseTokens response) of- (code, _) | code `elem` ["ERROR", "CLIENT_ERROR", "SERVER_ERROR"] ->- responseStatus response >> failResponse response "STATS"- (code, _) | code `elem` ["END", "RESET", "OK"] -> return (reverse values)- ("STAT", field:value) -> collect socket ((field, C.unwords value) : values)- (code, tokens) -> collect socket ((code, C.unwords tokens) : values)+ -- Each line counts as sent, line end included. The entries kept are+ -- slices of their lines, so the spaces between tokens take memory too.+ collect socket count size values = do+ when (count > maxStatsEntries || size > maxStatsBytes) $+ throwIO $ ProtocolError BadLength { protocolError = "stats reply too long" }+ line <- recvLine socket+ -- Split the line as 'recvResponse' does: on spaces alone, after removing+ -- the CR that 'recvLine' leaves on it.+ let tokens = filter (not . B.null) (C.split ' ' (fromMaybe line (B.stripSuffix "\r" line)))+ keep entry = collect socket (count + 1) (size + B.length line + 1) (entry : values)+ -- memcached sends an error only as a reply's first line, and never+ -- an empty line, so after the first line either can only be a+ -- listed key's bytes carried past its line break: return its error+ -- rather than fail the server over it.+ keyBytes err+ | count > 0 = return (Left err)+ | otherwise = throwIO err+ case tokens of+ [] -> keyBytes (ProtocolError UnknownPkt { protocolError = "" })+ code:rest+ | code `elem` ["ERROR", "CLIENT_ERROR", "SERVER_ERROR"] ->+ statsError (Response code rest B.empty) >>= keyBytes+ | code `elem` ["END", "RESET", "OK"] -> return (Right (reverse values))+ | otherwise -> case (code, rest) of+ ("STAT", field:value) -> keep (field, C.unwords value)+ _ -> keep (code, C.unwords rest) + -- The error an error line raises, caught so that a key's bytes can be+ -- returned rather than thrown. 'responseStatus' throws for each code that+ -- gets here, and 'failResponse' would for any it didn't.+ statsError response =+ handle return (responseStatus response >> failResponse response "STATS")++-- | The outcome of reading a server's 'stats' reply: its entries, or the+-- error of an empty or error line that a listed key's bytes caused. It is+-- thrown out of 'withSocket' so that the connection it was read from is+-- closed, and unwrapped afterwards.+newtype StatsRead = StatsRead (Either MemcacheError StatResults)+ deriving Show++instance Exception StatsRead++-- | Most entries 'stats' accepts in one server's reply. Memcached sends four+-- lines for each open connection in @stats conns@, and three for each+-- listening socket, so this allows it on a server with nearly 65,536+-- connections. Most other groups are far shorter:+-- @stats items@ has about 30 lines for each slab class, and @stats sizes@ one+-- for each 32-byte size bucket in use (at most 32,768 with the default 1 MiB+-- item size limit).+maxStatsEntries :: Int+maxStatsEntries = 256 * 1024++-- | Most bytes of entries 'stats' accepts in one server's reply, counting each+-- line as it was sent. Replies to @stats conns@ and @stats cachedump@ are+-- smaller: 262,144 entries of @stats conns@ come to 15 MiB even with+-- five-digit descriptors, the longest Unix socket path and state name, and+-- ten-digit idle times, and memcached stops @stats cachedump@ at 2 MiB.+maxStatsBytes :: Int+maxStatsBytes = 16 * 1024 * 1024+ -- | Return the version string of the Memcached cluster. Only queries -- one server and assumes all servers in the cluster run the same version.-version :: Cluster -> IO ByteString+version :: Client -> IO ByteString version c = do- response <- anyOp c versionRequest+ response <- anyOpExpecting c "VERSION" ["VERSION"] versionRequest status <- responseStatus response if responseCode response == "VERSION" && status == NoError then return $ fromMaybe B.empty (listToMaybe (responseTokens response)) else failResponse response "VERSION"++-- | The codes of the replies, besides error replies, that memcached 1.6.13+-- and later send to each request; a reply with any other code closes the+-- connection, as 'keyedOpExpecting' describes. Up to 1.6.17, memcached run+-- with @-o meta_response_old@ sends @OK@ in place of @HD@. Stores and deletes+-- take it as success; 'touch' doesn't.+getReplies, touchReplies, storeReplies, arithmeticReplies, deleteReplies :: [ByteString]+getReplies = ["VA", "EN"]+touchReplies = ["HD", "EN"]+storeReplies = ["HD", "OK", "NS", "EX", "NF"]+arithmeticReplies = ["VA", "NF", "NS", "EX"]+deleteReplies = ["HD", "OK", "NF", "EX"]++-- | 'keyedOpExpecting' for a store without a CAS version, which can be sent+-- again: it stores the same value. A store with one is sent once, as+-- 'keyedOpExpectingOnce' describes: its success changes the item's version,+-- so a second one would find it changed and report failure.+resendUnlessVersioned :: Version -> Cluster -> String -> [ByteString] -> Key -> (WireKey -> ByteString) -> IO Response+resendUnlessVersioned 0 = keyedOpExpecting+resendUnlessVersioned _ = keyedOpExpectingOnce -- | The value, flags and CAS version carried by a @VA@ response. valueResult :: Response -> (Value, Flags, Version)
Database/Memcache/Cluster.hs view
@@ -12,51 +12,216 @@ Handles a group of connections to different Memcached servers. -We use consistent hashing to choose which server to route a request to. On an-error, we mark the server as failed and remove it temporarily from the set of-servers available.+= Routing++Keys are spread over the servers by ketama consistent hashing, computed+exactly as libmemcached does for @MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED@ (see+"Database.Memcache.Ketama"). Each server gets 160 points on a 32-bit ring+(156 for some numbers of servers), taken from MD5 digests of its name, and a+key goes to the server owning the first point at or after the MD5 hash of the+key. Adding or removing a server only moves the keys on that server's points,+unless it changes the number of points each server gets (from 24 servers to+25, say), which moves a few percent of the other keys too.++Every server has weight 1, and is named from the host and port strings of its+'ServerSpec' exactly as written: the host alone for port @\"11211\"@,+otherwise @host:port@. Hosts are not resolved, so @localhost@ and @127.0.0.1@+are different names. A Unix socket is named by its path in the same way, as+the bytes the client connects to, one per 'Char' (see 'ssHost'), whereas a+host is named in UTF-8. The client doesn't use a socket's port to connect,+but port @\"11211\"@ names it by the path alone, as libmemcached's+@memcached_server_add@ does, and @\"0\"@ names it @path:0@, as libmemcached's+@memcached_server_add_unix_socket@ and pylibmc do. The key is hashed as given,+not in its base64 form on the wire. Other clients using libmemcached's+weighted ketama distribution therefore choose the same server for every key,+as long as they are given the same host strings and ports in the same order+and equal weights: libmemcached with @MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED@,+php-memcached with @OPT_LIBKETAMA_COMPATIBLE@, pylibmc with the+@ketama_weighted@ behavior, and twemproxy with @hash: md5@ and+@distribution: ketama@ and its servers listed without node names.++The order of the servers only matters in the rare case that two of them have+a point with the same value. The server listed first then owns the point, as+in libmemcached 1.1 and later. Earlier versions of libmemcached, and+twemproxy, leave such a tie to their C library's sort, so they may choose+differently for the keys just below that point.++'ketamaServerForKey' routes keys this way, and is the default+'optsGetServerForKey'.++= Failures++When every attempt at an operation against a server fails, we mark the+server as failed and remove it temporarily from the set of servers available,+unless the last attempt only ran out of time waiting for a free connection to+it (see 'optsServerRetries'). An operation whose request mustn't be sent+twice, such as an increment, makes no more attempts once one has failed after+its request may have reached the server, so that failure marks the server+failed (see 'keyedOpExpectingOnce'). A key whose server is marked failed goes+to the next live server found walking forward round the ring, so keys on live+servers stay where they are, and 'NoServersReady' is thrown only when every+server is marked failed. This is where libmemcached's auto-eject sends the key too,+except when the number of live servers changes how many points each server+gets (from 26 servers to 25, say): libmemcached rebuilds the ring from the+live servers, while we keep every server's points.++= Upgrading from 0.4++Versions before 0.5 compared a @hashable@ hash of the key with hashes of the+servers. They sent keys to no more than two servers at a time, and the+mapping could change with the @hashable@ version. That router is kept for+one release, deprecated, as 'legacyServerForKey'.++With /n/ servers, upgrading moves all but about one key in /n/ to a+different server, once: about half the keys with 2 servers, 70% with 3 and+88% with 8. Expect a burst of cache misses. A single server is unaffected.++While old and new versions run side by side, as in a rolling deploy, they+read and write a moved key on different servers. A change or delete made by+one is not seen by the other, which can read a stale copy until the copy+expires, even after the deploy.++One way to switch is all at once, at a quiet time. Copies stored under the+old mapping stay on their servers, where failover can still find them, so+you may want to flush the cache afterwards.++The other way avoids the stale reads of a rolling deploy by switching in+stages. Deploy each step to every instance before starting the next:++1. Upgrade with @optsGetServerForKey = legacyServerForKey@. Keys stay where+ they are, as long as the build uses the same @hashable@ version and+ platform as before, and the servers come from 'newCluster' or from+ ElastiCache discovery (see 'legacyServerForKey'). GHC warns that it is+ deprecated, so a build with @-Werror@ also needs @-Wno-deprecations@.++2. Add a second client for the same servers that uses 'ketamaServerForKey'.+ Whenever a key is changed or deleted, also delete it through the second+ client, unless both clients send it to the same server: compare the+ 'serverHost' and 'serverPort' of the servers their routers return for+ the key.++3. If a server was marked failed less than your longest expiry time ago,+ flush the cache first. Then swap the two clients' routers. Every change+ still deletes the key under the other mapping, so neither version reads a+ stale copy of a key the other has changed or deleted.++4. Once your longest expiry time has passed since step 3 was on every+ instance, remove the second client. Until then, copies stored under the+ old mapping can still be on a key's old server, where failover can find+ them, and the second client's deletes keep them from going stale. If+ some keys never expire, flush the cache instead of waiting, and then+ remove the second client.++The flush in step 3 removes copies that a failover stored on a key's new+server before step 2 was on every instance. Step 2's deletes remove such a+copy only when the key next changes. Counters and locks kept in memcached+can't be moved this way: while instances use different mappings, they+update them on different servers.++'optsServerRetries' changed too. It now counts the retries after an+operation's first attempt, where 0.4 counted the attempts, so a value of 1+or more that you set yourself makes one more attempt than in 0.4, after one+more 'optsFailRetryDelay'. Set it one lower to keep the old number of+attempts. The default went from 2 to 1, which still means two attempts. -} module Database.Memcache.Cluster ( -- * Cluster -- | A collection of Memcached servers with retry and routing state.- Cluster, ServerSpec(..), Options(..), newCluster, getServers, setServers,+ Cluster, ServerSpec(..), Options(..), defaultOptions, newCluster,+ getServers, setServers, + -- * Routing+ -- | Routers for 'optsGetServerForKey', and 'isAlive' for writing one.+ ketamaServerForKey, isAlive, legacyServerForKey,+ -- * Operations- Retries, keyedOp, keyedBatchOp, anyOp, allOp, allOp'+ Retries, keyedOp, keyedOpExpecting, keyedOpExpectingOnce, keyedBatchOp,+ anyOp, anyOpExpecting, allOp, allOpExpecting, allOp' ) where import Database.Memcache.Errors+import qualified Database.Memcache.Ketama as Ketama import Database.Memcache.Server import Database.Memcache.Types+import Database.Memcache.Types.Server (Server (slotTaken)) import Control.Concurrent (threadDelay) import Control.Concurrent.MVar (MVar, readMVar)-import Control.Monad (foldM, void)+import Control.Monad (foldM, unless, void, when, (<=<)) import Data.ByteString (ByteString) import qualified Data.ByteString as B+import qualified Data.ByteString.Char8 as C import Data.Default import Data.Fixed (Milli) import Data.Hashable (hash) import Data.IORef-import Data.List (find, sort)-import Data.Maybe (fromMaybe)+import Data.List (find)+import Data.Maybe (isJust)+import qualified Data.Text as T+import qualified Data.Text.Encoding as T import Data.Time.Clock (NominalDiffTime) import Data.Time.Clock.POSIX (getPOSIXTime) import qualified Data.Vector as V import System.Timeout-import UnliftIO.Exception (SomeException, fromException, handle, throwIO)-import Database.Memcache.Socket (Response(..), WireKey(..), encodeKey, noOpRequest, recvResponse, responseStatus, sendRaw)+import UnliftIO.Exception (Exception, SomeException, fromException, handle, throwIO, try)+import Database.Memcache.Socket (Response(..), WireKey, checkIdle, encodeKey, noOpRequest, recvResponse,+ responseStatus, sendRaw) --- | Number of times to retry an operation before considering it failed.+-- | Number of times to retry an operation against a server after its first+-- attempt fails, before giving up (see 'optsServerRetries'). type Retries = Int -- | Options specifies how a Memcached cluster should be configured. data Options = Options {- -- | Number of times to retry an operation on failure. If consecutive- -- failures exceed this value for a server, we mark it as down and- -- failover to a different server for the next operation.+ -- | Number of times to retry an operation against a server after it+ -- fails, so each operation is tried at most @1 + optsServerRetries@+ -- times. Zero or a negative value means it is tried once. If every+ -- attempt fails, we mark the server as down and failover to a+ -- different server for the next operation. An attempt that only ran+ -- out of time waiting for a free connection to the server uses up a+ -- retry like any failed attempt, but the last attempt decides: if it+ -- was such an attempt, the server isn't marked down, whatever the+ -- earlier attempts did (see 'optsServerTimeout'). --- -- Default is 2.+ -- A @SERVER_ERROR@ reply whose text names no status, such as+ -- memcached's proxy sends when a backend fails, is a failed attempt,+ -- as a timeout or a network error is: it closes the server's idle+ -- connections, and counts towards marking the server down. Other+ -- error replies are answers, thrown at once without counting against+ -- the server: @ERROR@, @CLIENT_ERROR@, and a @SERVER_ERROR@ whose text+ -- names a status, such as running out of memory.+ --+ -- A request that mustn't be sent twice is retried only after a+ -- failure before any of it was sent: while waiting for a free+ -- connection, connecting, authenticating or checking the server's+ -- version, or on finding that the server has closed or reset the+ -- connection, or that bytes from the server are already waiting on+ -- it. Those are @increment@, @decrement@, @append@, @prepend@, @add@+ -- and @delete@, and @cas@ and @replace@ with a version other than 0.+ -- Once the request may have reached the server, a failure such as a+ -- timeout, a network error, the server closing the connection, a+ -- reply that can't be read or a @SERVER_ERROR@ that names no status+ -- is thrown as it is, and the server is marked down as if every+ -- attempt had failed, whatever retries are left. The server may have+ -- carried out the request, and a retry could apply it twice, or+ -- report failure because the first one succeeded (see+ -- 'keyedOpExpectingOnce'). So behind memcached's proxy, a request for+ -- a key on a failed backend marks the whole proxy down: at once if it+ -- mustn't be sent twice, and otherwise once every attempt gets that+ -- reply.+ --+ -- Every other operation is retried after any failure, so the server+ -- may carry out its request more than once: gets, @touch@, @set@,+ -- @cas@ and @replace@ with version 0, the batches of @getMany@,+ -- @setMany@ and @deleteMany@, @flush@, @stats@ and @version@, and+ -- the operations of 'keyedOp', 'anyOp', 'allOp' and 'allOp''. Sent+ -- again, these leave the same data and get an answer as true as the+ -- first, but a store or delete repeated after another client has+ -- changed the key undoes that change. Set this to 0 for no retries.+ --+ -- Default is 1, so an operation is tried at most twice. Before+ -- version 0.5 this was the number of attempts, not retries, and+ -- defaulted to 2, which also meant two attempts. optsServerRetries :: Retries, -- | After an operation has failed, how long to wait before retrying it -- while still within the 'optsServerRetries' count?@@ -68,18 +233,29 @@ -- -- Default is 1500ms. optsDeadRetryDelay :: Milli,- -- | How long to wait for an operation to complete before considering- -- it failed.+ -- | How long each attempt at an operation may take before it is+ -- considered failed. @getMany@, @setMany@ and @deleteMany@ make their+ -- attempts for each batch, and @stats@ and @flush@ for each server in+ -- turn. The time counts from the start of the attempt: it covers+ -- waiting for a free connection in the server's pool, opening a new+ -- connection if one is needed (connecting, authenticating when there+ -- are credentials, and asking the server for its version), sending+ -- the request and reading the reply. --- -- When an attempt at an operation has to open a new connection, this- -- also covers opening it: connecting, authenticating when there are- -- credentials, and asking the server for its version.+ -- An attempt that runs out of time while it is still waiting for a+ -- free connection never reached the server. It is retried like any+ -- failed attempt, but if it was the last attempt, the operation+ -- throws v'Timeout' without marking the server down. Running out of+ -- time afterwards, while opening (including authenticating and+ -- checking the version) or using the connection, is a server failure+ -- like any other, however long the attempt waited first. -- -- Default is 750ms. optsServerTimeout :: Milli, -- | Figure out which server to talk to for a given key. --- -- Default is @getServerForKeyDefault@.+ -- Default is 'ketamaServerForKey'. A custom router can use 'isAlive'+ -- to skip servers marked failed. optsGetServerForKey :: Cluster -> Key -> IO (Maybe Server), -- | Convert server specifications into servers. --@@ -90,20 +266,41 @@ optsServerOptions :: ServerOptions } -instance Default Options where- def = Options {- optsServerRetries = 2,+-- | The default cluster options:+--+-- * 'optsServerRetries': 1 (one retry, so an operation is tried at most twice)+-- * 'optsFailRetryDelay': 200ms+-- * 'optsDeadRetryDelay': 1500ms+-- * 'optsServerTimeout': 750ms+-- * 'optsGetServerForKey': 'ketamaServerForKey'+-- * 'optsServerSpecsToServers': 'newServerDefault' for each server+-- * 'optsServerOptions': 'defaultServerOptions'+--+-- Change individual fields with a record update, for example+-- @defaultOptions { optsServerTimeout = 250 }@.+defaultOptions :: Options+defaultOptions = Options {+ optsServerRetries = 1, optsFailRetryDelay = 200, optsDeadRetryDelay = 1500, optsServerTimeout = 750,- optsGetServerForKey = getServerForKeyDefault,+ optsGetServerForKey = ketamaServerForKey, optsServerSpecsToServers = mapM . newServerDefault,- optsServerOptions = def+ optsServerOptions = defaultServerOptions } +-- | 'def' is 'defaultOptions'.+instance Default Options where+ def = defaultOptions+ -- | Memcached cluster. data Cluster = Cluster {- cServers :: Either (MVar (V.Vector Server)) (V.Vector Server),+ -- | Either servers that discovery replaces, or a fixed set of servers+ -- and their ketama ring, built on first use.+ cServers :: Either (MVar (V.Vector Server)) (V.Vector Server, Ketama.Ring),+ -- | For servers in an 'MVar': the ketama ring last built and the+ -- servers it was built for.+ cRingCache :: IORef (V.Vector Server, Ketama.Ring), -- See @Options@ for description of these values. @@ -116,20 +313,30 @@ -- | Get the currently configured servers. getServers :: Cluster -> IO (V.Vector Server)-getServers = either readMVar pure . cServers+getServers = either readMVar (pure . fst) . cServers -- | Replace the cluster's server set, preserving its routing and retry options.+--+-- Servers are kept in the order given, which breaks ties on the ketama ring+-- (see "Database.Memcache.Cluster"). A cluster given an 'MVar' caches the ring+-- for the servers in it, rebuilding it only when their hosts or ports change.+-- Every cluster made by 'setServers' shares that cache with the one it was+-- made from, so alternating between two clusters with different hosts in+-- their 'MVar's rebuilds the ring on each switch. setServers :: Cluster -> Either (MVar (V.Vector Server)) (V.Vector Server) -> Cluster-setServers c servers = c { cServers = servers }+setServers c servers = c { cServers = fmap withRing servers } -- | Establish a new connection to a group of Memcached servers. newCluster :: [ServerSpec] -> Options -> IO Cluster newCluster [] _ = throwIO $ ClientError NoServersReady newCluster hosts Options{..} = do s <- optsServerSpecsToServers optsServerOptions hosts+ let fixed = withRing (V.fromList s)+ cache <- newIORef fixed return $ Cluster {- cServers = Right $ V.fromList $ sort s,+ cServers = Right fixed,+ cRingCache = cache, cRetries = optsServerRetries , cFailDelay = toMicroseconds optsFailRetryDelay, cDeadDelay = fromRational $ toRational optsDeadRetryDelay / 1000,@@ -158,32 +365,143 @@ writeIORef (failed s) 0 return True +-- | Whether a server can be used: it isn't marked failed, or it was marked+-- failed at least 'optsDeadRetryDelay' ago, in which case this clears the+-- mark. A custom 'optsGetServerForKey' can use it to skip servers marked+-- failed, as 'ketamaServerForKey' does.+isAlive :: Cluster -> Server -> IO Bool+{-# INLINE isAlive #-}+isAlive c = serverAlive (cDeadDelay c)+ -- | The servers not currently marked as failed. aliveServers :: Cluster -> IO (V.Vector Server) {-# INLINE aliveServers #-} aliveServers c = getServers c >>= V.filterM (serverAlive $ cDeadDelay c) --- | Figure out which server to talk to for this key. I.e., the distribution--- method. We use consistent hashing based on the CHORD approach.-getServerForKeyDefault :: Cluster -> Key -> IO (Maybe Server)-{-# INLINE getServerForKeyDefault #-}-getServerForKeyDefault c k = do- let hashedKey = hash k- searchF s = sid s < hashedKey+-- | The default router: a key goes to its owner on the ketama ring, or if+-- that server is marked failed, to the next live server walking forward round+-- the ring (see the module description). 'Nothing' if the cluster has no+-- servers or every server is marked failed.+ketamaServerForKey :: Cluster -> Key -> IO (Maybe Server)+{-# INLINE ketamaServerForKey #-}+ketamaServerForKey c k = do+ (servers, ring) <- either (cachedRing c <=< readMVar) pure (cServers c)+ let firstAlive [] = return Nothing+ firstAlive (i:is) = do+ let s = servers V.! i+ alive <- isAlive c s+ if alive then return (Just s) else firstAlive is+ firstAlive $ Ketama.owners ring $ Ketama.keyHash k++-- | Servers read from the cluster's 'MVar', with their ketama ring. The ring+-- last built is cached with the servers it was built for, and reused while the+-- 'MVar' holds the same 'Server' values, as it does after an ElastiCache+-- refresh that finds the same nodes. Otherwise the ring is reused if the+-- hosts and ports are the same, and rebuilt only if they differ.+cachedRing :: Cluster -> V.Vector Server -> IO (V.Vector Server, Ketama.Ring)+cachedRing c servers = do+ cached@(builtFor, _) <- readIORef (cRingCache c)+ if sameServers builtFor servers+ then return cached+ -- Install the new ring before building it, so concurrent requests+ -- share one build.+ else atomicModifyIORef' (cRingCache c) $ \(builtFor', ring) ->+ let new | sameNames builtFor' servers = (servers, ring)+ | otherwise = withRing servers+ in (new, new)++-- | Whether two vectors hold the same 'Server' values in the same order. Each+-- 'Server' has its own 'failed' 'IORef', so this compares a pointer per server,+-- and allocates nothing.+sameServers :: V.Vector Server -> V.Vector Server -> Bool+sameServers = pairwise (\x y -> failed x == failed y)++-- | Whether two vectors hold servers with the same hosts and ports in the same+-- order, and so the same ketama ring.+sameNames :: V.Vector Server -> V.Vector Server -> Bool+sameNames = pairwise (\x y -> serverHost x == serverHost y && serverPort x == serverPort y)++-- | Whether two vectors are the same length and every pair of servers at the+-- same position passes the test, checked with a loop so nothing is allocated.+pairwise :: (Server -> Server -> Bool) -> V.Vector Server -> V.Vector Server -> Bool+{-# INLINE pairwise #-}+pairwise same xs ys = V.length xs == V.length ys && go 0+ where+ go i = i == V.length xs || (same (V.unsafeIndex xs i) (V.unsafeIndex ys i) && go (i + 1))++-- | Servers with their ketama ring, all with weight 1, named from the host and+-- port they were configured with. The ring is built when it is first used.+withRing :: V.Vector Server -> (V.Vector Server, Ketama.Ring)+withRing servers = (servers, Ketama.ring (map named (V.toList servers)))+ where+ named s = (Ketama.serverName (hostBytes (serverHost s)) (utf8 (serverPort s)), 1)+ -- A Unix socket's path holds a byte per Char, the bytes the client+ -- connects to, and libmemcached names a socket by those bytes. Encoding+ -- them in UTF-8, as a host's characters are, would give a path outside+ -- ASCII another name.+ hostBytes path@('/' : _) = C.pack path+ hostBytes host = utf8 host+ utf8 = T.encodeUtf8 . T.pack++-- | The router of versions before 0.5. It is kept for one release, so that+-- a fleet can switch to 'ketamaServerForKey' in stages (see the module+-- description).+--+-- Like those versions, it sends a key to the live server with the lowest+-- @sid@ if that @sid@ is below the key's @hashable@ hash, and otherwise to+-- the live server with the highest @sid@. It gives the same server as 0.4+-- only when built with the same @hashable@ version on the same platform, as+-- @hashable@'s hashes can differ between them. 0.4 also kept the servers of+-- 'newCluster' and of ElastiCache discovery sorted by @sid@, but searched+-- servers given to 'setServers' in the order given, so for those this+-- router can choose a different server than 0.4 did.+legacyServerForKey :: Cluster -> Key -> IO (Maybe Server)+legacyServerForKey c k = do servers' <- aliveServers c+ -- 0.4 kept the servers sorted by sid, and sent the key to the first one+ -- whose sid is below its hash, or else to the last one: so to the first+ -- server with the lowest sid, if that is below the hash, or else to the+ -- last one with the highest. The servers are no longer sorted, so find+ -- those two, in one pass each.+ let lowest = V.foldl1' (\s s' -> if sid s' < sid s then s' else s) servers'+ highest = V.foldl1' (\s s' -> if sid s' >= sid s then s' else s) servers' return $ if V.null servers' then Nothing- else Just $ fromMaybe (V.last servers') (V.find searchF servers')+ else Just $ if sid lowest < hash k then lowest else highest+{-# DEPRECATED legacyServerForKey "It sends keys to no more than two servers \+ \at a time. Use ketamaServerForKey, the default; this will be removed in \+ \the next major version." #-} +-- | Whether a request can be sent again after a failure that may have come+-- after the server received it.+data Resend+ -- | Sending it again leaves the same data and gets an answer as true as the+ -- first: a get, an unconditional set, a flush.+ = Resend+ -- | It is sent at most once: a retry could carry it out twice, or answer+ -- for the first one's effect, as an increment or an @add@ would.+ | SendOnce+ deriving Eq+ -- | Run a Memcached operation against a particular server, handling any--- failures that occur, retrying the specified number of times.-serverOp :: Cluster -> Server -> ByteString -> IO Response+-- failures that occur, retrying the specified number of times. The reply is+-- checked with the given function while the connection is held, so that a+-- reply it rejects closes the connection.+--+-- A request sent once is retried only after a failure before it was sent.+-- So that a connection the server closed while it sat idle in the pool+-- doesn't count as one, the connection is checked just before sending.+serverOp :: Cluster -> Server -> Resend -> (Response -> IO ()) -> ByteString -> IO Response {-# INLINE serverOp #-}-serverOp c s req = retryOp c s $ withSocket s $ \connection -> do- sendRaw connection req- response <- recvResponse connection- rejectProtocolError response- return response+serverOp c s resend check req = handle (\(OutOfStep err) -> throwIO err) $+ retryOp c s $ \sent s' -> withSocket s' $ \connection -> do+ when (resend == SendOnce) $ do+ checkIdle connection+ writeIORef sent True+ sendRaw connection req+ response <- recvResponse connection+ check response+ return response rejectProtocolError :: Response -> IO () rejectProtocolError response@@ -191,91 +509,258 @@ void (responseStatus response) | otherwise = return () --- | Run a keyed request against the server selected for the key.+-- | Reject a reply whose code is none of the given ones, as+-- 'keyedOpExpecting' describes.+expectReply :: String -> [ByteString] -> Response -> IO ()+expectReply operation codes response = do+ rejectProtocolError response+ unless (responseCode response `elem` codes) $ handle (throwIO . OutOfStep) $ do+ -- An unknown code throws 'UnknownStatus' here.+ void (responseStatus response)+ throwIO $ ProtocolError UnexpectedResponse+ { protocolError = "Expected " <> operation <> ", got: " <> show (responseCode response) }++-- | An error for a reply that shows its connection is out of step with its+-- requests. @expectReply@ throws it inside 'withSocket', so that the+-- connection is closed, and @serverOp@ throws the error it carries once+-- @retryOp@ has let it through.+newtype OutOfStep = OutOfStep MemcacheError+ deriving Show++instance Exception OutOfStep++-- | Run a keyed request against the server selected for the key. Of the+-- replies it can read, only an error reply throws: @ERROR@ throws+-- 'UnknownOp', and @CLIENT_ERROR@ and @SERVER_ERROR@ throw 'BadCommand' and+-- 'ServerError', or an 'OpError' when their text names a status, such as+-- 'ErrValueTooLarge'. Each is an answer, thrown at once, except 'ServerError':+-- a @SERVER_ERROR@ whose text names no status is a server failure, as a+-- timeout is. Any other reply is returned, and the connection goes back to+-- the pool, so a reply the request cannot get leaves it out of step;+-- 'keyedOpExpecting' rejects such a reply.+--+-- After a failure the request is sent again, even if the server may have+-- carried it out (see 'optsServerRetries'), so use it only for requests that+-- can be repeated. 'keyedOpExpectingOnce' is for those that can't. keyedOp :: Cluster -> Key -> (WireKey -> ByteString) -> IO Response {-# INLINE keyedOp #-}-keyedOp c k makeReq = do+keyedOp c = keyedOpWith c Resend rejectProtocolError++-- | Like 'keyedOp', for a request whose reply can only have one of the given+-- codes, such as @[\"VA\", \"EN\"]@ for a get with the @v@ flag. A reply with+-- any other code shows that the connection is out of step with its requests:+-- the reply is likely an earlier request's, and this request's own reply is+-- still to come. Such a reply throws while the connection is held, so the+-- connection is closed rather than returned to the pool, where later requests+-- would read the replies left on it. A code that 'responseStatus' doesn't+-- know throws 'UnknownStatus'. Any other throws 'UnexpectedResponse', whose+-- message names the given operation and the code, as in+-- @Expected GET, got: \"HD\"@. The server has received the request, so it+-- is not sent again, and the error doesn't count towards marking the server+-- dead. Error replies (@ERROR@, @CLIENT_ERROR@ and @SERVER_ERROR@) throw as+-- they do for 'keyedOp', where a @SERVER_ERROR@ whose text names no status+-- is a server failure. After a server failure, the request is sent again, as+-- with 'keyedOp'.+keyedOpExpecting :: Cluster -> String -> [ByteString] -> Key -> (WireKey -> ByteString) -> IO Response+{-# INLINE keyedOpExpecting #-}+keyedOpExpecting c operation codes = keyedOpWith c Resend (expectReply operation codes)++-- | Like 'keyedOpExpecting', for a request that mustn't be sent twice, such+-- as an increment, an @add@, a delete, or a store with a CAS version. Once+-- the request may have reached the server, a failure, such as a timeout, a+-- network error, the server closing the connection, a reply that can't be+-- read or a @SERVER_ERROR@ whose text names no status, isn't retried: the+-- server may have carried out the request, and a second one could apply it+-- twice, or report failure because the first one succeeded. The failure is+-- thrown as it is, and still counts against the server: its idle connections+-- are closed, and it is marked dead, as when the last of an operation's+-- attempts fails. A failure before any of the request is sent, such as+-- waiting for a connection, connecting, authenticating or checking the+-- server's version, is retried as for 'keyedOpExpecting'.+--+-- Before sending, the connection is checked, without waiting, for the server+-- having closed or reset it, as memcached closes idle connections when it+-- restarts or after @-o idle_timeout@, and for bytes from the server already+-- waiting on it. Such a connection is closed, and as after any failure+-- before sending, the request is retried on another if retries are left.+-- With 'optsServerRetries' at 0, the failure is thrown instead, and the+-- server is marked dead (see 'Database.Memcache.Socket.checkIdle').+keyedOpExpectingOnce :: Cluster -> String -> [ByteString] -> Key -> (WireKey -> ByteString) -> IO Response+{-# INLINE keyedOpExpectingOnce #-}+keyedOpExpectingOnce c operation codes = keyedOpWith c SendOnce (expectReply operation codes)++-- | Run a keyed request, checking its reply with the given function.+keyedOpWith :: Cluster -> Resend -> (Response -> IO ()) -> Key -> (WireKey -> ByteString) -> IO Response+{-# INLINE keyedOpWith #-}+keyedOpWith c resend check k makeReq = do wire <- either (throwIO . ClientError) return (encodeKey k) s' <- cGetServerForKey c c k case s' of- Just s -> serverOp c s (makeReq wire)+ Just s -> serverOp c s resend check (makeReq wire) Nothing -> throwIO $ ClientError NoServersReady --- | Run keyed requests grouped by their selected server.-keyedBatchOp :: Cluster -> [(Key, WireKey -> ByteString)] -> IO [(Key, Response)]+-- | Run keyed requests grouped by their selected server, pipelining each+-- server's requests in batches. Each request is built from its key's wire+-- form and an opaque. The request must send the opaque unchanged as its @O@+-- flag, in decimal, as the batch encoders in "Database.Memcache.Socket" do.+-- memcached copies it into the reply, and that matches the reply to its+-- request. A request may get no reply, as a quiet miss does. The results are+-- grouped by server, and keep the order given within a server.+--+-- Like 'keyedOp', a batch is sent again after a failure, even one after the+-- server may have carried out some of its requests, so use it only for+-- requests that can be repeated.+--+-- A miss reply (@EN@) with no opaque is skipped, as there is nothing to+-- return for it. memcached sends one when a read from extstore misses (for+-- quiet gets too, from 1.6.40) and, before 1.6.18, for any get miss that is+-- not quiet. As memcached answers in order, it counts as the reply to the+-- next request, after the one the previous reply answered. Error replies+-- (@ERROR@, @CLIENT_ERROR@ and @SERVER_ERROR@) throw as they do for+-- 'keyedOp', and here too a @SERVER_ERROR@ whose text names no status is a+-- server failure. A reply with an unknown code throws 'UnknownStatus', as its+-- length can't be known. Any other reply with no opaque, with the opaque of+-- no request in its batch, or out of order throws 'UnexpectedResponse', as+-- does a bare @EN@ after the reply to the batch's last request. Either way+-- the connection is closed rather than reused. Like any server failure, the+-- batch is then sent again on a new connection, and the failure counts+-- towards marking the server dead.+keyedBatchOp :: Cluster -> [(Key, WireKey -> Int -> ByteString)] -> IO [(Key, Response)] keyedBatchOp c requests = do- groups <- foldM add [] requests+ groups <- foldM add [] (zip [0..] requests) concat <$> mapM run groups where- add groups (key, makeRequest) = do+ add groups (opaque, (key, makeRequest)) = do wire <- either (throwIO . ClientError) return (encodeKey key) server <- cGetServerForKey c c key case server of Nothing -> throwIO $ ClientError NoServersReady- Just s -> return (insertGroup s (key, wire, makeRequest) groups)+ Just s -> return (insertGroup s (key, opaqueFlag opaque, makeRequest wire opaque) groups) + opaqueFlag opaque = C.pack ('O' : show opaque)+ insertGroup s item [] = [(s, [item])] insertGroup s item ((s', items):rest) | s == s' = (s', item:items) : rest | otherwise = (s', items) : insertGroup s item rest - run (server, items) = concat <$> mapM (runBatch server) (chunksOf batchSize (reverse items))+ run (server, items) = concat <$> mapM (runBatch server) (batches (reverse items)) -- Write the whole batch in one go, then read it back. Batches are bounded -- so the server's replies cannot fill the socket buffers while we are -- still writing, which would deadlock both ends until the op times out.- runBatch server items = retryOp c server $ withSocket server $ \connection -> do- sendRaw connection (B.concat (map request items) <> noOpRequest)- replies <- recvUntilMN connection []- return [(key, response) | response <- replies, Just key <- [responseKey response items]]+ runBatch server items = retryOp c server $ \_ server' -> withSocket server' $ \connection -> do+ sendRaw connection (B.concat (map request items ++ [noOpRequest]))+ recvUntilMN connection items [] - recvUntilMN connection acc = do+ -- memcached answers in order, so each reply must carry the opaque of a+ -- request after the one the previous reply answered; the requests in+ -- between got no reply. A bare EN is a miss whose opaque memcached+ -- dropped: we can't tell which request it answers, but it is one line+ -- with nothing to return, so skip it. Later replies answer later+ -- requests, so count it as the next request's reply. With no request+ -- left it is one reply too many, and a server sending bare ENs without+ -- end would otherwise keep the batch reading. Anything else means the+ -- replies are out of step with the requests, as may a reply with an+ -- unknown code, whose length we can't know. So throw while the connection+ -- is held, and it is closed rather than returned to the pool. Echoed keys+ -- can't be matched instead: memcached echoes a key raw or as base64+ -- according to how its item was stored, not how the request sent it, and+ -- a raw key can contain a newline.+ recvUntilMN connection pending acc = do response <- recvResponse connection- rejectProtocolError response- if responseCode response == "MN"- then return (reverse acc)- else recvUntilMN connection (response : acc)-- -- Match the echoed key together with its base64 flag: a plain key can- -- have the same bytes as another key's base64 form.- responseKey response items = do- echoed <- find (B.isPrefixOf "k") (responseTokens response)- let wire = WireKey (B.drop 1 echoed) ("b" `elem` responseTokens response)- (key, _, _) <- find ((== wire) . snd3) items- return key+ void (responseStatus response)+ case (responseCode response, find (B.isPrefixOf "O") (responseTokens response)) of+ ("MN", _) -> return (reverse acc)+ ("EN", Nothing) | _ : rest <- pending -> recvUntilMN connection rest acc+ (_, opaque) -> case dropWhile (\(_, opaque', _) -> Just opaque' /= opaque) pending of+ (key, _, _) : rest -> recvUntilMN connection rest ((key, response) : acc)+ [] -> throwIO $ ProtocolError UnexpectedResponse+ { protocolError = "Expected a reply to a request in the batch, got: "+ <> show (C.unwords (responseCode response : responseTokens response)) } - request (_, wire, makeRequest) = makeRequest wire- snd3 (_, wire, _) = wire+ request (_, _, req) = req batchSize = 256 :: Int - chunksOf n xs = case splitAt n xs of- (chunk, []) -> [chunk]- (chunk, rest) -> chunk : chunksOf n rest+ -- Each batch must be sent and answered within the server timeout, so a+ -- batch also closes once its requests reach 1 MiB, memcached's default+ -- item size limit. A batch is then no larger than a single large store;+ -- a request over the limit goes in a batch of its own.+ batchBytes = 1048576 :: Int + batches [] = []+ batches xs = batch : batches rest+ where (batch, rest) = splitBatch 0 0 xs++ splitBatch n bytes (x:xs)+ | n == 0 || (n < batchSize && bytes' <= batchBytes) =+ let (batch, rest) = splitBatch (n + 1) bytes' xs in (x : batch, rest)+ where bytes' = bytes + B.length (request x)+ splitBatch _ _ xs = ([], xs)+ -- | Run a Memcached operation against any single server in the cluster, -- handling any failures that occur, retrying the specified number of times.+-- As with 'keyedOp', of the replies it can read, only an error reply throws;+-- 'anyOpExpecting' also rejects a reply the request cannot get. Also as with+-- 'keyedOp', the request is sent again after a failure, so it must be one+-- that can be repeated. anyOp :: Cluster -> ByteString -> IO Response {-# INLINE anyOp #-}-anyOp c req = do+anyOp c = anyOpWith c rejectProtocolError++-- | Like 'anyOp', but rejects a reply whose code is none of the given ones,+-- as 'keyedOpExpecting' does.+anyOpExpecting :: Cluster -> String -> [ByteString] -> ByteString -> IO Response+{-# INLINE anyOpExpecting #-}+anyOpExpecting c operation codes = anyOpWith c (expectReply operation codes)++-- | Run a request against any single server, checking its reply with the+-- given function.+anyOpWith :: Cluster -> (Response -> IO ()) -> ByteString -> IO Response+{-# INLINE anyOpWith #-}+anyOpWith c check req = do servers' <- aliveServers c case V.uncons servers' of Nothing -> throwIO $ ClientError NoServersReady- Just (s, _) -> serverOp c s req+ Just (s, _) -> serverOp c s Resend check req -- | Run a Memcached operation against all servers in the cluster, handling any--- failures that occur, retrying the specified number of times.+-- failures that occur, retrying the specified number of times. As with+-- 'keyedOp', of the replies it can read, only an error reply throws;+-- 'allOpExpecting' also rejects a reply the request cannot get. Also as with+-- 'keyedOp', the request is sent again after a failure, so it must be one+-- that can be repeated. allOp :: Cluster -> ByteString -> IO [(Server, Response)] {-# INLINE allOp #-}-allOp c req = forAliveServers c $ \s -> serverOp c s req+allOp c req = forAliveServers c $ \s -> serverOp c s Resend rejectProtocolError req +-- | Like 'allOp', but rejects a reply whose code is none of the given ones,+-- as 'keyedOpExpecting' does. The servers after one whose reply is rejected+-- are not sent the request, as with a server that fails.+allOpExpecting :: Cluster -> String -> [ByteString] -> ByteString -> IO [(Server, Response)]+{-# INLINE allOpExpecting #-}+allOpExpecting c operation codes req =+ forAliveServers c $ \s -> serverOp c s Resend (expectReply operation codes) req+ -- | Run a Memcached operation against all servers in the cluster, handling any -- failures that occur, retrying the specified number of times. Similar to -- 'anyOp' but allows more flexible interaction with the 'Server' than a single--- request and response.+-- request and response. The operation must make its requests with+-- 'withSocket' on the 'Server' it is given, not on another value for the same+-- server, or its timeouts never mark the server down (see+-- 'optsServerTimeout'). Once one of its 'withSocket' calls has got a+-- connection, a timeout is a server failure, even while a later call waits+-- for one. So is any exception the operation throws other than an+-- 'OpError', a 'BadCommand' or 'UnknownOp' protocol error, or an+-- 'InvalidAuthentication' client error, which are rethrown at once. Each+-- server failure closes the server's idle connections, before the operation+-- is retried or the server marked down. The operation is run again from the+-- start after a failure, whatever requests it had sent, so its requests must+-- be ones that can be repeated. allOp' :: Cluster -> (Server -> IO a) -> IO [(Server, a)] {-# INLINE allOp' #-}-allOp' c op = forAliveServers c $ \s -> retryOp c s (op s)+allOp' c op = forAliveServers c $ \s -> retryOp c s (const op) -- | Run an action against every server not marked as failed, pairing each -- result with its server. Throws 'NoServersReady' if there are none.@@ -289,38 +774,98 @@ res <- V.mapM f servers' return $ V.toList $ V.zip servers' res --- | Run an IO operation multiple times if an exception is thrown, marking the--- server as dead if it fails more than the allowed number of retries.-retryOp :: forall a. Cluster -> Server -> IO a -> IO a+-- | Run an IO operation, retrying it up to the allowed number of retries if an+-- exception is thrown, and marking the server as dead if every attempt fails.+-- Each attempt is given the server with a new @slotTaken@, which 'withSocket'+-- sets once it holds a slot in the server's pool. An attempt that runs out of+-- time before that never reached the server. It is retried like any failure,+-- but if it is the last attempt, it throws v'Timeout' without marking the+-- server dead: the last attempt alone decides, whatever the earlier ones did.+-- Every other failure but a client fault first closes the server's idle+-- connections, before the operation is retried or the server marked dead.+--+-- Each attempt is also given an 'IORef' that it sets just before it sends a+-- request that mustn't be sent twice. A failure after that isn't retried,+-- whatever retries are left: it marks the server dead, as the last attempt's+-- failure does, and is rethrown.+retryOp :: forall a. Cluster -> Server -> (IORef Bool -> Server -> IO a) -> IO a {-# INLINE retryOp #-} retryOp Cluster{..} s op = go cRetries where+ -- An attempt's exception is handled once 'try' has returned it, rather+ -- than in a handler. A handler runs with asynchronous exceptions masked,+ -- and so would the retry it started: nothing could then interrupt the+ -- retry until it blocked, not even its own timeout. This 'try' is+ -- unliftio's, which lets asynchronous exceptions through, so that the+ -- caller can still interrupt the operation. go :: Int -> IO a {-# INLINE go #-}- go !n = handle (handleErrs $ n - 1) $ do- mr <- timeout cTimeout op+ go !n = do+ sent <- newIORef False+ try (attempt sent) >>= either (handleErrs n sent) return++ -- One attempt at the operation, with its own timeout and @slotTaken@.+ attempt :: IORef Bool -> IO a+ attempt sent = do+ taken <- newIORef False+ mr <- timeout cTimeout (op sent s { slotTaken = Just taken }) case mr of Just r -> return r- Nothing -> close s >> throwIO (ClientError Timeout)+ Nothing -> do+ held <- readIORef taken+ if held+ then throwIO (ClientError Timeout)+ else throwIO WaitedForSlot - handleErrs :: Int -> SomeException -> IO a+ handleErrs :: Int -> IORef Bool -> SomeException -> IO a {-# INLINE handleErrs #-}- handleErrs n err+ handleErrs n sent err | clientFault err = throwIO err- | n <= 0 = do- writeIORef (failed s) =<< getPOSIXTime- throwIO err- | otherwise = threadDelay cFailDelay >> go n+ | otherwise = case fromException err of+ Just WaitedForSlot -> retryOr n $ throwIO (ClientError Timeout)+ Nothing -> do+ -- The attempt's own connection, if it had one, is closed+ -- already, unless the operation had given it back before+ -- failing. What made the attempt fail, such as a restart of+ -- the server, may have broken the idle ones too, and the+ -- retry, or the next operation once the server is back,+ -- would take one of those.+ close s+ -- The server may have carried out a request that mustn't be+ -- sent twice, so its failure ends the operation, like the+ -- last attempt's.+ once <- readIORef sent+ let giveUp = do+ writeIORef (failed s) =<< getPOSIXTime+ throwIO err+ if once then giveUp else retryOr n giveUp + -- Retry the operation if it has retries left, or else give up.+ retryOr :: Int -> IO a -> IO a+ {-# INLINE retryOr #-}+ retryOr n giveUp+ | n <= 0 = giveUp+ | otherwise = threadDelay cFailDelay >> go (n - 1)++-- | Thrown by an attempt that runs out of time while it waits for a slot in+-- the server's pool. @retryOp@ retries it like a failure, but without closing+-- the idle connections, and if it is the last attempt, throws v'Timeout'+-- instead, without marking the server dead.+data WaitedForSlot = WaitedForSlot+ deriving Show++instance Exception WaitedForSlot+ -- | A rejected command or a reported operation status is an answer, not a -- server failure: retrying it will fail identically and must not count towards -- marking the server dead. That includes the 'UnknownOp' a new connection -- throws when it refuses the server's version: a retry would find the same--- version.+-- version. Nor is a reply out of step retried or counted: the server has the+-- request, and may have carried it out. clientFault :: SomeException -> Bool clientFault err = case fromException err of Just (ProtocolError BadCommand{}) -> True Just (ProtocolError UnknownOp{}) -> True Just (OpError _) -> True Just (ClientError InvalidAuthentication{}) -> True- _ -> False+ _ -> isJust (fromException err :: Maybe OutOfStep)
Database/Memcache/ElastiCacheClient.hs view
@@ -18,9 +18,9 @@ ) where -import Control.Concurrent (forkIO, threadDelay)+import Control.Concurrent (forkIOWithUnmask, threadDelay) import Control.Concurrent.MVar (MVar, modifyMVar_, newMVar)-import Control.Monad (forever, guard, (<=<))+import Control.Monad (guard, when) import Data.Bifunctor (first) import Data.ByteString (ByteString) import qualified Data.ByteString.Char8 as C@@ -32,15 +32,19 @@ import qualified Data.Vector as V import Data.Version (Version, makeVersion) import qualified Data.Version as Version-import Database.Memcache.Client (Client, optsServerSpecsToServers, optsServerOptions)+import Database.Memcache.Client (Client, optsServerSpecsToServers, optsServerOptions, optsServerTimeout) import qualified Database.Memcache.Client as Client import Database.Memcache.Cluster (Options, getServers, setServers)-import Database.Memcache.Server (Server, withSocket)+import Database.Memcache.Errors (ClientError (Timeout), MemcacheError (ClientError))+import Database.Memcache.Server (Server, closeAll, withSocket) import Database.Memcache.Socket (Connection, crlf, recvRawUntil, sendRaw, versionRequest)-import Database.Memcache.Types.ServerSpec (ServerSpec, parseServerSpec)-import Database.Memcache.Util (note)+import Database.Memcache.Types.Authentication (Authentication (NoAuth))+import Database.Memcache.Types.ServerSpec (ServerSpec (..), parseServerSpec)+import Database.Memcache.Util (note, withDetail)+import Network.Socket (HostName, ServiceName)+import System.Timeout (timeout) import Text.ParserCombinators.ReadP (readP_to_S)-import UnliftIO.Exception (bracket, throwString)+import UnliftIO.Exception (bracket, throwIO, throwString, tryAny, uninterruptibleMask_) -- | Parse an ElastiCache configuration endpoint. --@@ -56,17 +60,34 @@ -- Please see: -- https://github.com/memcached/memcached/blob/77709d04dd4fc7d59f59425802cb9645b2cfe0f8/doc/protocol.txt#L1805-L1812 -- ByteString format: "VERSION <version>\r\n"+--+-- The version only chooses the request for the cluster configuration, and is+-- read leniently, with 'Data.Version.parseVersion': the line's first 8 bytes+-- aren't checked, and a number too large for an 'Int' wraps around. But a+-- version longer than 'maxVersionLength' bytes is an error before it is+-- parsed: 'readP_to_S' gives every partial parse of a dotted version, so the+-- parse takes time that grows faster than the square of the version's+-- length. The endpoint can send a line of up to 1 MiB, and the time limit on+-- discovery can't interrupt the parse while the caller has asynchronous+-- exceptions masked. parseVersion :: ByteString -> Either String Version-parseVersion = interpretParseResults . delegateParse . C.unpack . C.dropEnd (C.length crlf) . C.drop (C.length "VERSION ")+parseVersion line = note ("Unrecognized version reply" `withDetail` C.unpack text) $ do+ let version = C.drop (C.length "VERSION ") text+ guard $ C.length version <= maxVersionLength+ -- The longest parse comes last, and must take the whole version.+ results <- NE.nonEmpty $ readP_to_S Version.parseVersion (C.unpack version)+ case NE.last results of+ (parsed, "") -> pure parsed+ _ -> Nothing where- delegateParse = readP_to_S Version.parseVersion- interpretParseResults parseResults = do- results <- note "Unable to parse input" $ NE.nonEmpty parseResults- let tail' = NE.last results- case tail' of- (version, "") -> pure version- (_, rest) -> Left $ "eof expected. Got: " <> rest+ text = C.dropEnd (C.length crlf) line +-- | The longest version @parseVersion@ reads, in bytes. memcached's versions,+-- such as @1.6.12@, are a few bytes long, so this leaves plenty of room for+-- tags.+maxVersionLength :: Int+maxVersionLength = 64+ -- | A validated ElastiCache configuration endpoint. -- -- Use the smart constructor @parseConfigurationEndpoint@ to create one.@@ -78,11 +99,56 @@ -- | Create a new client using service discovery. -- -- This function discovers all the nodes in a cluster and sets up a new thread to run autodiscovery on an interval.+-- The thread runs with asynchronous exceptions unmasked, even if this is+-- called with them masked, as in the acquire action of a @bracket@. --+-- The first discovery must finish within 'optsServerTimeout' (connecting,+-- which asks for the version, then asking for it again and for the cluster+-- configuration), or this throws v'ClientError' 'Timeout'. If the endpoint+-- sends an error reply, such as the @ERROR@ that plain memcached sends as it+-- has no @config@ command, this throws a @StringException@ from+-- "UnliftIO.Exception", as it does for a reply that can't be parsed, a+-- version of more than 64 bytes, and a configuration that lists no nodes,+-- more than 1,000, an empty node, the same host and port twice, or a node+-- that isn't a plain host and port: one whose host or port is empty, or whose+-- host has a character, such as @\/@, @?@ or @\@@, that would change the host+-- or port, add credentials or make the node a Unix socket on this machine. A+-- host given as an IPv6 address in brackets is rejected too. If a reply goes+-- on for more than 1 MiB without the line end or the @END@ line that+-- discovery is waiting for, this throws+-- v'Database.Memcache.Errors.ProtocolError'+-- 'Database.Memcache.Errors.BadLength'.+--+-- Servers are made by calling 'optsServerSpecsToServers' with one node at a+-- time. A refresh keeps the 'Server', and so the failure state, of every node+-- still listed with the same host and port, and its connections too unless the+-- node is listed with another IP address. It makes servers only for new nodes,+-- and closes the connections of nodes no longer listed: idle ones at once, and+-- one in use as soon as its request finishes. It closes the connections of a+-- node listed with another IP address the same way, as ElastiCache lists a+-- node it has replaced, so that its requests connect again, looking its name+-- up afresh. If the name still resolves to the replaced node's address then,+-- they connect to it, and later refreshes don't close those connections, as+-- the listed address no longer changes.+--+-- Each refresh connects to the endpoint afresh, looking its name up again, and+-- closes that connection when it is done, whether it worked or not: the name+-- leads to one of the nodes, so a connection kept from an earlier refresh+-- could lead to a node that ElastiCache has since replaced, and get no reply.+-- Each refresh has the same time and length limits as the first discovery. A+-- refresh that fails, for example because it runs out of time or the reply is+-- an error, is too long, can't be parsed, lists no nodes, more than 1,000, an+-- empty node or one twice, or lists a node that isn't a plain host and port,+-- leaves the servers as they are, and discovery tries again after the next+-- interval.+-- -- Discovery connects to the configuration endpoint like any other server, so -- it fails for a cluster running a memcached older than 1.6.10, having sent -- only the authentication exchange, when the endpoint has credentials, and--- @version@ (see "Database.Memcache.Client").+-- @version@ (see "Database.Memcache.Client"). Each refresh opens a new+-- connection to the endpoint, so against an endpoint older than that it fails+-- like any other refresh, and the connections the client opens to the nodes+-- it discovers are checked in the same way. -- -- N.B. The interval is in the same units as 'threadDelay'. newClient :: Options -> Int -> ConfigurationEndpoint -> IO Client@@ -91,28 +157,81 @@ acquireCfgClient = Client.newClient [unConfigurationEndpoint cfgEndpoint] options releaseCfgClient = Client.quit resolveCluster' cfgClient = do- serverSpecs <- resolveCluster cfgClient-- -- Create client where cServers = Right <Vector Server>- client <- Client.newClient serverSpecs options+ serverSpecs <- resolveClusterWithin options cfgClient+ (nodes, _, _) <- reuseServers options [] serverSpecs -- Prepare MVar- serversMVar <- newMVar =<< getServers client- _ <- forkIO $ repeatAutoDiscover options interval cfgEndpoint serversMVar+ serversMVar <- newMVar $ nodeServers nodes+ -- Refresh with asynchronous exceptions unmasked, whatever the caller's+ -- masking state. A thread forked inside mask_, as in a bracket's+ -- acquire, would otherwise refresh masked for good, and each refresh's+ -- time limit could interrupt it only where it waits.+ _ <- forkIOWithUnmask $ \unmask ->+ unmask $ repeatAutoDiscover options interval cfgEndpoint serversMVar nodes - -- Update client where cServers = Left <MVar <Vector Server>>- pure $ setServers client $ Left serversMVar+ -- The configuration client was made with the same options, so it only+ -- needs its servers replacing: cServers = Left <MVar <Vector Server>>+ pure $ setServers cfgClient $ Left serversMVar -repeatAutoDiscover :: Options -> Int -> ConfigurationEndpoint -> MVar (V.Vector Server) -> IO ()-repeatAutoDiscover opts interval cfgEndpoint mVar = do+repeatAutoDiscover :: Options -> Int -> ConfigurationEndpoint -> MVar (V.Vector Server) -> Nodes -> IO ()+repeatAutoDiscover opts interval cfgEndpoint mVar initialNodes = do cfgClient <- Client.newClient [unConfigurationEndpoint cfgEndpoint] opts- forever $ do- threadDelay interval- serverSpecs <- resolveCluster cfgClient- servers <- optsServerSpecsToServers opts (optsServerOptions opts) serverSpecs- modifyMVar_ mVar $ \_curServers -> pure $ V.fromList $ sort servers+ let refresh nodes = do+ threadDelay interval+ -- A refresh that fails or does not finish in time keeps the current+ -- servers. 'tryAny' lets asynchronous exceptions, such as+ -- 'killThread', through.+ result <- tryAny $ do+ serverSpecs <- resolveClusterWithin opts cfgClient+ reuseServers opts nodes serverSpecs+ -- Close the connection to the endpoint, whether the refresh worked or+ -- not, so that the next refresh connects again, looking the+ -- endpoint's name up afresh. The name leads to one of the nodes, and+ -- once ElastiCache replaces that node, a connection made before then+ -- may get no reply, so the next refresh would run out of time on it.+ Client.quit cfgClient+ case result of+ Left _ -> refresh nodes+ Right (nodes', gone, moved) -> do+ -- Swap in the new servers, then close the connections of nodes+ -- that have gone, or whose IP address has changed. A request+ -- still using one of them finishes normally, and its connection+ -- is closed then instead of going back to the pool. Mask the swap+ -- and the loop together, so that an asynchronous exception can't+ -- arrive between them, or stop the loop part way, and leave some+ -- of those connections open. Neither waits long: only this thread+ -- takes the 'MVar', and 'closeAll' only ever waits briefly.+ uninterruptibleMask_ $ do+ modifyMVar_ mVar $ \_curServers -> pure $ nodeServers nodes'+ mapM_ closeAll (gone ++ moved)+ refresh nodes'+ refresh initialNodes -resolveCluster :: Client -> IO [ServerSpec]+-- | The nodes found by discovery, by host and port, with the IP address listed+-- for each and the servers made for each.+type Nodes = [((HostName, ServiceName), (Text, [Server]))]++-- | The servers of the given nodes, in the order they were discovered.+nodeServers :: Nodes -> V.Vector Server+nodeServers = V.fromList . concatMap (snd . snd)++-- | Pair each discovered node with its servers from the given nodes, making+-- servers only for nodes not among them. Also returns the servers of the given+-- nodes that were not discovered again, and those of nodes discovered again+-- with another IP address.+reuseServers :: Options -> Nodes -> [(ServerSpec, Text)] -> IO (Nodes, [Server], [Server])+reuseServers _ old [] = pure ([], concatMap (snd . snd) old, [])+reuseServers opts old ((spec, ip) : specs) = do+ let node = (ssHost spec, ssPort spec)+ (servers, moved, rest) <- case break ((== node) . fst) old of+ (before, (_, (oldIp, servers)) : after) -> pure (servers, oldIp /= ip, before ++ after)+ (_, []) -> do+ servers <- optsServerSpecsToServers opts (optsServerOptions opts) [spec]+ pure (servers, False, old)+ (nodes, gone, moved') <- reuseServers opts rest specs+ pure ((node, (ip, servers)) : nodes, gone, if moved then servers ++ moved' else moved')++resolveCluster :: Client -> IO [(ServerSpec, Text)] resolveCluster cfgClient = do servers <- getServers cfgClient @@ -137,6 +256,17 @@ unTry $ handleRawNodeInfo rawNodeInfo +-- Run 'resolveCluster', throwing 'ClientError' 'Timeout' unless it finishes,+-- connecting and both requests included, within the options'+-- 'optsServerTimeout'.+resolveClusterWithin :: Options -> Client -> IO [(ServerSpec, Text)]+resolveClusterWithin opts cfgClient = do+ found <- timeout serverTimeout $ resolveCluster cfgClient+ maybe (throwIO $ ClientError Timeout) pure found+ where+ -- 'optsServerTimeout' is in milliseconds; 'timeout' takes microseconds.+ serverTimeout = truncate (optsServerTimeout opts * 1000)+ -- Handle the raw response from /memcached/. -- -- See: https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.AddingToYourClientLibrary.html#AutoDiscovery.AddingToYourClientLibrary.OutputFormat@@ -151,17 +281,77 @@ -- Handle node information -- -- Note that we chose to use the CNAME rather than the IP. The IP /may/ be missing. The CNAME will always be present.+-- The IP is returned only to notice a node that ElastiCache has replaced. -- -- See: https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.AddingToYourClientLibrary.html#AutoDiscovery.AddingToYourClientLibrary.OutputFormat-handleRawNodeInfo :: Text -> Either String [ServerSpec]-handleRawNodeInfo = mapM (parseServerSpec <=< pure . T.unpack <=< convertToUri) . T.splitOn " "+handleRawNodeInfo :: Text -> Either String [(ServerSpec, Text)]+handleRawNodeInfo nodeInfo = do+ -- A line listing no nodes is empty, as it comes stripped of whitespace, and+ -- would split into one empty entry.+ when (T.null nodeInfo) $+ Left "Configuration lists no nodes"+ -- Count the nodes before parsing any of them.+ let count = T.count " " nodeInfo + 1+ when (count > maxNodes) $+ Left $ "Configuration lists " <> show count <> " nodes, over the limit of " <> show maxNodes+ nodes <- mapM parseEntry (T.splitOn " " nodeInfo)+ -- A node listed twice would get a second server, at the same points on the+ -- ring as the first, so that when the first is marked failed, its keys+ -- would fall back to the same node rather than to another.+ case listedTwice (sort [(ssHost spec, ssPort spec) | (spec, _) <- nodes]) of+ (host, port) : _ -> Left $ "Node listed twice" `withDetail` (host <> ":" <> port)+ [] -> pure nodes where- convertToUri :: Text -> Either String Text- convertToUri t =+ listedTwice sorted = [node | (node, next) <- zip sorted (drop 1 sorted), node == next]++ -- The line comes stripped of whitespace, and isn't empty, so an entry is+ -- empty only where two spaces come in a row.+ parseEntry t | T.null t = Left "Configuration lists an empty node"+ parseEntry t = case T.splitOn "|" t of- [host, _ip, port] -> pure $ "memcached://" <> host <> ":" <> port- xs -> Left $ "Unrecognized components in node info: " <> show xs+ [host, ip, port] -> do+ spec <- parseNode (T.unpack host) (T.unpack port)+ pure (spec, ip)+ _ -> Left $ "Unrecognized components in node info" `withDetail` T.unpack t + -- Node addresses come from the server and hold no credentials, so name+ -- the one that failed.+ parseNode :: HostName -> ServiceName -> Either String ServerSpec+ parseNode host port = first (`withDetail` uri) $ do+ -- A node is a host to reach over the network, on the port listed. The+ -- URI pastes the two together, so a host such as "h/x" or "h?x" would+ -- lose the port, one such as "u:p@h" would give the node credentials+ -- of the endpoint's choosing, and one such as "/var/run/docker.sock?",+ -- whose "?" puts the port in the URI's query, would make the node a+ -- Unix socket on this machine. An empty host would be 127.0.0.1, and+ -- an empty port would fail only at the first request. So the URI must+ -- give back the host and port listed, neither of them empty, and no+ -- credentials.+ --+ -- No plain host or port holds an "@", so a node whose host or port+ -- does is turned down before the URI is parsed, whatever+ -- 'parseServerSpec' would make of the "@": its errors, such as those+ -- about credentials, are for someone who wrote the URI, which no one+ -- did here.+ note notPlain $ guard $ notElem '@' (host <> port)+ spec <- parseServerSpec uri+ note notPlain $+ guard $ not (null host || null port) && spec == ServerSpec host port NoAuth+ pure spec+ where+ uri = "memcached://" <> host <> ":" <> port+ notPlain = "Node is not a plain host and port"++-- | The most nodes a configuration can list. ElastiCache allows 60 nodes in a+-- Memcached cluster unless that quota is raised, so a reply that lists more+-- than this comes from a broken or hostile endpoint. Without a limit, a reply+-- well inside the 1 MiB that discovery reads could list over 100,000 nodes of+-- a few bytes each, and discovery would make a server, with its pool of+-- connections, for each one, and the first request would wait while the ring+-- that routes keys was built, with 160 points for each node.+maxNodes :: Int+maxNodes = 1000+ -- This is the first version within which the /memcache/ engine on /ElastiCache/ uses the custom /config/ command. -- -- https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.AddingToYourClientLibrary.html@@ -173,7 +363,7 @@ resolveViaAutoDiscoveryKey :: Connection -> IO ByteString resolveViaAutoDiscoveryKey socket = do sendRaw socket ("get " <> autoDiscoveryKey <> crlf)- recvRawUntil ("END" <> crlf) socket+ recvConfigReply socket -- Before 'firstCfgCmdVersion', /ElastiCache/ stored configuration in a special-purpose key within the /Configuration Endpoint/. --@@ -186,10 +376,28 @@ resolveViaConfigCmd :: Connection -> IO ByteString resolveViaConfigCmd socket = do sendRaw socket (configGetCluster <> crlf)- recvRawUntil ("END" <> crlf) socket+ recvConfigReply socket configGetCluster :: ByteString configGetCluster = "config get cluster"++-- Receive the reply to the config command or to the get of the cluster key,+-- through the END line that ends it. An error reply is a single line with no+-- END line, and the server keeps the connection open after it, so throw on+-- one instead of waiting for END.+recvConfigReply :: Connection -> IO ByteString+recvConfigReply socket = do+ line <- recvRawUntil crlf socket+ let text = C.dropEnd (C.length crlf) line+ when (C.takeWhile (/= ' ') text `elem` ["ERROR", "CLIENT_ERROR", "SERVER_ERROR"]) $+ throwString $ "Configuration endpoint replied" `withDetail` C.unpack text+ -- Otherwise read through the first END. This line can end with it: when+ -- the cluster key is missing, the reply is the END line alone.+ if end `C.isSuffixOf` line+ then pure line+ else (line <>) <$> recvRawUntil end socket+ where+ end = "END" <> crlf unTry :: Either String a -> IO a unTry = either throwString pure
Database/Memcache/Errors.hs view
@@ -21,10 +21,16 @@ ) where import Database.Memcache.Types+import Database.Memcache.Util (withDetail) import UnliftIO.Exception -- | All exceptions that a Memcached client may throw.+--+-- 'displayException' gives a one-line message for logs and users, such as+-- @memcached operation failed: key not found@. Any detail, such as text the+-- server sent, is cut to its first 200 characters and has characters outside+-- printable ASCII escaped. 'show' gives the derived form. data MemcacheError -- | Memcached operation error. = OpError Status@@ -34,13 +40,31 @@ | ProtocolError ProtocolError deriving (Eq, Show) -instance Exception MemcacheError+instance Exception MemcacheError where+ displayException (OpError s) =+ "memcached operation failed: " <> describeStatus s+ displayException (ClientError e) =+ "memcache client error: " <> describeClientError e+ displayException (ProtocolError e) =+ "memcached protocol error: " <> describeProtocolError e -- | Errors that occur on the client. data ClientError- -- | All servers are currently marked failed.+ -- | No server is available: the cluster has none, every server is marked+ -- failed, or the router ('Database.Memcache.Cluster.optsGetServerForKey')+ -- chose none. = NoServersReady- -- | Timeout occurred sending request to server.+ -- | The attempt that ended an operation ran out of time+ -- ('Database.Memcache.Cluster.optsServerTimeout'): the last attempt, or,+ -- for a request that mustn't be sent twice, the attempt that had started+ -- sending it. The attempt may have been waiting for a free connection to+ -- the server, connecting, authenticating, checking the server's version,+ -- sending a request or reading a reply; the error doesn't say which. An+ -- attempt that only waited for its first connection sent nothing, but for+ -- an operation that is retried after any failure, an earlier attempt may+ -- have sent the request. The server may have carried out any request that+ -- was sent. ElastiCache's 'Database.Memcache.ElastiCacheClient.newClient'+ -- also throws it when its first discovery takes longer than the timeout. | Timeout -- | Credentials were rejected before being sent: they contain a -- forbidden delimiter, or the server is not in auth-file mode.@@ -78,3 +102,36 @@ -- | Convert a status to 'MemcacheError' exception. throwStatus :: Status -> IO a throwStatus s = throwIO $ OpError s++describeStatus :: Status -> String+describeStatus s = case s of+ NoError -> "no error"+ ErrKeyNotFound -> "key not found"+ ErrKeyExists -> "key already exists"+ ErrValueTooLarge -> "value too large"+ ErrItemNotStored -> "item not stored"+ ErrValueNonNumeric -> "value is not numeric"+ ErrOutOfMemory -> "server out of memory"+ TextAuthFail -> "authentication failed"++describeClientError :: ClientError -> String+describeClientError e = case e of+ NoServersReady -> "no servers ready"+ Timeout -> "request timed out"+ InvalidAuthentication m -> "invalid authentication" `withDetail` m+ KeyTooLong m -> "invalid key or argument" `withDetail` m++-- The library throws 'UnknownPkt' with no detail for an empty response line.+describeProtocolError :: ProtocolError -> String+describeProtocolError e = problem `withDetail` protocolError e+ where+ problem = case e of+ UnknownPkt "" -> "empty response line"+ UnknownPkt _ -> "unrecognized response line"+ UnknownOp _ -> "command not recognized by server"+ UnknownStatus _ -> "unknown response code"+ BadLength _ -> "malformed response"+ UnexpectedResponse _ -> "unexpected response"+ ServerError _ -> "server error"+ BadCommand _ -> "command rejected by server"+ UnexpectedEOF _ -> "connection closed by server"
+ Database/Memcache/Ketama.hs view
@@ -0,0 +1,185 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Module : Database.Memcache.Ketama+Description : libmemcached-compatible consistent hashing+Copyright : (c) David Terei, 2026+License : BSD+Maintainer : code@davidterei.com+Stability : stable+Portability : GHC++Ketama consistent hashing, computed exactly as libmemcached does for+@MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED@. The cluster uses this to choose the+server for each key. It is exposed for working out which server a key belongs+to, and for building rings with weights, which the cluster does not use.++A ring has 160 points per server on a 32-bit circle, shared out in proportion+to the servers' weights. The points of a server named @n@ come from the MD5+digests of @n-0@, @n-1@, and so on, each digest giving four little-endian+32-bit points. A key hashes to the first four bytes of its MD5 digest, read+little-endian, and belongs to the server owning the first point at or after+that hash, wrapping round to the lowest point.++With equal weights, adding or removing a server only moves the keys on that+server's points, unless it changes the number of points each server gets+(from 24 servers to 25, say). With unequal weights, it changes the other+servers' numbers of points too, so some of their keys move between them.++The details follow libmemcached so that both choose the same server:++* The number of points is computed in single precision, which gives 156+ rather than 160 points per server for some numbers of equally weighted+ servers, such as 25.++* Points with the same value are ordered by server position, so the earlier+ server in the list wins the tie, as in libmemcached 1.1 and later.++* A server's name is its host alone on port 11211 and @host:port@ on any other+ port (see 'serverName').+-}+module Database.Memcache.Ketama (+ -- * Rings+ Ring, ring, serverName,++ -- * Lookup+ keyHash, owner, owners,++ -- * Inspection+ ringPoints+ ) where++import Control.Monad.ST (runST)+import qualified Crypto.Hash.MD5 as MD5+import Data.Bits (shiftL, (.|.))+import Data.ByteString (ByteString)+import qualified Data.ByteString as B+import qualified Data.ByteString.Char8 as C+import Data.List (sort)+import qualified Data.Vector.Unboxed as VU+import qualified Data.Vector.Unboxed.Mutable as VUM+import Data.Word (Word32)++-- | A ketama ring (libmemcached calls it the continuum) over a list of+-- servers. Servers are identified by their position in the list the ring was+-- built from, starting at 0.+data Ring = Ring {+ -- | Point values, in ascending order.+ rValues :: !(VU.Vector Word32),+ -- | The position of the server owning each point.+ rOwners :: !(VU.Vector Int),+ -- | How many servers the ring was built from.+ rSize :: !Int,+ -- | How many servers have at least one point.+ rServers :: !Int+ }++-- | Build a ring from each server's name (see 'serverName') and weight, in+-- the order that decides ties. Weights below 1 count as 1, as libmemcached+-- treats a weight of 0. Give every server the same weight to match clients+-- that are not configured with weights.+ring :: [(ByteString, Int)] -> Ring+ring servers = Ring values owners' (length servers) (length (filter (> 0) counts))+ where+ weights = map (max 1 . snd) servers+ total = sum weights+ counts = [pointsPerServer w total (length servers) | w <- weights]+ (values, owners') = VU.unzip $ VU.fromList $ sort+ [ (point, index)+ | (index, (name, _), count) <- zip3 [0 ..] servers counts+ , i <- [0 .. count `quot` 4 - 1]+ , point <- digestPoints (name <> "-" <> C.pack (show (i :: Int)))+ ]++-- | The name libmemcached derives a server's points from, given its host and+-- port as written in its configuration: the host alone for the default port+-- 11211, otherwise @host:port@. Hosts are not resolved, so @localhost@ and+-- @127.0.0.1@ are different servers on the ring.+serverName :: ByteString -> ByteString -> ByteString+serverName host port+ | port == "11211" = host+ | otherwise = host <> ":" <> port++-- | The number of points a server gets: libmemcached's+-- @floor(weight / total * 160 / 4 * servers + 1e-10) * 4@, evaluated left to+-- right in single precision as the C code does.+pointsPerServer :: Int -> Int -> Int -> Int+pointsPerServer weight total servers =+ 4 * floor (share * 160 / 4 * fromIntegral servers + 1.0e-10 :: Float)+ where+ share = fromIntegral weight / fromIntegral total :: Float++-- | The four points of a name: its MD5 digest read as four little-endian+-- 32-bit words.+digestPoints :: ByteString -> [Word32]+digestPoints name = [littleEndian digest offset | offset <- [0, 4, 8, 12]]+ where+ digest = MD5.hash name++-- | The position of a key on the ring: the first four bytes of the MD5 digest+-- of the key, read little-endian. This is libmemcached's MD5 key hash; it is+-- taken over the key as given, not its base64 form on the wire.+keyHash :: ByteString -> Word32+keyHash key = littleEndian (MD5.hash key) 0++littleEndian :: ByteString -> Int -> Word32+littleEndian bytes offset =+ byte 0 .|. byte 1 `shiftL` 8 .|. byte 2 `shiftL` 16 .|. byte 3 `shiftL` 24+ where+ byte i = fromIntegral (B.index bytes (offset + i))++-- | The server owning a hash: the owner of the first point at or after it,+-- wrapping round to the lowest point. 'Nothing' only for a ring with no+-- servers.+owner :: Ring -> Word32 -> Maybe Int+owner r hash+ | VU.null (rOwners r) = Nothing+ | otherwise = Just (VU.unsafeIndex (rOwners r) (firstPoint r hash))++-- | Every server on the ring, in the order a key with this hash falls back to+-- them: the 'owner' first, then each other server in the order its first point+-- comes walking forward round the ring from the hash. Keeping the ring and+-- skipping servers that are down this way leaves every other key where it was.+-- Taking only the head costs no more than 'owner'; the rest of the list takes+-- one walk round the ring, which stops once every server has been found.+owners :: Ring -> Word32 -> [Int]+owners r hash+ | n == 0 = []+ | otherwise = first : others+ where+ points = rOwners r+ n = VU.length points+ start = firstPoint r hash+ first = VU.unsafeIndex points start+ others = runST $ do+ seen <- VUM.replicate (rSize r) False+ VUM.unsafeWrite seen first True+ let walk i found acc+ | found == rServers r || i == start + n = return (reverse acc)+ | otherwise = do+ let server = VU.unsafeIndex points (if i < n then i else i - n)+ known <- VUM.unsafeRead seen server+ if known+ then walk (i + 1) found acc+ else do+ VUM.unsafeWrite seen server True+ walk (i + 1) (found + 1) (server : acc)+ walk (start + 1) (1 :: Int) []++-- | The index of the first point at or after the hash, or 0 if there is none.+firstPoint :: Ring -> Word32 -> Int+firstPoint r hash = if found == VU.length values then 0 else found+ where+ values = rValues r+ found = search 0 (VU.length values)+ search lo hi+ | lo >= hi = lo+ | VU.unsafeIndex values mid < hash = search (mid + 1) hi+ | otherwise = search lo mid+ where+ mid = (lo + hi) `quot` 2++-- | Every point on the ring in ascending order, with the position of the+-- server that owns it.+ringPoints :: Ring -> [(Word32, Int)]+ringPoints r = VU.toList (VU.zip (rValues r) (rOwners r))
Database/Memcache/Server.hs view
@@ -26,15 +26,17 @@ -} module Database.Memcache.Server ( -- * Server- Server(sid, failed), newServerDefault, withSocket, close,+ Server(sid, failed), serverHost, serverPort, newServerDefault,+ withSocket, close, closeAll, -- * ServerOptions- ServerOptions(..)+ ServerOptions(..), defaultServerOptions ) where import Database.Memcache.Auth import Database.Memcache.Errors import Database.Memcache.Socket+import Database.Memcache.Types.Server import Control.Monad (void, when) import qualified Data.ByteString.Char8 as C@@ -44,7 +46,6 @@ import Data.IORef import qualified Data.Pool as P import Data.Pool (Pool)-import Data.Time.Clock.POSIX (POSIXTime) import Database.Memcache.Types (ServerSpec (..)) import UnliftIO.Exception @@ -55,31 +56,14 @@ import Data.Time.Clock (NominalDiffTime) #endif --- | Memcached server connection.-data Server = Server {- -- | ID of server for consistent hashing.- sid :: {-# UNPACK #-} !Int,- -- | Connection pool to server.- pool :: Pool Connection,- -- | Hostname of server.- addr :: !HostName,- -- | Port number of server.- port :: !ServiceName,- -- | Credentials for server.- auth :: !Authentication,- -- | When did the server fail? 0 if it is alive.- failed :: IORef POSIXTime- }--instance Show Server where- show Server{..} =- "Server [" ++ show sid ++ "] " ++ addr ++ ":" ++ show port--instance Eq Server where- (==) x y = sid x == sid y+-- | The host the server was configured with, as given in its 'ServerSpec'.+serverHost :: Server -> HostName+serverHost = addr -instance Ord Server where- compare x y = compare (sid x) (sid y)+-- | The port or service name the server was configured with, as given in its+-- 'ServerSpec'.+serverPort :: Server -> ServiceName+serverPort = port -- | Configurable options when creating a @Server@. --@@ -87,9 +71,19 @@ -- data ServerOptions = ServerOptions- -- | Maximum number of pooled connections per stripe.+ -- | Maximum number of pooled connections per stripe. With resource-pool 0.3,+ -- see 'soNumStripes'. With 0.4 or later, it is split between the stripes+ -- instead, and must be at least 'soNumStripes', or creating the server (and+ -- so a client) throws an @ErrorCall@. Each stripe then holds up to+ -- 'soNumResources' divided by 'soNumStripes', rounded up with 0.4, and+ -- rounded down or up with 0.5 so that the total is 'soNumResources'. { soNumResources :: Int- -- | Number of pool stripes.+ -- | Number of pool stripes. With resource-pool 0.3, a pool has a stripe for+ -- each capability instead, and each stripe holds up to 'soNumResources'+ -- times 'soNumStripes' connections divided by the number of capabilities,+ -- rounded up. A thread then takes connections from its own capability's+ -- stripe, so a request can wait for one while other stripes have room. With+ -- 0.4 and 0.5.0.0, stripes beyond the number of capabilities are never used. , soNumStripes :: Int #if MIN_VERSION_resource_pool(0,4,0) -- | Connection keep-alive duration.@@ -100,18 +94,31 @@ #endif } -instance Default ServerOptions where- def = ServerOptions+-- | The default server options:+--+-- * 'soNumResources': 1+-- * 'soNumStripes': 1+-- * 'soKeepAlive': 300 seconds+--+-- That is, at most one connection to each server (with resource-pool 0.3, one+-- for each capability), closed once it has been idle for 300 seconds.+defaultServerOptions :: ServerOptions+defaultServerOptions = ServerOptions { soNumResources = 1 , soNumStripes = 1 , soKeepAlive = 300 } +-- | 'def' is 'defaultServerOptions'.+instance Default ServerOptions where+ def = defaultServerOptions+ -- | Create a server using the default socket-pool implementation. newServerDefault :: ServerOptions -> ServerSpec -> IO Server-newServerDefault serverOptions ss@ServerSpec{..} = do+newServerDefault serverOptions ServerSpec{..} = do fat <- newIORef 0- pSock <- getNewPool serverOptions ss+ gen <- newIORef 0+ pSock <- getNewPool serverOptions return Server { sid = serverHash , pool = pSock@@ -119,46 +126,126 @@ , port = ssPort , auth = ssAuth , failed = fat+ , generation = gen+ , slotTaken = Nothing } where serverHash = hash (ssHost, ssPort) --- | Run a function with access to a pooled server connection.+-- | Run a function with access to a pooled server connection. Afterwards the+-- connection goes back to the pool, unless the function threw or 'closeAll'+-- has run since the connection was opened, in which case it is closed. The+-- function is never given a connection opened before the latest 'closeAll'. withSocket :: Server -> (Connection -> IO a) -> IO a {-# INLINE withSocket #-}-withSocket svr = P.withResource $ pool svr+withSocket svr act = mask $ \restore -> do+ (slot, local) <- P.takeResource (pool svr)+ -- Record that the request has stopped waiting (see 'slotTaken').+ mapM_ (`writeIORef` True) (slotTaken svr)+ (born, conn) <- connectionIn slot `onException` P.destroyResource (pool svr) local slot+ r <- restore (act conn) `onException` P.destroyResource (pool svr) local slot+ stale <- (/= born) <$> currentGeneration svr+ if stale+ then P.destroyResource (pool svr) local slot+ else do+ P.putResource local slot+ -- 'closeAll' may have run between the check and the put, and closed+ -- the idle connections before this one was back. If so, close them+ -- again, without letting an exception cut that short.+ raced <- (/= born) <$> currentGeneration svr+ when raced $ uninterruptibleMask_ $ P.destroyAllResources (pool svr)+ return r+ where+ -- The slot's connection, opening one if the slot has none or closing and+ -- replacing one opened before the latest 'closeAll'. 'closeAll' may run+ -- while a connection is being opened, so check the new one too.+ connectionIn slot = do+ held <- readIORef slot+ current <- currentGeneration svr+ case held of+ Just res@(born, _) | born == current -> return res+ _ -> do+ releaseSlot slot+ writeIORef slot Nothing+ res <- connectTagged (generation svr) (ServerSpec (addr svr) (port svr) (auth svr))+ writeIORef slot (Just res)+ connectionIn slot --- | Close all pooled connections. A later operation re-establishes them.+-- | Close the connections that are idle in the pool. Connections in use stay+-- open and go back to the pool when their request finishes; 'closeAll' closes+-- those too. A later request re-establishes connections as needed. An+-- asynchronous exception can't cut 'close' short: it is delivered once every+-- idle connection is closed. close :: Server -> IO () {-# INLINE close #-}-close srv = P.destroyAllResources $ pool srv+close srv = uninterruptibleMask_ $ P.destroyAllResources $ pool srv +-- | Close every connection to the server: idle connections now, and each+-- connection in use as soon as the request using it finishes, instead of+-- returning it to the pool. It doesn't wait for those requests or interrupt+-- them, and it never connects or throws. Requests that start later never use+-- a connection opened before 'closeAll'; they open new ones. Like 'close', it+-- can't be cut short by an asynchronous exception.+closeAll :: Server -> IO ()+closeAll srv = uninterruptibleMask_ $ do+ atomicModifyIORef' (generation srv) (\g -> (g + 1, ()))+ P.destroyAllResources (pool srv)++-- | Read the generation. The read is atomic for simplicity; what makes the+-- checks in 'withSocket' sound is that it and 'closeAll' both go through the+-- pool's per-stripe lock, which orders the bump against a returned connection.+currentGeneration :: Server -> IO Int+currentGeneration svr = atomicModifyIORef' (generation svr) (\g -> (g, g))++-- Slots start empty: 'withSocket' opens a connection in a slot once it holds+-- the slot, so that connecting isn't part of waiting for one. #if MIN_VERSION_resource_pool(0,4,0)-getNewPool :: ServerOptions -> ServerSpec -> IO (Pool Connection)-getNewPool serverOptions ss =+getNewPool :: ServerOptions -> IO (Pool Slot)+getNewPool serverOptions = P.newPool $ P.setNumStripes (Just $ soNumStripes serverOptions)- $ P.defaultPoolConfig (connectSocket ss) releaseSocket (soKeepAlive serverOptions) (soNumResources serverOptions)+ $ P.defaultPoolConfig (newIORef Nothing) releaseSlot (soKeepAlive serverOptions) (soNumResources serverOptions) #else-getNewPool :: ServerOptions -> ServerSpec -> IO (Pool Connection)-getNewPool serverOptions ss =- P.createPool (connectSocket ss) releaseSocket (soNumStripes serverOptions) (soKeepAlive serverOptions) (soNumResources serverOptions)+getNewPool :: ServerOptions -> IO (Pool Slot)+getNewPool serverOptions =+ P.createPool (newIORef Nothing) releaseSlot (soNumStripes serverOptions) (soKeepAlive serverOptions) (soNumResources serverOptions) #endif +-- | Close the slot's connection, if it has one.+releaseSlot :: Slot -> IO ()+releaseSlot slot = readIORef slot >>= mapM_ (releaseSocket . snd)++-- | Open a connection, paired with the generation current when it was opened.+connectTagged :: IORef Int -> ServerSpec -> IO (Int, Connection)+connectTagged gen ss = do+ born <- atomicModifyIORef' gen (\g -> (g, g))+ conn <- connectSocket ss+ return (born, conn)+ connectSocket :: ServerSpec -> IO Connection connectSocket ServerSpec{..} = do let hints = S.defaultHints { S.addrSocketType = S.Stream }- addr:_ <- getAddrInfo (Just hints) (Just ssHost) (Just ssPort)+ -- A path is a Unix socket: there is nothing to look up, so it is the+ -- only address to try.+ addr:addrs <- case ssHost of+ '/' : _ -> return [S.defaultHints {+ S.addrFamily = S.AF_UNIX,+ S.addrSocketType = S.Stream,+ S.addrAddress = S.SockAddrUnix ssHost+ }]+ _ -> getAddrInfo (Just hints) (Just ssHost) (Just ssPort) bracketOnError- (S.socket (S.addrFamily addr) (S.addrSocketType addr) (S.addrProtocol addr))+ (connectFirst addr addrs) S.close (\s -> do- S.connect s $ S.addrAddress addr- S.setSocketOption s S.KeepAlive 1- S.setSocketOption s S.NoDelay 1+ -- A Unix socket rejects NoDelay, and has no TCP connection to+ -- keep alive.+ when (S.addrFamily addr /= S.AF_UNIX) $ do+ S.setSocketOption s S.KeepAlive 1+ S.setSocketOption s S.NoDelay 1 buffered <- newConnection s authenticate buffered ssAuth checkVersion buffered@@ -174,8 +261,9 @@ -- | Ask the server for its version, and throw unless it is at least -- @minimumVersion@. Every new connection does this, after authenticating when--- there are credentials, before it is used for anything else; @connectSocket@--- closes the connection when this throws.+-- there are credentials, before it is used for anything else, whichever of+-- the host's addresses it connected to, or through a Unix socket;+-- @connectSocket@ closes the connection when this throws. -- -- An error reply, such as @CLIENT_ERROR unauthenticated@ from a server in -- auth-file mode when no credentials are set, throws what it throws as the@@ -215,6 +303,30 @@ (digits, remainder) | not (C.null digits) -> Just (read (C.unpack digits), remainder) _ -> Nothing responseVersion _ = Nothing++-- | Connect a socket to the first of the resolved addresses that accepts,+-- trying them in order, so a host whose first address refuses the connection+-- (such as ::1 when the server listens on 127.0.0.1 only) can still be reached+-- through the others. If none accepts, rethrows the first address's error:+-- the resolver sorts addresses this host cannot use, such as IPv6 where it is+-- disabled, last, so the later errors are often just about those.+--+-- Only an 'IOException' from creating or connecting a socket moves on to the+-- next address, so the attempt's timeout ends the loop. Errors once+-- connected, such as rejected credentials, are raised after this returns:+-- they come from the server, and trying another address would hide them.+connectFirst :: S.AddrInfo -> [S.AddrInfo] -> IO S.Socket+connectFirst addr addrs = do+ result <- tryIO $ connectTo addr+ case result of+ Right s -> return s+ Left err -> foldr orElse (throwIO err) addrs+ where+ orElse next rest = tryIO (connectTo next) >>= either (const rest) return+ connectTo a = bracketOnError+ (S.socket (S.addrFamily a) (S.addrSocketType a) (S.addrProtocol a))+ S.close+ (\s -> S.connect s (S.addrAddress a) >> return s) releaseSocket :: Connection -> IO () releaseSocket = closeConnection
Database/Memcache/Socket.hs view
@@ -1,3 +1,5 @@+{-# LANGUAGE CApiFFI #-}+{-# LANGUAGE CPP #-} {-# LANGUAGE OverloadedStrings #-} {-|@@ -18,7 +20,7 @@ crlf, -- * Operations- newConnection, closeConnection,+ newConnection, closeConnection, checkIdle, sendRaw, recvExact, recvLine, recvRawUntil, recvResponse, @@ -27,9 +29,9 @@ -- * Constructors encodeKey, getRequest, getManyRequest, gatRequest, touchRequest,- storeRequest, arithmeticRequest, deleteRequest, flushRequest,- versionRequest, modifyRequest, statsRequest, quitRequest, authRequest,- noOpRequest+ storeRequest, setManyRequest, arithmeticRequest, deleteRequest,+ deleteManyRequest, flushRequest, versionRequest, modifyRequest,+ statsRequest, quitRequest, authRequest, noOpRequest ) where import Database.Memcache.Errors@@ -44,14 +46,22 @@ untrimmedStrategy) import qualified Data.ByteString.Char8 as C import qualified Data.ByteString.Lazy as BL+import Data.Char (toLower) import Data.IORef import Data.List (find, isInfixOf) import Data.Maybe (fromMaybe) import Data.Word (Word64) import qualified Network.Socket as N import qualified Network.Socket.ByteString as N-import Text.Read (readMaybe) import UnliftIO.Exception (throwIO)+#if !defined(mingw32_HOST_OS)+import Data.Bits ((.|.))+import Foreign.C.Error (eAGAIN, eINTR, eWOULDBLOCK, errnoToIOError, getErrno)+import Foreign.C.Types (CChar, CInt (..), CSize (..))+import Foreign.Marshal.Alloc (allocaBytes)+import Foreign.Ptr (Ptr)+import System.Posix.Types (CSsize (..))+#endif -- | A connected Memcached socket and its receive buffer. data Connection = Connection N.Socket (IORef ByteString)@@ -60,7 +70,9 @@ data Response = Response -- | The response code, such as @VA@ or @HD@. { responseCode :: ByteString- -- | Tokens following the response code.+ -- | Tokens following the response code. As in memcached, only spaces+ -- separate tokens, so a token can contain other whitespace bytes, such as+ -- a tab or 0xa0. , responseTokens :: [ByteString] -- | The response value, when the response contains one. , responseValue :: ByteString@@ -71,7 +83,8 @@ -- This doesn't check the server's version. Connections that -- "Database.Memcache.Server" makes are checked, and refuse a server older than -- memcached 1.6.10; a connection made here directly is not, so check the--- server's version before sending it 'storeRequest' or 'modifyRequest'.+-- server's version before sending it 'storeRequest', 'setManyRequest' or+-- 'modifyRequest'. newConnection :: N.Socket -> IO Connection newConnection s = Connection s <$> newIORef B.empty @@ -79,6 +92,58 @@ closeConnection :: Connection -> IO () closeConnection (Connection s _) = N.close s +-- | Check, without waiting, that the connection is as a finished exchange+-- leaves it: everything the server sent has been read, and the server hasn't+-- closed the connection. Throws as reading from it would, if the server has+-- closed it ('UnexpectedEOF') or reset it (an+-- t'Control.Exception.IOException'), and throws 'UnexpectedResponse' if bytes+-- are waiting to be read, which would be taken as the reply to the next+-- request.+--+-- Use it on a connection taken from a pool before sending a request that+-- mustn't be sent twice. memcached closes idle connections when it restarts+-- or after @-o idle_timeout@, and a request sent on one of those never+-- reaches it but fails as if it might have, with 'UnexpectedEOF'. It costs a+-- system call, and it can't see a connection that the network dropped+-- without telling either end, or one the server closes after the check.+-- On Windows it only checks the bytes already read from the socket.+checkIdle :: Connection -> IO ()+checkIdle (Connection s ref) = do+ buffered <- readIORef ref+ waiting <- if B.null buffered then peekSocket s else return (Just (B.length buffered))+ case waiting of+ Nothing -> return ()+ Just 0 -> throwIO eofError+ Just _ -> throwIO $ ProtocolError UnexpectedResponse+ { protocolError = "Expected no data on an idle connection" }++-- | Look, without waiting or taking anything, at what the socket has received:+-- 'Nothing' if nothing, @Just 0@ if the peer has closed the connection, and+-- otherwise how many bytes are waiting (here at most 1). Throws the error of a+-- connection that was reset.+peekSocket :: N.Socket -> IO (Maybe Int)+#if defined(mingw32_HOST_OS)+-- Windows has no MSG_DONTWAIT, so there is no look that can't wait.+peekSocket _ = return Nothing+#else+peekSocket s = N.withFdSocket s $ \fd -> allocaBytes 1 $ \buffer -> peekInto fd buffer+ where+ peekInto fd buffer = do+ n <- c_recv fd buffer 1 (msgPeek .|. msgDontWait)+ if n >= 0 then return (Just (fromIntegral n)) else getErrno >>= failed fd buffer+ failed fd buffer errno+ | errno == eINTR = peekInto fd buffer+ | errno == eAGAIN || errno == eWOULDBLOCK = return Nothing+ | otherwise = throwIO (errnoToIOError "recv" errno Nothing Nothing)++foreign import ccall unsafe "recv"+ c_recv :: CInt -> Ptr CChar -> CSize -> CInt -> IO CSsize++foreign import capi "sys/socket.h value MSG_PEEK" msgPeek :: CInt++foreign import capi "sys/socket.h value MSG_DONTWAIT" msgDontWait :: CInt+#endif+ -- | Send raw bytes on the underlying socket. sendRaw :: Connection -> ByteString -> IO () sendRaw (Connection s _) = N.sendAll s@@ -87,7 +152,7 @@ recvResponse :: Connection -> IO Response recvResponse s = do line <- recvLine s- case C.words line of+ case splitTokens line of [] -> throwIO $ ProtocolError UnknownPkt { protocolError = "" } code:tokens -> case tokens of n:_ | code == "VA" -> do@@ -96,12 +161,20 @@ unless (trailer == "\r") $ throwIO $ ProtocolError BadLength { protocolError = "value is not followed by an empty line" } return $ Response code tokens value+ [] | code == "VA" -> throwIO $ ProtocolError BadLength { protocolError = "invalid value length" } _ -> return $ Response code tokens B.empty where- parseLength x = case readMaybe (C.unpack x) of- Just n | n >= 0 && n <= 128 * 1024 * 1024 -> return n+ parseLength x = case readDecimal x of+ Just n | n <= 128 * 1024 * 1024 -> return n _ -> throwIO $ ProtocolError BadLength { protocolError = "invalid value length" } + -- Split the line as memcached does: on spaces alone, after removing the+ -- CR that 'recvLine' leaves on it. 'C.words' would also split on tabs+ -- and on byte 0xa0, which a key the server echoes can contain: it is+ -- part of many UTF-8 characters, such as U+00E0 (C3 A0).+ splitTokens line =+ filter (not . B.null) (C.split ' ' (fromMaybe line (B.stripSuffix "\r" line)))+ -- | Convert a response code and tokens to the corresponding status. responseStatus :: Response -> IO Status responseStatus (Response code tokens _) = case code of@@ -127,10 +200,12 @@ -- Meta Text reports these as free-form error text where the binary -- protocol had dedicated status codes; match them back so callers keep- -- seeing 'OpError' rather than an opaque protocol failure.+ -- seeing 'OpError' rather than an opaque protocol failure. memcached+ -- capitalises some of its out-of-memory errors, such as "Out of memory+ -- during read", so that match ignores case. serverFailure | "object too large" `isInfixOf` message = throwStatus ErrValueTooLarge- | "out of memory" `isInfixOf` message = throwStatus ErrOutOfMemory+ | "out of memory" `isInfixOf` map toLower message = throwStatus ErrOutOfMemory | otherwise = throwIO $ ProtocolError ServerError { protocolError = message } clientFailure@@ -139,21 +214,39 @@ -- | Extract the CAS version from a response, or zero when absent or invalid. responseCas :: Response -> Version-responseCas = token64 "c" . responseTokens+responseCas = tokenNumber "c" . responseTokens -- | Extract the flags from a response, or zero when absent or invalid. responseFlags :: Response -> Flags-responseFlags = fromIntegral . token64 "f" . responseTokens+responseFlags = tokenNumber "f" . responseTokens -- | Parse the numeric value in a response, or zero when it is absent or invalid. responseNumber :: Response -> Word64-responseNumber = fromMaybe 0 . readMaybe . C.unpack . responseValue+responseNumber = fromMaybe 0 . readDecimal . responseValue -token64 :: ByteString -> [ByteString] -> Word64-token64 prefix xs = fromMaybe 0 $ do+tokenNumber :: (Integral a, Bounded a) => ByteString -> [ByteString] -> a+tokenNumber prefix xs = fromMaybe 0 $ do token <- find (B.isPrefixOf prefix) xs- readMaybe (C.unpack (B.drop (B.length prefix) token))+ readDecimal (B.drop (B.length prefix) token) +-- | Read a number from a reply. Memcached writes value lengths, CAS values,+-- flags and counter values as plain decimal digits, so accept exactly that: at+-- least one digit and nothing else, not even a sign or a space. A number too+-- large for the type is rejected rather than wrapped around.+readDecimal :: (Integral a, Bounded a) => ByteString -> Maybe a+readDecimal input+ | B.null input = Nothing+ | otherwise = go 0 input+ where+ go !n rest = case B.uncons rest of+ Nothing -> Just n+ Just (byte, rest')+ | byte < 0x30 || byte > 0x39 -> Nothing+ | n > limit || (n == limit && digit > lastDigit) -> Nothing+ | otherwise -> go (n * 10 + digit) rest'+ where digit = fromIntegral (byte - 0x30)+ (limit, lastDigit) = maxBound `quotRem` 10+ -- | Longest response line accepted before assuming the stream is desynchronised. maxLineLength :: Int maxLineLength = 8 * 1024@@ -183,7 +276,8 @@ then writeBuffer s (B.drop takeN buffer) >> go (wanted - takeN) (B.take takeN buffer : acc) else readChunk wanted s >> go wanted acc --- | Receive through the first occurrence of a delimiter, including it.+-- | Receive through the first occurrence of a delimiter, including it. Throws+-- 'BadLength' once it has read more than 1 MiB without the delimiter. recvRawUntil :: ByteString -> Connection -> IO ByteString recvRawUntil end s = go B.empty where@@ -194,8 +288,22 @@ (before, rest) | not (B.null rest) -> do writeBuffer s (B.drop (B.length end) rest) return (before <> end)- _ -> writeBuffer s B.empty >> readChunk 0 s >> go combined+ _ | B.length combined > maxRawLength ->+ throwIO $ ProtocolError BadLength { protocolError = "response too long" }+ | otherwise -> writeBuffer s B.empty >> readChunk 0 s >> go combined +-- | 'recvRawUntil' gives up once it has read more than this without its+-- delimiter. ElastiCache discovery reads its cluster configuration with it,+-- which lists each node as a host name (at most 253 bytes), an IP address and+-- a port: at most about 300 bytes a node, so this holds over 3,000 nodes.+-- ElastiCache allows 60 nodes in a Memcached cluster unless that quota is+-- raised. A node can take as little as a few bytes, though, so discovery also+-- refuses a configuration that lists more than 1,000 nodes (@maxNodes@ in+-- "Database.Memcache.ElastiCacheClient"): that limit, not this one, bounds+-- the servers it makes.+maxRawLength :: Int+maxRawLength = 1024 * 1024+ readChunk :: Int -> Connection -> IO () readChunk n (Connection s ref) = do chunk <- N.recv s (max 4096 (min maxRecvSize n))@@ -329,9 +437,14 @@ getRequest :: WireKey -> ByteString getRequest key = cmdMetaGet <> " " <> keyArgs key <> " v f c" <> crlf --- | Encode a quiet multi-get request for one key.-getManyRequest :: WireKey -> ByteString-getManyRequest key = cmdMetaGet <> " " <> keyArgs key <> " v f c k q" <> crlf+-- | Encode a get request for a pipelined batch. It is 'getRequest' plus the+-- @q@ flag, so a miss gets no reply, and the given opaque as the @O@ flag,+-- which memcached copies into the reply so that it can be matched to its+-- request.+getManyRequest :: WireKey -> Int -> ByteString+getManyRequest key opaque = render (B.length (wireKey key)) $+ bytes cmdMetaGet <> bytes " " <> bytes (keyArgs key) <> bytes " v f c q O" <> intDec opaque+ <> bytes crlf -- | Encode a get-and-touch request. gatRequest :: WireKey -> Expiration -> ByteString@@ -352,10 +465,10 @@ -- that "Database.Memcache.Server" makes check the server's version, but a -- connection made with 'newConnection' directly does not. ----- Nor send a value longer than 1 GiB, the largest item memcached can store:--- memcached mishandles a data length over 2147483645 bytes, and then runs the--- rest of the value's bytes as commands. The operations in--- "Database.Memcache.Client" refuse such a value; this encoder doesn't.+-- Nor send it with a value longer than 1 GiB, the largest item memcached can+-- store: from 2147483646 bytes, every memcached release up to 1.6.45 runs the+-- value's bytes as commands. The operations in "Database.Memcache.Client" check+-- the length before sending anything, but this encodes a value of any length. storeRequest :: StoreMode -> WireKey -> Value -> Flags -> Expiration -> Version -> ByteString storeRequest mode key value flags expiry version = render (B.length (wireKey key) + B.length value) $ bytes cmdMetaSet <> bytes " " <> bytes (wireKey key) <> bytes " " <> intDec (B.length value)@@ -363,6 +476,26 @@ <> bytes " M" <> bytes (storeMode mode) <> bytes (keyFlag key) <> bytes " c" <> bytes crlf <> byteStringCopy value <> bytes crlf +-- | Encode a set request for a pipelined batch. It is 'storeRequest' in 'Set'+-- mode without a CAS, plus the given opaque as the @O@ flag, which memcached+-- copies into the reply so that it can be matched to its request.+--+-- Send this only to memcached 1.6.10 or later. Older servers reject it without+-- reading the value, and then run the value's bytes as commands. Connections+-- that "Database.Memcache.Server" makes check the server's version, but a+-- connection made with 'newConnection' directly does not.+--+-- Nor send it with a value longer than 1 GiB, the largest item memcached can+-- store: from 2147483646 bytes, every memcached release up to 1.6.45 runs the+-- value's bytes as commands. The operations in "Database.Memcache.Client" check+-- the length before sending anything, but this encodes a value of any length.+setManyRequest :: WireKey -> Value -> Flags -> Expiration -> Int -> ByteString+setManyRequest key value flags expiry opaque = render (B.length (wireKey key) + B.length value) $+ bytes cmdMetaSet <> bytes " " <> bytes (wireKey key) <> bytes " " <> intDec (B.length value)+ <> bytes " F" <> word32Dec flags <> bytes " T" <> word32Dec expiry+ <> bytes " M" <> bytes (storeMode Set) <> bytes (keyFlag key) <> bytes " c O" <> intDec opaque+ <> bytes crlf <> byteStringCopy value <> bytes crlf+ -- | Encode an increment or decrement request. Unless the expiration is -- 'maxBound', the request asks the server to create a missing key with the -- initial value and that expiration.@@ -382,10 +515,10 @@ -- that "Database.Memcache.Server" makes check the server's version, but a -- connection made with 'newConnection' directly does not. ----- Nor send a value longer than 1 GiB, the largest item memcached can store:--- memcached mishandles a data length over 2147483645 bytes, and then runs the--- rest of the value's bytes as commands. The operations in--- "Database.Memcache.Client" refuse such a value; this encoder doesn't.+-- Nor send it with a value longer than 1 GiB, the largest item memcached can+-- store: from 2147483646 bytes, every memcached release up to 1.6.45 runs the+-- value's bytes as commands. The operations in "Database.Memcache.Client" check+-- the length before sending anything, but this encodes a value of any length. modifyRequest :: StoreMode -> WireKey -> Value -> Version -> ByteString modifyRequest mode key value version = render (B.length (wireKey key) + B.length value) $ bytes cmdMetaSet <> bytes " " <> bytes (wireKey key) <> bytes " " <> intDec (B.length value)@@ -397,6 +530,13 @@ deleteRequest key 0 = cmdMetaDelete <> " " <> keyArgs key <> crlf deleteRequest key version = render (B.length (wireKey key)) $ bytes cmdMetaDelete <> bytes " " <> bytes (keyArgs key) <> cas version <> bytes crlf++-- | Encode a delete request for a pipelined batch. It is 'deleteRequest'+-- without a CAS, plus the given opaque as the @O@ flag, which memcached copies+-- into the reply so that it can be matched to its request.+deleteManyRequest :: WireKey -> Int -> ByteString+deleteManyRequest key opaque = render (B.length (wireKey key)) $+ bytes cmdMetaDelete <> bytes " " <> bytes (keyArgs key) <> bytes " O" <> intDec opaque <> bytes crlf -- | Encode a flush request, optionally delayed until the given expiration. flushRequest :: Maybe Expiration -> ByteString
Database/Memcache/Types.hs view
@@ -17,13 +17,13 @@ -- * Fields & Values Key, Value, Initial, Delta, Expiration, Flags, Version, Status(..),- ServerSpec(..), parseServerSpec+ ServerSpec(..), defaultServerSpec, parseServerSpec ) where import Data.ByteString (ByteString) import Data.Word import Database.Memcache.Types.Authentication (Authentication(..), Username, Password)-import Database.Memcache.Types.ServerSpec (ServerSpec(..), parseServerSpec)+import Database.Memcache.Types.ServerSpec (ServerSpec(..), defaultServerSpec, parseServerSpec) -- | Memcached key bytes. type Key = ByteString@@ -49,9 +49,9 @@ | ErrKeyNotFound -- | The key already exists. | ErrKeyExists- -- | The value exceeds the server limit. The client also throws this,- -- without sending anything, for a value larger than 1 GiB, the largest- -- item memcached can store.+ -- | The value exceeds the server limit. The client also throws this+ -- itself, before sending anything, for a value larger than 1 GiB, the+ -- largest item any memcached server can store. | ErrValueTooLarge -- | The item was not stored. | ErrItemNotStored
Database/Memcache/Types/Authentication.hs view
@@ -9,11 +9,16 @@ -} module Database.Memcache.Types.Authentication ( -- * Text authentication- Authentication(..), Username, Password,+ Authentication(..), Username, Password, validateCredentials, ) where import Data.ByteString (ByteString)+import qualified Data.ByteString.Char8 as B8 -- | Username and password information for text auth-file authentication.+--+-- 'show' prints the password as @\<redacted\>@, so logging credentials, or a+-- server specification holding them, does not reveal it. The username is+-- shown. data Authentication = Auth { -- | Username to send to the server.@@ -23,10 +28,30 @@ } -- | Do not authenticate the connection. | NoAuth- deriving (Eq, Show)+ deriving (Eq) +-- | The derived format, with the password replaced by @\<redacted\>@.+instance Show Authentication where+ showsPrec d (Auth user _) = showParen (d >= 11) $+ showString "Auth {username = " . shows user+ . showString ", password = <redacted>}"+ showsPrec _ NoAuth = showString "NoAuth"+ -- | Username for text authentication. type Username = ByteString -- | Password for text authentication. type Password = ByteString++-- | Check credentials before they are sent: neither part may contain a+-- space, tab, CR, LF or NUL, and the username may not contain @:@. The+-- password may contain @:@, as memcached's auth file splits each line at+-- its first @:@.+validateCredentials :: Username -> Password -> Either String ()+validateCredentials user pass+ | B8.elem ':' user || B8.any forbidden user =+ Left "username contains a forbidden delimiter"+ | B8.any forbidden pass = Left "password contains a forbidden delimiter"+ | otherwise = Right ()+ where+ forbidden c = c == ' ' || c == '\t' || c == '\r' || c == '\n' || c == '\0'
+ Database/Memcache/Types/Server.hs view
@@ -0,0 +1,70 @@+{-# LANGUAGE RecordWildCards #-}++{-|+Module : Database.Memcache.Types.Server+Description : The Server type+Copyright : (c) David Terei, 2016+License : BSD+Maintainer : code@davidterei.com+Stability : stable+Portability : GHC++The 'Server' type, with all its fields. "Database.Memcache.Server" exports it+with only some of them; "Database.Memcache.Cluster" also sets 'slotTaken'.+Not exposed.+-}+module Database.Memcache.Types.Server (+ Server(..), Slot+ ) where++import Database.Memcache.Socket (Connection)+import Database.Memcache.Types (Authentication)++import Data.IORef+import Data.Pool (Pool)+import Data.Time.Clock.POSIX (POSIXTime)+import Network.Socket (HostName, ServiceName)++-- | Memcached server connection.+data Server = Server {+ -- | Hash of the server's host and port, which the 'Eq' and 'Ord'+ -- instances compare. Only the deprecated @legacyServerForKey@ routes+ -- keys by this value; see "Database.Memcache.Cluster".+ sid :: {-# UNPACK #-} !Int,+ -- | Connection pool to server. It holds slots rather than+ -- connections, so that waiting for a free slot is all a request does+ -- before it holds one; it connects afterwards, if it needs to.+ pool :: Pool Slot,+ -- | Hostname of server.+ addr :: !HostName,+ -- | Port number of server.+ port :: !ServiceName,+ -- | Credentials for server.+ auth :: !Authentication,+ -- | When did the server fail? 0 if it is alive.+ failed :: IORef POSIXTime,+ -- | Bumped by @closeAll@. A connection opened in an earlier generation+ -- is closed instead of being used again or returned to the pool.+ generation :: IORef Int,+ -- | Set to 'True' by @withSocket@ as soon as it holds a slot. Cluster+ -- operations give each attempt a copy of the server with a new one,+ -- so that a timeout can tell an attempt still waiting for a slot,+ -- which never reached the server, from one that has taken a slot.+ -- 'Nothing' on the server itself.+ slotTaken :: Maybe (IORef Bool)+ }++-- | A place in a server's pool for one connection, which @withSocket@ opens+-- when a request first needs it. The connection is paired with the+-- 'generation' it was opened in.+type Slot = IORef (Maybe (Int, Connection))++instance Show Server where+ show Server{..} =+ "Server [" ++ show sid ++ "] " ++ addr ++ ":" ++ show port++instance Eq Server where+ (==) x y = sid x == sid y++instance Ord Server where+ compare x y = compare (sid x) (sid y)
Database/Memcache/Types/ServerSpec.hs view
@@ -14,26 +14,49 @@ -} module Database.Memcache.Types.ServerSpec ( ServerSpec (..),+ defaultServerSpec, parseServerSpec, ) where import Control.Monad (guard)+import Data.ByteString (ByteString)+import qualified Data.ByteString as B+import qualified Data.ByteString.Char8 as C+import Data.Char (digitToInt, isHexDigit, ord) import Data.Default import Data.Maybe (fromMaybe)-import qualified Data.Text as T-import Data.Text.Encoding (encodeUtf8)-import Database.Memcache.Types.Authentication (Authentication (..))+import Database.Memcache.Types.Authentication (Authentication (..), validateCredentials) import Database.Memcache.Util (note) import Network.Socket (HostName, ServiceName)-import Network.URI (URI (..), URIAuth (..), parseAbsoluteURI)-import Data.Bifunctor (second)+import Network.URI (URI (..), URIAuth (..), parseAbsoluteURI, unEscapeString) -- | A Memcached server connection configuration.+--+-- 'show' hides the password; see 'Authentication'. data ServerSpec = ServerSpec- { -- | Hostname of server to connect to.+ { -- | Hostname of server to connect to. If it resolves to several+ -- addresses, each is tried in turn until one accepts the connection. If+ -- each refuses it or fails in some other way, connecting fails with the+ -- first address's error. The addresses share the time an attempt at an+ -- operation has (@optsServerTimeout@): an address that doesn't answer in+ -- that time uses it up, and that attempt then tries none of the later+ -- ones.+ --+ -- If it begins with @/@, it is instead the path of a Unix socket. The+ -- client connects to it without looking it up or using 'ssPort', though+ -- the port still names the server: see 'ssPort'. Each 'Char' of the path+ -- is one byte of it, both to connect and to name the server, so a path+ -- outside ASCII is given as its bytes, as 'parseServerSpec' gives it. ssHost :: HostName,- -- | Port number server is running on.+ -- | Port number server is running on. A Unix socket has none, and the+ -- client doesn't use this to connect to one, but it still names the+ -- server, as it does any other: the host and port make the server's name+ -- on the hash ring that routes keys (see "Database.Memcache.Cluster") and+ -- its @sid@, so changing a socket's port moves keys to other servers.+ -- @\"11211\"@ names a socket by its path alone, as libmemcached's+ -- @memcached_server_add@ does, and @\"0\"@ names it @path:0@, as+ -- libmemcached's @memcached_server_add_unix_socket@ and pylibmc do. ssPort :: ServiceName, -- | Credentials to use for text auth-file authentication with this server. ssAuth :: Authentication@@ -41,37 +64,107 @@ deriving (Eq, Show) -- | Parse a @memcached://@ URI into a @ServerSpec@.+--+-- The username and password are percent-decoded, so+-- @memcached:\/\/user:p%40ss\@cache.example@ has the password @p\@ss@, and+-- must pass 'validateCredentials'; one given without the other is ignored.+-- An IPv6 address goes in brackets, as in @memcached:\/\/[::1]:11211@.+--+-- A path after an empty host is a Unix socket, as in+-- @memcached:\/\/\/run\/memcached.sock@, or+-- @memcached:\/\/user:secret\@\/run\/memcached.sock@ with credentials. The+-- path is percent-decoded to bytes too, which 'ssHost' holds one per 'Char',+-- so @%C3%A9@ is the two bytes that encode U+00E9 in UTF-8. It mustn't contain+-- @%00@, and an @\@@ in it must be written @%40@: a URI with one slash too+-- many, such as @memcached:\/\/\/user:secret\@cache.example@, is then an+-- error, not a socket whose path holds the password. The error says that+-- credentials go right after @memcached:\/\/@: such a URI needs a slash fewer,+-- not its @\@@ written @%40@, which would make it that socket after all. A+-- socket takes no port, so @memcached:\/\/:11211\/run\/memcached.sock@ is an+-- error, and its 'ssPort' is @\"11211\"@, which names it by its path alone+-- (see 'ssPort'). A path of only slashes is no socket: @memcached:\/\/\/@,+-- @memcached:\/\/\/\/@ and @memcached:\/\/:11211\/@ are @127.0.0.1@ on port+-- @11211@, as @memcached:\/\/@ is.+--+-- Error messages never quote the URI, as it may contain a password. parseServerSpec :: String -> Either String ServerSpec parseServerSpec s = do- uri <- note ("Not a valid URI: " <> s) $ parseAbsoluteURI s+ uri <- note "Not a valid URI" $ parseAbsoluteURI s note "Must begin memcached://" $ guard $ uriScheme uri == "memcached:" let mAuth = uriAuthority uri+ auth <- maybe (pure Nothing) (readAuthentication . uriUserInfo) mAuth+ socketPath <- readSocketPath uri pure+ . maybe id setSocketPath socketPath . maybe id setHost mAuth . maybe id setPort mAuth- . maybe id setAuth (readAuthentication . uriUserInfo =<< mAuth)- $ def+ . maybe id setAuth auth+ $ defaultServerSpec -readAuthentication :: String -> Maybe Authentication-readAuthentication = go . T.pack- where- go a = do- (u, p) <- second (T.drop 1) . T.breakOn ":" <$> T.stripSuffix "@" a+-- | Read the credentials from a URI's user information, @user:password\@@.+-- There are none unless both parts are non-empty.+readAuthentication :: String -> Either String (Maybe Authentication)+readAuthentication userInfo = case break (== ':') (takeWhile (/= '@') userInfo) of+ (u@(_ : _), ':' : p@(_ : _)) -> do+ let user = unEscapeBytes u+ pass = unEscapeBytes p+ validateCredentials user pass+ pure $ Just $ Auth user pass+ _ -> pure Nothing - guard $ not $ T.null u- guard $ not $ T.null p+-- | Read the path of a Unix socket from a URI whose host is empty, such as+-- @memcached:\/\/\/run\/memcached.sock@. There is none unless the URI has+-- the @\/\/@ of an authority, so that a URI missing a slash, such as+-- @memcached:\/user:secret\@cache.example@, never puts its password in+-- 'ssHost', which 'show' prints, and unless the path has more than slashes,+-- so that @memcached:\/\/\/@ is still the default server, and+-- @memcached:\/\/\/\/@ isn't the root directory.+--+-- The path is percent-decoded to bytes, one 'Char' each, as+-- 'Network.Socket.SockAddrUnix' writes a path one byte per 'Char'. A NUL+-- would end the path early, so it is an error. So is an @\@@ that isn't+-- written @%40@, as in @memcached:\/\/\/user:secret\@cache.example@, which+-- has one slash too many: as a path, its password would be in 'ssHost' too.+-- The error says that credentials go right after @memcached:\/\/@ before it+-- says that an @\@@ in a socket path is written @%40@: in such a URI, a @%40@+-- would put the password in 'ssHost' after all.+readSocketPath :: URI -> Either String (Maybe FilePath)+readSocketPath uri = case uriAuthority uri of+ Just a | null (uriRegName a), any (/= '/') path -> do+ note "A Unix socket path takes no port" $ guard $ null (uriPort a)+ note "Unix socket path contains NUL" $ guard $ notElem '\0' path+ note rawAtMessage $ guard $ notElem '@' (uriPath uri)+ pure (Just path)+ _ -> pure Nothing+ where+ -- After an authority, a path is empty or begins with a slash, so this+ -- one begins with a slash whenever it has more than slashes.+ path = C.unpack (unEscapeBytes (uriPath uri))+ rawAtMessage =+ "Unix socket path contains @; credentials go right after memcached://, as in "+ <> "memcached://user:password@host, and an @ in a socket path is written %40" - pure- Auth- { username = encodeUtf8 u,- password = encodeUtf8 p- }+-- | Percent-decode a string to the bytes it encodes. Unlike+-- 'unEscapeString', this doesn't decode them as UTF-8, which would replace+-- invalid sequences such as @%FF@ with U+FFFD. 'parseAbsoluteURI' has already+-- rejected malformed escapes and any character outside ASCII.+unEscapeBytes :: String -> ByteString+unEscapeBytes = B.pack . go+ where+ go ('%' : a : b : rest)+ | isHexDigit a && isHexDigit b =+ fromIntegral (digitToInt a * 16 + digitToInt b) : go rest+ go (c : rest) = fromIntegral (ord c) : go rest+ go [] = [] setHost :: URIAuth -> ServerSpec -> ServerSpec setHost auth ss = case uriRegName auth of "" -> ss+ -- An IP literal such as an IPv6 address: drop the brackets, and decode the+ -- "%25" that introduces a zone ID (RFC 6874).+ '[' : literal -> ss {ssHost = unEscapeString $ takeWhile (/= ']') literal} rn -> ss {ssHost = rn} setPort :: URIAuth -> ServerSpec -> ServerSpec@@ -82,8 +175,17 @@ p -> Just p pure $ ss {ssPort = p} +setSocketPath :: FilePath -> ServerSpec -> ServerSpec+setSocketPath path ss = ss {ssHost = path}+ setAuth :: Authentication -> ServerSpec -> ServerSpec setAuth auth ss = ss {ssAuth = auth} +-- | The default server: @127.0.0.1@ on memcached's default port, @11211@,+-- without authentication ('NoAuth').+defaultServerSpec :: ServerSpec+defaultServerSpec = ServerSpec "127.0.0.1" "11211" NoAuth++-- | 'def' is 'defaultServerSpec'. instance Default ServerSpec where- def = ServerSpec "127.0.0.1" "11211" NoAuth+ def = defaultServerSpec
Database/Memcache/Util.hs view
@@ -10,9 +10,27 @@ Small helpers shared by the library's modules. Not exposed. -} module Database.Memcache.Util (- note+ note,+ withDetail ) where +import Data.Char (isAscii, isPrint, showLitChar)+ -- | Turn a 'Nothing' into an error. note :: e -> Maybe a -> Either e a note e = maybe (Left e) Right++-- | Append a detail, which may hold bytes a server sent, keeping the message+-- on one line of ASCII: characters outside printable ASCII and backslashes are+-- escaped as in a Haskell string literal, and only the first 200 characters+-- are kept.+withDetail :: String -> String -> String+withDetail problem "" = problem+withDetail problem detail = problem <> ": " <> escape kept <> ellipsis+ where+ (kept, rest) = splitAt 200 detail+ ellipsis = if null rest then "" else "..."+ escape = foldr escapeChar ""+ escapeChar c+ | isAscii c && isPrint c && c /= '\\' = (c :)+ | otherwise = showLitChar c
README.md view
@@ -12,8 +12,11 @@ authentication. Keys containing whitespace or control bytes are sent using its base64 binary-key mode. The encoded key must still fit within 250 bytes. It supports connecting to a single, or a cluster of memcached servers. When-connecting to a cluster, consistent hashing is used for routing requests to-the appropriate server.+connecting to a cluster, keys are routed by ketama consistent hashing,+compatible with libmemcached's weighted ketama distribution, so a key goes to+the same server as from libmemcached-based clients given the same server list.+Version 0.5 changed the routing, so upgrading moves keys to different servers,+once; see [Upgrading from 0.4](#upgrading-from-04). **This library requires memcached 1.6.10 or newer.** Older servers reject the meta set command this library stores values with, without reading the value@@ -85,6 +88,37 @@ documentation covers every operation, along with the timeouts, retries and failover configured through `Options`. +Create one client when your program starts and share it: it is safe to use+from many threads, and it keeps its connections to each server open between+requests. A client created for each request opens a new connection to every+server it uses (and authenticates, if credentials are set) every time.+`M.withClient` closes a client's connections when an action ends, so use it+around your whole program, a worker or a test, not around single requests.+Each server's pool holds one connection by default (with resource-pool 0.3,+one for each capability), so concurrent requests to the same server wait for+each other; see [Architecture Notes](#architecture-notes) to allow more. The+wait counts towards each attempt's `optsServerTimeout`. An attempt that runs+out of time while still waiting is retried like any failed attempt, but if it+was the last attempt, the operation fails with `Timeout` and the server isn't+marked dead. An attempt that runs out of time afterwards, while opening,+authenticating or using its connection, counts as a server failure, however+long it waited first.++A failed operation is retried, except that a request that mustn't be sent+twice (`increment`, `decrement`, `append`, `prepend`, `add`, `delete`, and+`cas` or `replace` with a version) isn't sent again once it may have reached+the server: the failure is thrown, and the server is marked dead. Other+operations are retried after any failure, so the server may carry out their+request twice. Set `optsServerRetries` to 0 for no retries.++A `SERVER_ERROR` reply is a failure too, as a timeout is, unless its text+names a status, such as running out of memory. One that names a status, and+the other error replies, such as `CLIENT_ERROR`, are thrown at once and don't+count against the server. memcached's proxy sends a `SERVER_ERROR` when a+backend fails, so behind it, a request for a key on a failed backend marks the+whole proxy dead: at once if the request mustn't be sent twice, and otherwise+once its retries get the same reply.+ ### Servers and authentication `newClient` takes one `ServerSpec` per server, and routes each key to one of@@ -114,6 +148,36 @@ credentials and all data are sent unencrypted; only rely on them within a trusted network. +### Unix sockets++A host that begins with `/` is the path of a Unix socket instead, for a+memcached on the same machine. Start memcached with `-s <path>` to listen on+the socket, which stops it listening on the network, and with `-a <mode>` to+set the socket's permissions: the default, `0700`, lets only memcached's own+user connect.++```haskell+ mc <- M.newClient [M.ServerSpec "/run/memcached/memcached.sock" "11211" M.NoAuth] M.def+```++As a URI, the path follows an empty host, and there is no port:+`memcached:///run/memcached/memcached.sock`, or+`memcached://user:secret@/run/memcached/memcached.sock` with credentials. An+`@` in the path is written `%40`; a raw one is an error, so that a URI with one+slash too many, such as `memcached:///user:secret@cache.example`, is never+taken for a socket path holding the password. The error says that credentials+go right after `memcached://`: such a URI needs a slash fewer, not its `@`+written `%40`, which would make it that socket path after all.++The client doesn't use a socket's port to connect, but the port still names+the server, along with the path, when keys are routed, so changing it moves+keys to other servers. `"11211"`, which a URI gives, names the socket by its+path alone, as libmemcached's `memcached_server_add` does, and `"0"` names it+`path:0`, as libmemcached's `memcached_server_add_unix_socket` and pylibmc do.++The operating system limits the path's length: 108 bytes on Linux and 104 on+macOS, including the terminating NUL.+ ### Amazon ElastiCache `Database.Memcache.ElastiCacheClient` creates a client from an ElastiCache@@ -132,6 +196,52 @@ Discovery connects to the configuration endpoint like any other server, so it fails for a cluster running a memcached older than 1.6.10. +### Upgrading from 0.4++Version 0.5 requires memcached 1.6.10 or newer and refuses an older server, so+upgrade any older memcached first, including any behind a proxy.++Version 0.5 routes keys with ketama consistent hashing, where 0.4 sent keys to+no more than two servers at a time. With n servers, upgrading moves all but+about one key in n to a different server, once: about half the keys with 2+servers, 70% with 3 and 88% with 8. Expect a burst of cache misses.++While old and new versions run side by side, as in a rolling deploy, they read+and write a moved key on different servers. One can then read a stale copy of+a key that the other has changed or deleted, until the copy expires. To avoid+that, either switch every instance at once, at a quiet time, optionally+flushing the cache afterwards, or switch in stages, deploying each step to+every instance before starting the next:++1. Upgrade with `optsGetServerForKey = legacyServerForKey`, 0.4's router,+ now deprecated, from `Database.Memcache.Cluster`. Keys stay where they+ are, as long as the build uses the same `hashable` version and platform+ as before. With `-Werror`, also pass `-Wno-deprecations`.+2. Add a second client for the same servers that uses `ketamaServerForKey`.+ Whenever a key is changed or deleted, also delete it through that client,+ unless both clients send it to the same server.+3. If a server was marked failed less than your longest expiry time ago,+ flush the cache first. Then swap the two clients' routers.+4. Once your longest expiry time has passed since step 3 was on every+ instance, remove the second client. If some keys never expire, flush the+ cache instead of waiting, and then remove the second client.++The+[`Database.Memcache.Cluster`](https://hackage.haskell.org/package/memcache/docs/Database-Memcache-Cluster.html)+documentation has the details.++`optsServerRetries` changed too. It now counts the retries after an+operation's first attempt, where 0.4 counted the attempts, so a value of 1 or+more that you set yourself makes one more attempt than in 0.4. Set it one+lower to keep the old number of attempts. The default went from 2 to 1, which+still means two attempts.++When a server's host name resolves to several addresses, the client now tries+each in turn, where 0.4 tried only the first. If the addresses are separate+memcached instances, give each its own `ServerSpec`: while the first refuses+connections, new connections go to the next one that accepts, and so to another+cache.+ ## Licensing This library is BSD-licensed.@@ -150,14 +260,16 @@ ## Architecture Notes We're relying on `Data.Pool` for thread safety right now. Each server has its-own pool, and grabbing a connection from it (`withResource`) blocks other-requests using that connection until the operation completes. `getMany` is the-exception: it pipelines quiet Meta Text requests over one pooled connection.+own pool, and grabbing a connection from it (`Server.withSocket`) blocks other+requests using that connection until the operation completes. `getMany`,+`setMany` and `deleteMany` are the exception: they pipeline Meta Text requests+to each server in batches, each batch over one pooled connection. -By default each server's pool holds a single connection (`soNumResources = 1`),-so concurrent operations against the same server wait for each other. Raising-`soNumResources` in the `ServerOptions` set through `optsServerOptions` allows-concurrent operations without requiring a separate pool implementation.+By default each server's pool holds a single connection (`soNumResources = 1`;+with resource-pool 0.3, one for each capability), so concurrent operations+against the same server wait for each other. Raising `soNumResources` in the+`ServerOptions` set through `optsServerOptions` allows concurrent operations+without requiring a separate pool implementation. Either way, a pool is fine for now. @@ -203,15 +315,19 @@ - `spec` holds the unit and property tests, which run against an in-process mock server and need nothing installed. - `integration` starts its own `memcached` servers on free local ports and- tests against them, so it needs memcached 1.6.10 or newer on your `PATH`.+ tests against them, so it needs memcached 1.6.13 or newer on your `PATH`. Without it, the suite reports a single pending test instead of failing.+ With 1.6.10 to 1.6.12, which the library accepts, two of its tests fail,+ as those servers answer an `append` or `prepend` with the version the+ request carried, or 0, rather than the new one. memcached refuses to run as root without `-u`, so run the suite as an ordinary user. CI lints with [HLint](https://github.com/ndmitchell/hlint) 3.10, so please check that `hlint .` prints `No hints` before opening a pull request. For any-user-visible change, add a line to [CHANGELOG.md](CHANGELOG.md) under an-`Unreleased` heading at the top, adding the heading if it isn't there.+user-visible change, propose a line for [CHANGELOG.md](CHANGELOG.md) in the+pull request's description; don't edit the file, as the next release's pull+request adds the lines. CI runs both build tools on every pull request. The `CI` workflow builds and tests with Stack against each `stack*.yaml` snapshot, and runs HLint and
TODO.md view
@@ -1,6 +1,6 @@ ## ToDo -Behavior and API fixes, including routing and virtual nodes, are tracked in+Behavior and API fixes are tracked in [#46](https://github.com/dterei/memcache-hs/issues/46). Optional:
bench/Parser.hs view
@@ -1,13 +1,18 @@ {-# LANGUAGE OverloadedStrings #-} --- | Micro-benchmarks for the pure parts of the Meta Text protocol layer: key--- encoding, request encoding and reading fields out of responses.+-- | Micro-benchmarks for the pure parts of the Meta Text protocol layer (key+-- encoding, request encoding and reading fields out of responses) and for+-- choosing a server for a key. module Main (main) where import Criterion.Main-import qualified Data.ByteString as B-import qualified Data.ByteString.Char8 as C+import qualified Data.ByteString as B+import qualified Data.ByteString.Char8 as C+import qualified Database.Memcache.Client as Client+import Database.Memcache.Cluster (Options (..), newCluster)+import qualified Database.Memcache.Ketama as Ketama import Database.Memcache.Socket+import Database.Memcache.Types (Authentication (NoAuth), ServerSpec (..)) main :: IO () main = defaultMain@@ -27,10 +32,21 @@ , bench "flags" $ whnf responseFlags valueResponse , bench "number" $ whnf responseNumber (Response "VA" ["5"] "12345") ]+ , bgroup "routing"+ [ bench "keyHash" $ whnf Ketama.keyHash "user:1234:profile"+ , bench "owner, 8 servers" $ whnf (Ketama.owner (Ketama.ring servers)) 0x9e3779b9+ , bench "build ring, 8 servers" $ whnf (\s -> Ketama.owner (Ketama.ring s) 0) servers+ -- A Cluster has no NFData instance, but a function does.+ , env (optsGetServerForKey Client.def <$> newCluster specs Client.def) $ \route ->+ bench "cluster, 8 servers" $ whnfIO (route "user:1234:profile")+ ] ] where key = either (error . show) id (encodeKey "key!") valueResponse = Response "VA" ["3", "f7", "c123456789", "kkey"] "abc"+ hosts = ["10.0.1." <> show i | i <- [1 .. 8 :: Int]]+ servers = [(C.pack host, 1) | host <- hosts]+ specs = [ServerSpec host "11211" NoAuth | host <- hosts] -- | Force the whole encoded key by measuring it, failing if the key is -- rejected so a benchmark can't quietly time the error path instead.
memcache.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name: memcache-version: 0.4.2.1+version: 0.5.0.0 synopsis: A memcached client library. description: A client library for a Memcached cluster. Memcached is an in-memory key-value@@ -15,10 +15,10 @@ Username/password authentication is supported through memcached's @-Y authfile@ mode; the binary protocol and SASL authentication are no longer supported. It supports connecting to a single, or a cluster of Memcached- servers. When connecting to a cluster, consistent hashing is used for routing- requests to the appropriate server. Timeouts, retrying failed operations, and- failover to a different server are all supported, as is Amazon ElastiCache- auto discovery.+ servers. When connecting to a cluster, keys are routed by ketama consistent+ hashing, compatible with libmemcached's weighted ketama distribution.+ Timeouts, retrying failed operations, and failover to a different server are+ all supported, as is Amazon ElastiCache auto discovery. Multi-get is supported through the pipelined @getMany@. Keys containing whitespace or control bytes are sent using the protocol's base64 binary-key@@ -38,8 +38,12 @@ > print v You should only need to import 'Database.Memcache.Client', but other modules- are exposed for lower-level use. Version 0.4 is a breaking change from 0.3;- see the changelog for details.+ are exposed for lower-level use. Version 0.5 is a breaking change from 0.4.+ Keys are now routed by ketama, so with /n/ servers, upgrading moves all but+ about one key in /n/ to a different server, once. The @optsServerRetries@+ option now counts retries, not attempts. Version 0.5, like 0.4.2.1, needs+ memcached 1.6.10 or later. See /Upgrading from 0.4/ in the README, and the+ changelog, for details. category: Database homepage: https://github.com/dterei/memcache-hs#readme@@ -86,16 +90,19 @@ Database.Memcache.Cluster Database.Memcache.ElastiCacheClient Database.Memcache.Errors+ Database.Memcache.Ketama Database.Memcache.Server Database.Memcache.Socket Database.Memcache.Types other-modules: Database.Memcache.Types.Authentication+ Database.Memcache.Types.Server Database.Memcache.Types.ServerSpec Database.Memcache.Util other-extensions:+ CApiFFI CPP OverloadedStrings RecordWildCards@@ -106,6 +113,7 @@ build-depends: , base64-bytestring >=1.2.1.0+ , cryptohash-md5 >=0.11.101.0 , data-default >=0.7.1.1 , hashable >=1.4.2.0 , network >=3.1.4.0@@ -121,17 +129,31 @@ type: exitcode-stdio-1.0 hs-source-dirs: test main-is: Spec.hs- other-modules: MockServer- other-extensions: OverloadedStrings+ other-modules:+ FileLimit+ KetamaVectors+ MockServer+ RoutingSpec+ other-extensions:+ CPP+ OverloadedStrings ghc-options: -threaded -rtsopts "-with-rtsopts=-N" build-depends: , base64-bytestring >=1.2.1.0+ , directory+ , hashable , hspec , memcache , network , QuickCheck+ , time , unliftio + -- To raise the limit on open files; unix doesn't build on Windows.+ if !os(windows)+ build-depends:+ , unix+ -- Runs against a real memcached; reports a pending test if none is on PATH. test-suite integration import: shared@@ -147,6 +169,7 @@ , network , process , time+ , unliftio benchmark parser import: shared
test-integration/Integration.hs view
@@ -3,27 +3,39 @@ module Main where import Control.Concurrent (threadDelay)-import Control.Exception (SomeException, bracket, try)-import Control.Monad (forM_, void)+import Control.Exception (bracket, finally, onException)+import Control.Monad (filterM, forM, forM_, unless, void) import qualified Data.ByteString.Char8 as B-import Data.Either (isRight)+import Data.Char (isAscii)+import Data.Foldable (toList)+import Data.IORef (readIORef) import Data.Maybe (isJust) import Data.Time.Clock.POSIX (getPOSIXTime) import qualified Database.Memcache.Client as M+import qualified Database.Memcache.Cluster as Cluster+import qualified Database.Memcache.ElastiCacheClient as EC+import qualified Database.Memcache.Ketama as K+import qualified Database.Memcache.Server as Server import Database.Memcache.Types (ServerSpec (..)) import Network.Socket (Family (AF_INET), SocketOption (ReuseAddr),- SocketType (Stream), addrAddress, addrFamily,- addrSocketType, bind, close, connect,- defaultHints, defaultProtocol, getAddrInfo,+ SocketType (Stream), bind, close, defaultProtocol, setSocketOption, socket, socketPort, tupleToHostAddress) import qualified Network.Socket as N-import System.Directory (findExecutable, getTemporaryDirectory, removeFile)-import System.IO (hClose, hPutStr, openTempFileWithDefaultPermissions)-import System.Process (ProcessHandle, createProcess, proc,- getProcessExitCode, terminateProcess,+import qualified Network.Socket.ByteString as NB+import System.Directory (doesFileExist, findExecutable, getTemporaryDirectory,+ removeFile, removePathForcibly)+import qualified System.Directory as Dir+import System.Exit (ExitCode)+import System.IO (hClose, hPutStr, openTempFile,+ openTempFileWithDefaultPermissions)+import System.Process (ProcessHandle, callProcess, createProcess, getCurrentPid,+ getPid, getProcessExitCode, proc, terminateProcess, waitForProcess)+import System.Timeout (timeout) import Test.Hspec+import qualified UnliftIO.Async as Async+import UnliftIO.Exception (StringException (..), try) main :: IO () main = do@@ -38,10 +50,17 @@ withAuthfile $ \authfile -> withMemcached path ["-Y", authfile] $ \authPort -> withMemcacheds path 3 $ \clusterPorts ->+ -- 2 MiB of memory, and -M to report running out instead of evicting.+ withMemcached path ["-M", "-m", "2"] $ \fullPort -> hspec $ do describe "single server" $ integrationSpecs port describe "auth-file authentication" $ authSpecs authPort describe "three-server cluster" $ clusterSpecs clusterPorts+ describe "restarted server" $ restartSpecs path+ describe "paused server" $ pausedSpecs path+ describe "Unix socket" $ unixSocketSpecs path+ describe "server out of memory" $ outOfMemorySpecs fullPort+ describe "ElastiCache discovery" $ elastiCacheSpecs port integrationSpecs :: Int -> Spec integrationSpecs port = beforeAll (newClient port) $ do@@ -90,7 +109,7 @@ M.touch client "integration-no-touch" 60 `shouldReturn` Nothing it "tells a plain key from another key's base64 form" $ \client -> do- -- " " is sent, and echoed, as IA== with the b flag.+ -- " " is sent as IA== with the b flag. _ <- M.set client "IA==" "plain" 0 0 _ <- M.set client " " "space" 0 0 values <- M.getMany client ["IA==", " "]@@ -111,6 +130,67 @@ lookup "integration-many" values `shouldSatisfy` isJust lookup "integration key" values `shouldSatisfy` isJust + it "sets many keys, including encoded keys, and gets them back" $ \client -> do+ let items = [("integration-set-many", "one", 1, 0), ("integration set many", "two", 2, 0)]+ keys = [key | (key, _, _, _) <- items]+ versions <- M.setMany client items+ -- One server, so the results are in the order given.+ map fst versions `shouldBe` keys+ values <- M.getMany client keys+ values `shouldMatchList`+ [(key, (value, flags, version)) | ((key, value, flags, _), (_, version)) <- zip items versions]++ it "sets a plain key and another key's base64 form separately" $ \client -> do+ -- " " is sent as IA== with the b flag.+ versions <- M.setMany client [("IA==", "plain", 0, 0), (" ", "space", 0, 0)]+ map fst versions `shouldBe` ["IA==", " "]+ M.getMany client ["IA==", " "] `shouldReturn`+ [(key, (value, 0, version)) | ((key, version), value) <- zip versions ["plain", "space"]]++ it "sets and gets many keys containing byte 0xa0" $ \client -> do+ -- In UTF-8, 0xa0 is the last byte of U+00E0 (C3 A0) and of the Cyrillic+ -- Er (D0 A0). Cut off at 0xa0, the third key is the fourth.+ let keys =+ ["integration-voil\xc3\xa0", "integration-city:\xd0\xa0\xd0\xbe", "integration-x\xa0", "integration-x"]+ versions <- M.setMany client [(key, key, 0, 0) | key <- keys]+ map fst versions `shouldBe` keys+ M.getMany client keys `shouldReturn` [(key, (key, 0, version)) | (key, version) <- versions]++ it "sets a key repeated in a batch to its last value" $ \client -> do+ let key = "integration-set-many-twice"+ [(_, old), (_, new)] <- M.setMany client [(key, "old", 0, 0), (key, "new", 0, 0)]+ new `shouldSatisfy` (/= old)+ M.get client key `shouldReturn` Just ("new", 0, new)++ it "deletes many keys, present and missing" $ \client -> do+ let keys = ["integration-delete-many", "integration delete many", "integration-delete-many-missing"]+ _ <- M.setMany client [(key, "value", 0, 0) | key <- take 2 keys]+ M.deleteMany client keys+ M.getMany client keys `shouldReturn` []+ -- Every key is missing now, which is not an error.+ M.deleteMany client keys++ it "gets many keys that increments created" $ \client -> do+ -- memcached echoes the key of an item an increment created raw, even when+ -- the request sent it as base64. Echoed, "integration-x\ny" would end its+ -- reply line at the newline and pass for "integration-x".+ let keys = ["integration-incr-many", "integration incr many", "integration-x\ny"]+ created <- mapM (\key -> M.increment client key 5 1 0 0) keys+ map (fmap fst) created `shouldBe` replicate 3 (Just 5)+ plain <- M.set client "integration-x" "plain" 0 0+ M.getMany client (keys ++ ["integration-x"]) `shouldReturn`+ [(key, ("5", 0, version)) | (key, Just (_, version)) <- zip keys created]+ ++ [("integration-x", ("plain", 0, plain))]++ it "gets many keys after an append" $ \client -> do+ -- An append replaces the item, and memcached then echoes an encoded key raw.+ let keys = ["integration-append-many", "integration append many"]+ forM_ keys $ \key -> M.set client key "x" 0 0+ appended <- mapM (\key -> M.append client key "y" 0) keys+ appended `shouldSatisfy` all isJust+ M.getMany client keys `shouldReturn`+ [(key, ("xy", 0, version)) | (key, Just version) <- zip keys appended]+ it "increments and decrements counters" $ \client -> do _ <- M.delete client "integration-counter" 0 incremented <- M.increment client "integration-counter" 10 2 0 0@@ -119,6 +199,12 @@ decremented <- M.decrement client "integration-counter" 0 3 0 0 decremented `shouldSatisfy` isJust + it "reads the largest counter and flags values" $ \client -> do+ _ <- M.set client "integration-max" "18446744073709551614" maxBound 0+ fmap fst <$> M.increment client "integration-max" 0 1 0 0 `shouldReturn` Just maxBound+ result <- M.get client "integration-max"+ fmap (\(_, flags, _) -> flags) result `shouldBe` Just maxBound+ it "appends and prepends" $ \client -> do cas <- M.set client "integration-modify" "middle" 0 0 appended <- M.append client "integration-modify" "-end" cas@@ -130,6 +216,19 @@ result <- M.get client "integration-modify" fmap first result `shouldBe` Just "start-middle-end" + it "appends and prepends only when the key exists and the CAS matches" $ \client -> do+ old <- M.set client "integration-modify-cas" "middle" 0 0+ Just appended <- M.append client "integration-modify-cas" "-end" old+ M.append client "integration-modify-cas" "-stale" old `shouldReturn` Nothing++ Just prepended <- M.prepend client "integration-modify-cas" "start-" appended+ M.prepend client "integration-modify-cas" "stale-" old `shouldReturn` Nothing++ M.get client "integration-modify-cas" `shouldReturn` Just ("start-middle-end", 0, prepended)++ M.append client "integration-no-modify" "-end" old `shouldReturn` Nothing+ M.prepend client "integration-no-modify" "start-" old `shouldReturn` Nothing+ it "modifies encoded keys" $ \client -> do _ <- M.set client "integration modify key" "middle" 0 0 _ <- M.append client "integration modify key" "-end" 0@@ -145,6 +244,35 @@ stats <- M.stats client Nothing stats `shouldSatisfy` not . all (null . snd) + -- Count connections over one socket kept open for the whole test, rather+ -- than through stats: stats closes the connection it used, which memcached+ -- notices only asynchronously, so each count through stats would leave a+ -- connection of its own that the next count might still see.+ it "closes the connection withClient opened" $ \_ ->+ withConnectionCount port $ \count -> do+ let -- memcached notices a closed connection asynchronously.+ waitFor expected attempts = do+ current <- count+ if current == expected || attempts <= (0 :: Int)+ then current `shouldBe` expected+ else threadDelay 10000 >> waitFor expected (attempts - 1)+ -- The count a previous test's closed connection leaves behind takes a+ -- moment to clear, so read the baseline only once two readings agree.+ settle attempts = do+ a <- count+ threadDelay 50000+ b <- count+ if a == b || attempts <= (0 :: Int) then return b else settle (attempts - 1)+ initial <- settle 100+ other <- M.withClient [ServerSpec "127.0.0.1" (show port) M.NoAuth] M.def $ \c -> do+ _ <- M.version c+ waitFor (initial + 1) 100+ return c+ waitFor initial 100+ -- Using the client afterwards keeps it reachable, so the garbage+ -- collector can't be what closed its connection.+ M.quit other+ it "roundtrips a large value" $ \client -> do let value = B.replicate 100000 'x' _ <- M.set client "integration-large" value 0 0@@ -174,24 +302,52 @@ it "rejects a client without credentials" $ do client <- M.newClient [server M.NoAuth] M.def M.get client "integration-auth" `shouldThrow` isBadCommand++ it "accepts a password containing ':'" $ do+ client <- M.newClient [server (M.Auth "colon" "se:cr:et")] M.def+ _ <- M.set client "integration-auth-colon" "value" 0 0+ fmap first <$> M.get client "integration-auth-colon" `shouldReturn` Just "value" where server = ServerSpec "127.0.0.1" (show port) isBadCommand (M.ProtocolError (M.BadCommand _)) = True isBadCommand _ = False --- | Tests against several servers. Key routing currently leaves at least one--- server without keys (#46), so these don't assume keys reach every server.+-- | Tests against several servers. clusterSpecs :: [Int] -> Spec clusterSpecs ports = beforeAll newClusterClient $ do it "stores and reads back keys" $ \client -> do storeKeys client forM_ keys $ \key -> fmap first <$> M.get client key `shouldReturn` Just key + it "stores each key on the server the ketama ring assigns it" $ \client -> do+ storeKeys client+ -- Ask each server on its own which keys it holds.+ held <- forM ports $ \port -> do+ single <- M.newClient [ServerSpec "127.0.0.1" (show port) M.NoAuth] M.def+ found <- filterM (fmap isJust . M.get single) keys+ M.quit single+ return found+ -- Keys are hashed as given, not in the base64 form that keys with spaces+ -- are sent in.+ let ring = K.ring [(K.serverName "127.0.0.1" (B.pack (show port)), 1) | port <- ports]+ owner key = (ports !!) <$> K.owner ring (K.keyHash key)+ -- Before 0.5 only two servers got keys, so one of the three got none (#46).+ held `shouldSatisfy` (not . any null)+ held `shouldBe` [filter ((== Just port) . owner) keys | port <- ports]+ it "gets many keys" $ \client -> do storeKeys client values <- M.getMany client keys map fst values `shouldMatchList` keys + it "sets and deletes many keys" $ \client -> do+ versions <- M.setMany client [(key, key, 0, 0) | key <- keys]+ map fst versions `shouldMatchList` keys+ values <- M.getMany client keys+ values `shouldMatchList` [(key, (key, 0, version)) | (key, version) <- versions]+ M.deleteMany client keys+ M.getMany client keys `shouldReturn` []+ it "reads stats from every server" $ \client -> do stats <- M.stats client Nothing length stats `shouldBe` length ports@@ -203,7 +359,10 @@ flushCounts client `shouldReturn` map (+ 1) counts M.getMany client keys `shouldReturn` [] where- keys = [B.pack ("integration-cluster-" <> show n) | n <- [1 .. 300 :: Int]]+ -- Keys with spaces are sent in base64.+ keys =+ [B.pack ("integration-cluster-" <> show n) | n <- [1 .. 300 :: Int]]+ <> [B.pack ("integration cluster " <> show n) | n <- [1 .. 60 :: Int]] storeKeys client = forM_ keys $ \key -> M.set client key key 0 0 -- How many times each server has run flush_all, in server order. flushCounts client = do@@ -214,6 +373,157 @@ M.flush client Nothing return client +-- | memcached closes every connection when it stops, so a pool that held+-- several connections holds only closed ones once it is back.+restartSpecs :: FilePath -> Spec+restartSpecs executable = do+ it "isn't marked dead when the pool held several connections" $ do+ (port, client, server) <- withMemcached executable [] $ \port -> do+ client <- M.newClient [ServerSpec "127.0.0.1" (show port) M.NoAuth]+ M.def { M.optsServerOptions = M.def { M.soNumResources = 3 } }+ [server] <- toList <$> Cluster.getServers client+ _ <- M.set client "integration-restart" "value" 0 0+ -- Hold three connections at once, so that the pool keeps three.+ Server.withSocket server $ \_ -> Server.withSocket server $ \_ ->+ Server.withSocket server $ \_ -> return ()+ return (port, client, server)+ -- That memcached has exited, closing all three. A new one starts empty,+ -- on the same port, which another process could take in between.+ withMemcachedOn executable [] port $ do+ M.get client "integration-restart" `shouldReturn` Nothing+ readIORef (Server.failed server) `shouldReturn` 0++ -- An increment mustn't be sent twice, so it can't be retried once sent. The+ -- client checks the pooled connection before sending it, finds it closed,+ -- and sends the increment once, on a new connection.+ it "sends an increment once, on a new connection, after a restart" $ do+ (port, client, server) <- withMemcached executable [] $ \port -> do+ client <- M.newClient [ServerSpec "127.0.0.1" (show port) M.NoAuth] M.def+ [server] <- toList <$> Cluster.getServers client+ _ <- M.set client "integration-restart-counter" "0" 0 0+ return (port, client, server)+ withMemcachedOn executable [] port $ do+ other <- M.newClient [ServerSpec "127.0.0.1" (show port) M.NoAuth] M.def+ _ <- M.set other "integration-restart-counter" "5" 0 0+ fmap fst <$> M.increment client "integration-restart-counter" 0 1 0 0 `shouldReturn` Just 6+ readIORef (Server.failed server) `shouldReturn` 0+ fmap first <$> M.get other "integration-restart-counter" `shouldReturn` Just "6"++-- | Tests against a memcached stopped with SIGSTOP while a request is on its+-- way, so that the request's reply times out, and the server carries the+-- request out once it continues.+pausedSpecs :: FilePath -> Spec+pausedSpecs executable = do+ it "doesn't send an increment again after its reply times out" $+ withPaused "integration-paused-counter" (Just "0") "incr_hits"+ (\client key -> void (M.increment client key 0 1 0 0)) (Just "1")++ it "doesn't send an add again after its reply times out" $+ withPaused "integration-paused-add" Nothing "cmd_set"+ (\client key -> void (M.add client key "added" 0 0)) (Just "added")+ where+ -- Store the key's value, if any, then stop memcached and run the operation+ -- with a 300 ms timeout. memcached continues 650 ms later, after the first+ -- attempt's timeout and before a retry would start, 1 s after it. The+ -- operation must throw Timeout and mark the server dead, without a retry,+ -- and memcached must then carry the request out once: 300 ms after+ -- memcached continues, another client finds the key's value, and the stat+ -- counting such requests one higher than before. A retry would open a new+ -- connection, which asks for the server's version before it sends the+ -- request, so it can't send the request while memcached is stopped. It+ -- would start once memcached has continued, and send the request, which+ -- memcached would carry out too.+ withPaused key initial stat run expected =+ withMemcachedProcess executable [] $ \(port, process) -> do+ let spec = ServerSpec "127.0.0.1" (show port) M.NoAuth+ client <- M.newClient [spec]+ M.def { M.optsServerTimeout = 300, M.optsFailRetryDelay = 1000 }+ other <- M.newClient [spec] M.def+ [server] <- toList <$> Cluster.getServers client+ mapM_ (\value -> M.set other key value 0 0) initial+ let counted = do+ [(_, stats)] <- M.stats other Nothing+ maybe (fail $ "no " <> B.unpack stat) (return . read @Int . B.unpack) (lookup stat stats)+ earlier <- counted+ -- Open the client's connection first, so the request is all that+ -- waits for memcached.+ _ <- M.get client key+ -- Whatever fails from the STOP on, continue memcached: it acts on the+ -- SIGTERM that ends it after the test only once it continues.+ -- Otherwise it continues 650 ms after the operation starts, whether or+ -- not the operation has finished. Give the operation 5 s more, so that+ -- one that never ends fails the test rather than hanging it, and cancel+ -- it if it is still running then.+ result <- (signal "STOP" process >> waitStopped process+ >> Async.withAsync (run client key) (\running -> do+ threadDelay 650000+ signal "CONT" process+ timeout 5000000 (try (Async.wait running))))+ `finally` signal "CONT" process+ result `shouldBe` Just (Left (M.ClientError M.Timeout))+ readIORef (Server.failed server) >>= (`shouldNotBe` 0)+ threadDelay 300000+ fmap first <$> M.get other key `shouldReturn` expected+ counted `shouldReturn` earlier + 1++-- | Tests against a memcached that listens on a Unix socket.+unixSocketSpecs :: FilePath -> Spec+unixSocketSpecs executable =+ it "sets and gets through a memcached:/// URI" $+ withSocketPath $ \path -> do+ -- memcached makes its socket at the path's bytes, which ssHost holds+ -- one per Char. Outside ASCII, the path's Chars needn't be its bytes:+ -- under a UTF-8 or the POSIX locale they aren't. The URI below encodes+ -- the Chars, so the test is pending for a path that isn't ASCII.+ unless (all isAscii path) $+ pendingWith $ "the socket's path isn't ASCII: " <> show path+ -- With -s, memcached listens on the socket alone. The port only names+ -- its pid file.+ port <- freePort+ withMemcachedOn executable ["-s", path, "-a", "0700"] port $ do+ -- The path is percent-encoded, but for its slashes and unreserved+ -- characters, as in the spec's test of the same URI, in case the+ -- temporary directory's name has an ASCII character a URI can't hold+ -- as it is, such as a space, or an @, which a socket path's URI must+ -- write as %40.+ let unreserved = ['A' .. 'Z'] <> ['a' .. 'z'] <> ['0' .. '9'] <> "-._~"+ hex = B.index "0123456789ABCDEF"+ escape c+ | c == '/' || c `elem` unreserved = [c]+ | otherwise = ['%', hex (fromEnum c `div` 16), hex (fromEnum c `mod` 16)]+ spec <- either fail return $ M.parseServerSpec ("memcached://" <> concatMap escape path)+ ssHost spec `shouldBe` path+ client <- M.newClient [spec] M.def+ cas <- M.set client "integration-unix" "value" 5 0+ M.get client "integration-unix" `shouldReturn` Just ("value", 5, cas)++-- | Tests against a server that reports running out of memory instead of+-- evicting.+outOfMemorySpecs :: Int -> Spec+outOfMemorySpecs port =+ it "reports a counter it has no memory to create as a status, and isn't marked dead" $ do+ client <- newClient port+ -- 200 batches of 256 small values need over twice the server's 2 MiB.+ let batch n = [(B.pack ("integration-fill-" <> show (n * 256 + i)), "x", 0, 0) | i <- [0 .. 255]]+ mapM_ (M.setMany client . batch) [0 .. 199 :: Int] `shouldThrow` (== M.OpError M.ErrOutOfMemory)+ -- The counter needs as much memory as one of the values, so there is none+ -- left for it either: memcached replies "Out of memory allocating new item".+ M.increment client "integration-counter" 0 1 0 0 `shouldThrow` (== M.OpError M.ErrOutOfMemory)+ fmap first <$> M.get client "integration-fill-0" `shouldReturn` Just "x"++-- | ElastiCache discovery against plain memcached, which has no config+-- command: it answers config get cluster with ERROR and keeps the connection+-- open.+elastiCacheSpecs :: Int -> Spec+elastiCacheSpecs port =+ it "throws when the endpoint isn't ElastiCache" $ do+ -- The endpoint check only looks for ".cfg" somewhere in the URI.+ endpoint <- either fail pure $+ EC.parseConfigurationEndpoint ("memcached://127.0.0.1:" <> show port <> "/x.cfg")+ -- The outer timeout turns a hang into a failure.+ timeout 5000000 (EC.newClient M.def (3600 * 1000000) endpoint) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: ERROR"+ newClient :: Int -> IO M.Client newClient port = do client <- M.newClient [ServerSpec "127.0.0.1" (show port) M.NoAuth] M.def@@ -223,6 +533,37 @@ first :: (a, b, c) -> a first (value, _, _) = value +-- | Run an action that can read memcached's @curr_connections@ over a single+-- socket, opened before the action and closed after it. Counting connections+-- this way keeps the counting itself from opening and closing connections,+-- which memcached notices only asynchronously. A count that takes more than+-- 5 s fails, so that a memcached that stops answering fails the test rather+-- than hanging it.+withConnectionCount :: Int -> (IO Int -> IO a) -> IO a+withConnectionCount port action = bracket open close $ \sock -> action (currConnections sock)+ where+ open = do+ sock <- socket AF_INET Stream defaultProtocol+ N.connect sock (N.SockAddrInet (fromIntegral port) (tupleToHostAddress (127, 0, 0, 1)))+ `onException` close sock+ return sock+ -- Send a plain stats request and read its reply up to the END line.+ currConnections sock = do+ reply <- timeout 5000000 (NB.sendAll sock "stats\r\n" >> recvToEnd sock "")+ case reply of+ Just r -> return (fromReply r)+ Nothing -> fail "memcached didn't answer a stats request within 5 s"+ recvToEnd sock acc = do+ chunk <- NB.recv sock 4096+ if B.null chunk+ then fail "memcached closed the stats connection"+ else let reply = acc <> chunk+ in if "END\r\n" `B.isSuffixOf` reply then return reply else recvToEnd sock reply+ fromReply reply =+ case [n | line <- B.lines reply, ["STAT", "curr_connections", n] <- [B.words line]] of+ value:_ -> read (B.unpack value)+ [] -> 0+ -- | Run an action against a memcached started with the given extra -- arguments on a free port. withMemcached@@ -230,18 +571,76 @@ -> [String] -> (Int -> IO a) -> IO a-withMemcached executable args action = bracket acquire release (action . fst)+withMemcached executable args action = withMemcachedProcess executable args (action . fst)++-- | Like 'withMemcached', giving the action memcached's process too.+withMemcachedProcess+ :: FilePath+ -> [String]+ -> ((Int, ProcessHandle) -> IO a)+ -> IO a+withMemcachedProcess executable args = bracket (start (5 :: Int)) (stopMemcached . snd) where- acquire = do+ -- Another process can take the port before memcached listens on it, as+ -- when several runs of these tests share a machine. memcached then exits,+ -- so try another port.+ start attempts = do port <- freePort- (_, _, _, process) <-- createProcess (proc executable (["-l", "127.0.0.1", "-p", show port, "-U", "0"] ++ args))- waitForPort port process- return (port, process)- release (_, process) = do- terminateProcess process- void (waitForProcess process)+ started <- startMemcached executable args port+ case started of+ Right process -> return (port, process)+ Left status+ | attempts > 1 -> start (attempts - 1)+ | otherwise -> fail $ "memcached exited before becoming ready: " <> show status +-- | Run an action against a memcached started with the given extra+-- arguments on the given port, and stop it once the action is done.+withMemcachedOn+ :: FilePath+ -> [String]+ -> Int+ -> IO a+ -> IO a+withMemcachedOn executable args port action =+ withMemcachedProcessOn executable args port (const action)++-- | Like 'withMemcachedOn', giving the action memcached's process.+withMemcachedProcessOn+ :: FilePath+ -> [String]+ -> Int+ -> (ProcessHandle -> IO a)+ -> IO a+withMemcachedProcessOn executable args port = bracket acquire stopMemcached+ where+ acquire = startMemcached executable args port+ >>= either (\status -> fail $ "memcached exited before becoming ready: " <> show status) return++-- | Send a signal, such as @STOP@ or @CONT@, to a process with kill(1).+signal :: String -> ProcessHandle -> IO ()+signal name process = do+ pid <- getPid process+ mapM_ (\p -> callProcess "kill" ["-" <> name, show p]) pid++-- | Wait until every thread of a process sent @STOP@ has stopped. kill(1)+-- returns once the signal is pending: on Linux one thread takes it and only+-- then stops the others, which until then can still read and answer a+-- request. A stopped thread is in state T, or t if the process is traced, as+-- under strace. Where there is no @\/proc@ to check, wait a moment instead.+waitStopped :: ProcessHandle -> IO ()+waitStopped process = getPid process >>= mapM_ (go (500 :: Int))+ where+ go 0 pid = fail $ "process " <> show pid <> " did not stop within 5 s of SIGSTOP"+ go attempts pid = do+ let dir = "/proc/" <> show pid <> "/task/"+ checkable <- Dir.doesDirectoryExist dir+ if not checkable then threadDelay 100000 else do+ threads <- Dir.listDirectory dir+ -- A thread's state follows its name, which is in parentheses.+ states <- forM threads $ \thread ->+ B.take 1 . B.drop 1 . snd . B.breakEnd (== ')') <$> B.readFile (dir <> thread <> "/stat")+ if all (`elem` ["T", "t"]) states then return () else threadDelay 10000 >> go (attempts - 1) pid+ -- | Run an action against @n@ memcached servers. withMemcacheds :: FilePath -> Int -> ([Int] -> IO a) -> IO a withMemcacheds _ 0 action = action []@@ -249,56 +648,76 @@ withMemcached executable [] $ \port -> withMemcacheds executable (n - 1) $ \ports -> action (port : ports) --- | Run an action with an auth file allowing @user@ / @secret@.+-- | Run an action with an auth file allowing @user@ / @secret@ and+-- @colon@ / @se:cr:et@. withAuthfile :: (FilePath -> IO a) -> IO a withAuthfile action = do tmp <- getTemporaryDirectory -- Default permissions, so a memcached that drops privileges (e.g. when -- started as root with -u) can still read it. bracket (openTempFileWithDefaultPermissions tmp "memcached-auth") (removeFile . fst) $ \(path, handle) -> do- hPutStr handle "user:secret\n"+ hPutStr handle "user:secret\ncolon:se:cr:et\n" hClose handle action path +-- | Run an action with the path of a Unix socket in the temporary directory,+-- at which nothing exists yet, and remove whatever is there afterwards, such+-- as the socket memcached made.+withSocketPath :: (FilePath -> IO a) -> IO a+withSocketPath = bracket freshPath removePathForcibly+ where+ -- A temporary file's name is one no other file has. Remove the file, so+ -- that memcached can make its socket in its place.+ freshPath = do+ tmp <- getTemporaryDirectory+ (path, file) <- openTempFile tmp "memcached.sock"+ hClose file+ removeFile path+ return path+ freePort :: IO Int freePort = bracket (socket AF_INET Stream defaultProtocol) close $ \sock -> do setSocketOption sock ReuseAddr 1 bind sock (N.SockAddrInet 0 (tupleToHostAddress (127, 0, 0, 1))) fromIntegral <$> socketPort sock -waitForPort- :: Int- -> ProcessHandle- -> IO ()-waitForPort port process = go 50- where- go :: Int -> IO ()- go 0 = do- -- Still running but never listening: stop it rather than waiting on it.- terminateProcess process- void (waitForProcess process)- fail $ "memcached did not start listening on port " <> show port- go attempts = do- ready <- tryConnect port- if ready- then return ()- else do- exited <- getProcessExitCode process- case exited of- Just status -> fail $ "memcached exited before becoming ready: " <> show status- Nothing -> threadDelay 100000 >> go (attempts - 1)+-- | Start memcached with the given extra arguments on the given port, and+-- wait until it listens there. If it exits first, as it does when another+-- process listens on the port, return how it exited.+startMemcached+ :: FilePath+ -> [String]+ -> Int+ -> IO (Either ExitCode ProcessHandle)+startMemcached executable args port = do+ -- memcached writes its pid file once it listens on the port. A connection+ -- to the port would show that something listens there, but not that it is+ -- this memcached. The file's name has this run's pid, so no other run uses+ -- it, but a run that was killed can leave it behind.+ tmp <- getTemporaryDirectory+ pid <- getCurrentPid+ let pidFile = tmp <> "/memcached-" <> show pid <> "-" <> show port <> ".pid"+ removePathForcibly pidFile+ (_, _, _, process) <- createProcess $+ proc executable (["-l", "127.0.0.1", "-p", show port, "-U", "0", "-P", pidFile] ++ args)+ let wait :: Int -> IO (Either ExitCode ProcessHandle)+ wait 0 = fail $ "memcached did not write its pid file " <> pidFile+ <> " within 5 s of starting on port " <> show port+ wait attempts = do+ listening <- doesFileExist pidFile+ if listening+ then removeFile pidFile >> return (Right process)+ else do+ exited <- getProcessExitCode process+ case exited of+ Just status -> return (Left status)+ Nothing -> threadDelay 100000 >> wait (attempts - 1)+ -- If it is still running but never wrote the file, or anything else goes+ -- wrong, stop it rather than leave it running.+ wait 50 `onException` stopMemcached process -tryConnect- :: Int- -> IO Bool-tryConnect port = do- result <- try $ do- addr : _ <-- getAddrInfo- (Just defaultHints{addrSocketType = Stream})- (Just "127.0.0.1")- (Just (show port))- sock <- socket (addrFamily addr) Stream defaultProtocol- connect sock (addrAddress addr)- close sock- return (isRight @SomeException @() result)+-- | Stop a memcached and wait for it to exit.+stopMemcached :: ProcessHandle -> IO ()+stopMemcached process = do+ terminateProcess process+ void (waitForProcess process)
+ test/FileLimit.hs view
@@ -0,0 +1,30 @@+{-# LANGUAGE CPP #-}++-- | The limit on how many files the test process can have open at once.+module FileLimit (raiseFileLimit) where++#if !defined(mingw32_HOST_OS)+import Control.Monad (void)+import System.Posix.Resource (Resource (..), ResourceLimit (..), ResourceLimits (..),+ getResourceLimit, setResourceLimit)+import UnliftIO.Exception (tryIO)+#endif++-- | Raise the soft limit on open files by @n@, or as far as the hard limit+-- allows. A limit that can't be raised is left as it is. On Windows this does+-- nothing.+raiseFileLimit :: Int -> IO ()+#if defined(mingw32_HOST_OS)+raiseFileLimit _ = return ()+#else+raiseFileLimit n = void . tryIO $ do+ limits <- getResourceLimit ResourceOpenFiles+ case softLimit limits of+ ResourceLimit soft -> do+ let wanted = soft + toInteger n+ raised = case hardLimit limits of+ ResourceLimit hard -> min hard wanted+ _ -> wanted+ setResourceLimit ResourceOpenFiles limits { softLimit = ResourceLimit raised }+ _ -> return ()+#endif
+ test/KetamaVectors.hs view
@@ -0,0 +1,297 @@+-- | Server choices made by the real libmemcached, for checking that our ring+-- routes every key the same way.+--+-- Generated with libmemcached-awesome 1.1.4 (commit aafec5e) with+-- @MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED@ set, by asking+-- @memcached_generate_hash@ for each key. The weighted keys and the+-- @VDEAAAAA@ wrap-around case are libmemcached's own test fixture+-- (@test/fixtures/ketama.hpp@).+module KetamaVectors (Scenario, scenarios) where++-- | A name, the servers as (host, port, weight) in configured order, and each+-- key with the @host:port@ libmemcached chose for it.+type Scenario = (String, [(String, Int, Int)], [(String, String)])++scenarios :: [Scenario]+scenarios =+ [ ( "fixture8_weighted"+ , [("10.0.1.1", 11211, 600), ("10.0.1.2", 11211, 300), ("10.0.1.3", 11211, 200), ("10.0.1.4", 11211, 350), ("10.0.1.5", 11211, 1000), ("10.0.1.6", 11211, 800), ("10.0.1.7", 11211, 950), ("10.0.1.8", 11211, 100)]+ ,+ [ ("SVa_]_V41)", "10.0.1.7:11211")+ , ("*/Z;?V(.\\8", "10.0.1.1:11211")+ , ("30C1*Z*S/_", "10.0.1.2:11211")+ , ("ERR:EC58G>", "10.0.1.7:11211")+ , ("1I=cTMNTKF", "10.0.1.5:11211")+ , ("]VG<`I*Z8)", "10.0.1.5:11211")+ , ("UUTC`-V159", "10.0.1.5:11211")+ , ("@7RU6C6T+Z", "10.0.1.5:11211")+ , ("/XLN0@+36;", "10.0.1.1:11211")+ , ("4(`X;\\V.^c", "10.0.1.1:11211")+ , ("726bW=9*a4", "10.0.1.7:11211")+ , ("\\`)<B)UE,c", "10.0.1.1:11211")+ , ("P1[Ma3=K1/", "10.0.1.8:11211")+ , ("C89I.-V?cT", "10.0.1.8:11211")+ , ("D[HE+cFXDK", "10.0.1.3:11211")+ , ("P1L?NAB[)K", "10.0.1.1:11211")+ , ("cDT0)Z5P6,", "10.0.1.5:11211")+ , ("@JW`+[WAO8", "10.0.1.5:11211")+ , ("\\39DKW^)N_", "10.0.1.3:11211")+ , ("EM75N0+[X1", "10.0.1.4:11211")+ , ("`,SS]NBP,b", "10.0.1.1:11211")+ , ("XX1a9LT+F?", "10.0.1.5:11211")+ , ("Zc\\-,F-c6V", "10.0.1.5:11211")+ , ("1*RTMC7,03", "10.0.1.8:11211")+ , ("*Xc+V0P>32", "10.0.1.7:11211")+ , ("U))Fb-(`,.", "10.0.1.7:11211")+ , ("R-08RNTaRT", "10.0.1.5:11211")+ , ("(LHcO203I3", "10.0.1.5:11211")+ , ("=256P+;Qc8", "10.0.1.5:11211")+ , ("OI5XZ_BBT(", "10.0.1.7:11211")+ , ("2TLRL/UL;:", "10.0.1.7:11211")+ , ("WHD\\O1`ZRW", "10.0.1.5:11211")+ , (".=54)_c;=T", "10.0.1.1:11211")+ , (";G<W-XWZ@b", "10.0.1.5:11211")+ , ("(,>E`)FT\\4", "10.0.1.2:11211")+ , ("HZAU*;P*N]", "10.0.1.7:11211")+ , ("NZ@ZE=O84_", "10.0.1.7:11211")+ , ("6,cEI`F_P>", "10.0.1.6:11211")+ , ("c,5AQ/T5)6", "10.0.1.8:11211")+ , (".O,>>BT)RX", "10.0.1.5:11211")+ , ("XY\\X::LX50", "10.0.1.6:11211")+ , ("+550F^/.01", "10.0.1.6:11211")+ , ("<.X9E2S5+9", "10.0.1.7:11211")+ , ("]\\.UH8_0a1", "10.0.1.4:11211")+ , ("8(6=(T0/Z0", "10.0.1.7:11211")+ , ("8*6a;Sc*X+", "10.0.1.2:11211")+ , ("<QW:;3K6;H", "10.0.1.5:11211")+ , ("7C@EY@-Y?_", "10.0.1.5:11211")+ , ("aPb3E1WD4K", "10.0.1.1:11211")+ , ("?@12R<=1BH", "10.0.1.8:11211")+ , ("QR(a+Q=1FU", "10.0.1.6:11211")+ , ("`C9^FV,960", "10.0.1.6:11211")+ , ("UNHVP..^8H", "10.0.1.4:11211")+ , (":Y.2W2[(35", "10.0.1.7:11211")+ , ("M/HV^_HZ4O", "10.0.1.7:11211")+ , ("ZY16KQ<ICD", "10.0.1.4:11211")+ , ("bV2,`a.PY9", "10.0.1.1:11211")+ , ("U;9:-+5N]9", "10.0.1.1:11211")+ , ("1S/:aJ[1(;", "10.0.1.2:11211")+ , ("Nb-X^]M)I:", "10.0.1.6:11211")+ , ("2;M;ES>J5/", "10.0.1.7:11211")+ , ("[>RZHG97Q9", "10.0.1.6:11211")+ , ("J3/G[)9<^Z", "10.0.1.7:11211")+ , ("N-)88>[O`,", "10.0.1.5:11211")+ , ("NP:=FR\\OaA", "10.0.1.7:11211")+ , ("`@L+W;a,O[", "10.0.1.6:11211")+ , ("W2`P:-+1T[", "10.0.1.5:11211")+ , ("-6G7K^YDIN", "10.0.1.7:11211")+ , ("U>*>9ZI6V5", "10.0.1.6:11211")+ , (".I?^6Ic9RK", "10.0.1.6:11211")+ , ("0OZH^9BKM[", "10.0.1.8:11211")+ , ("5?50UGZ:ML", "10.0.1.5:11211")+ , ("?K2NF@3=IU", "10.0.1.1:11211")+ , ("YI@G-2X?UB", "10.0.1.5:11211")+ , ("7cY</BSaL=", "10.0.1.6:11211")+ , ("A(`KF:[RH8", "10.0.1.6:11211")+ , (";=ZT\\W^P+H", "10.0.1.4:11211")+ , ("b2?WFF56;R", "10.0.1.4:11211")+ , ("CTR74,J+N.", "10.0.1.8:11211")+ , ("<b;*R+QDST", "10.0.1.5:11211")+ , ("\\R^7=9UCG`", "10.0.1.5:11211")+ , ("1bQS5]WOXB", "10.0.1.4:11211")+ , ("M(@X^b[L:K", "10.0.1.1:11211")+ , ("431cBF8,YO", "10.0.1.7:11211")+ , ("(bEIQJ:E./", "10.0.1.6:11211")+ , ("WS/3H*)7F;", "10.0.1.5:11211")+ , ("ZJF[Ia6Q)+", "10.0.1.7:11211")+ , ("<]*QCK8U,>", "10.0.1.7:11211")+ , ("\\[a\\^=V_M0", "10.0.1.6:11211")+ , ("7;RM+8J9YC", "10.0.1.7:11211")+ , ("4*=.SPR[AV", "10.0.1.1:11211")+ , ("-2F+^88P4U", "10.0.1.7:11211")+ , ("X;-F`(N?9D", "10.0.1.7:11211")+ , ("R=F_D-K2a]", "10.0.1.7:11211")+ , ("X*+2aaC.EG", "10.0.1.5:11211")+ , ("[1ZXONX2]a", "10.0.1.4:11211")+ , ("FL;\\GWacaV", "10.0.1.4:11211")+ , ("\\MQ_XNT7L-", "10.0.1.7:11211")+ , ("VD6D0]ba_\\", "10.0.1.1:11211")+ , ("VDEAAAAA", "10.0.1.3:11211")+ ]+ )+ , ( "eq3_mixed_ports"+ , [("10.0.0.1", 11211, 1), ("10.0.0.2", 11211, 1), ("10.0.0.3", 11212, 1)]+ ,+ [ ("a", "10.0.0.2:11211")+ , ("b", "10.0.0.2:11211")+ , ("c", "10.0.0.1:11211")+ , ("foo", "10.0.0.2:11211")+ , ("bar", "10.0.0.2:11211")+ , ("baz", "10.0.0.1:11211")+ , ("hello", "10.0.0.2:11211")+ , ("world", "10.0.0.3:11212")+ , ("user:1", "10.0.0.2:11211")+ , ("user:2", "10.0.0.2:11211")+ , ("user:3", "10.0.0.1:11211")+ , ("user:42", "10.0.0.1:11211")+ , ("session:abcdef", "10.0.0.3:11212")+ , ("counter", "10.0.0.3:11212")+ , ("memcache-hs", "10.0.0.3:11212")+ , ("key1", "10.0.0.3:11212")+ , ("key2", "10.0.0.1:11211")+ , ("key3", "10.0.0.2:11211")+ , ("key4", "10.0.0.1:11211")+ , ("0", "10.0.0.3:11212")+ , ("1", "10.0.0.3:11212")+ , ("2", "10.0.0.3:11212")+ , ("3", "10.0.0.2:11211")+ , ("4", "10.0.0.3:11212")+ , ("5", "10.0.0.2:11211")+ , ("6", "10.0.0.2:11211")+ , ("7", "10.0.0.1:11211")+ , ("8", "10.0.0.1:11211")+ , ("9", "10.0.0.1:11211")+ , ("10", "10.0.0.2:11211")+ ]+ )+ , ( "host4"+ , [("cache1.example.com", 11211, 1), ("cache2.example.com", 11211, 1), ("cache3.example.com", 11211, 1), ("cache4.example.com", 11211, 1)]+ ,+ [ ("a", "cache2.example.com:11211")+ , ("b", "cache4.example.com:11211")+ , ("c", "cache1.example.com:11211")+ , ("foo", "cache2.example.com:11211")+ , ("bar", "cache3.example.com:11211")+ , ("baz", "cache4.example.com:11211")+ , ("hello", "cache4.example.com:11211")+ , ("world", "cache4.example.com:11211")+ , ("user:1", "cache4.example.com:11211")+ , ("user:2", "cache3.example.com:11211")+ , ("user:3", "cache4.example.com:11211")+ , ("user:42", "cache3.example.com:11211")+ , ("session:abcdef", "cache4.example.com:11211")+ , ("counter", "cache4.example.com:11211")+ , ("memcache-hs", "cache2.example.com:11211")+ , ("key1", "cache2.example.com:11211")+ , ("key2", "cache1.example.com:11211")+ , ("key3", "cache3.example.com:11211")+ , ("key4", "cache3.example.com:11211")+ , ("0", "cache1.example.com:11211")+ , ("1", "cache1.example.com:11211")+ , ("2", "cache2.example.com:11211")+ , ("3", "cache3.example.com:11211")+ , ("4", "cache2.example.com:11211")+ , ("5", "cache3.example.com:11211")+ , ("6", "cache4.example.com:11211")+ , ("7", "cache3.example.com:11211")+ , ("8", "cache2.example.com:11211")+ , ("9", "cache1.example.com:11211")+ , ("10", "cache1.example.com:11211")+ ]+ )+ , ( "host4_minus_cache3"+ , [("cache1.example.com", 11211, 1), ("cache2.example.com", 11211, 1), ("cache4.example.com", 11211, 1)]+ ,+ [ ("a", "cache2.example.com:11211")+ , ("b", "cache4.example.com:11211")+ , ("c", "cache1.example.com:11211")+ , ("foo", "cache2.example.com:11211")+ , ("bar", "cache1.example.com:11211")+ , ("baz", "cache4.example.com:11211")+ , ("hello", "cache4.example.com:11211")+ , ("world", "cache4.example.com:11211")+ , ("user:1", "cache4.example.com:11211")+ , ("user:2", "cache2.example.com:11211")+ , ("user:3", "cache4.example.com:11211")+ , ("user:42", "cache1.example.com:11211")+ , ("session:abcdef", "cache4.example.com:11211")+ , ("counter", "cache4.example.com:11211")+ , ("memcache-hs", "cache2.example.com:11211")+ , ("key1", "cache2.example.com:11211")+ , ("key2", "cache1.example.com:11211")+ , ("key3", "cache1.example.com:11211")+ , ("key4", "cache2.example.com:11211")+ , ("0", "cache1.example.com:11211")+ , ("1", "cache1.example.com:11211")+ , ("2", "cache2.example.com:11211")+ , ("3", "cache1.example.com:11211")+ , ("4", "cache2.example.com:11211")+ , ("5", "cache2.example.com:11211")+ , ("6", "cache4.example.com:11211")+ , ("7", "cache4.example.com:11211")+ , ("8", "cache2.example.com:11211")+ , ("9", "cache1.example.com:11211")+ , ("10", "cache1.example.com:11211")+ ]+ )+ , ( "eq25"+ , [("10.0.0.1", 11211, 1), ("10.0.0.2", 11211, 1), ("10.0.0.3", 11211, 1), ("10.0.0.4", 11211, 1), ("10.0.0.5", 11211, 1), ("10.0.0.6", 11211, 1), ("10.0.0.7", 11211, 1), ("10.0.0.8", 11211, 1), ("10.0.0.9", 11211, 1), ("10.0.0.10", 11211, 1), ("10.0.0.11", 11211, 1), ("10.0.0.12", 11211, 1), ("10.0.0.13", 11211, 1), ("10.0.0.14", 11211, 1), ("10.0.0.15", 11211, 1), ("10.0.0.16", 11211, 1), ("10.0.0.17", 11211, 1), ("10.0.0.18", 11211, 1), ("10.0.0.19", 11211, 1), ("10.0.0.20", 11211, 1), ("10.0.0.21", 11211, 1), ("10.0.0.22", 11211, 1), ("10.0.0.23", 11211, 1), ("10.0.0.24", 11211, 1), ("10.0.0.25", 11211, 1)]+ ,+ [ ("a", "10.0.0.4:11211")+ , ("b", "10.0.0.16:11211")+ , ("c", "10.0.0.15:11211")+ , ("foo", "10.0.0.19:11211")+ , ("bar", "10.0.0.8:11211")+ , ("baz", "10.0.0.25:11211")+ , ("hello", "10.0.0.21:11211")+ , ("world", "10.0.0.9:11211")+ , ("user:1", "10.0.0.16:11211")+ , ("user:2", "10.0.0.6:11211")+ , ("user:3", "10.0.0.18:11211")+ , ("user:42", "10.0.0.1:11211")+ , ("session:abcdef", "10.0.0.13:11211")+ , ("counter", "10.0.0.19:11211")+ , ("memcache-hs", "10.0.0.19:11211")+ , ("key1", "10.0.0.5:11211")+ , ("key2", "10.0.0.12:11211")+ , ("key3", "10.0.0.3:11211")+ , ("key4", "10.0.0.7:11211")+ , ("0", "10.0.0.5:11211")+ , ("1", "10.0.0.4:11211")+ , ("2", "10.0.0.15:11211")+ , ("3", "10.0.0.16:11211")+ , ("4", "10.0.0.12:11211")+ , ("5", "10.0.0.25:11211")+ , ("6", "10.0.0.21:11211")+ , ("7", "10.0.0.1:11211")+ , ("8", "10.0.0.9:11211")+ , ("9", "10.0.0.17:11211")+ , ("10", "10.0.0.6:11211")+ ]+ )+ , ( "localhost_2ports"+ , [("localhost", 11211, 1), ("localhost", 11212, 1)]+ ,+ [ ("a", "localhost:11212")+ , ("b", "localhost:11211")+ , ("c", "localhost:11211")+ , ("foo", "localhost:11211")+ , ("bar", "localhost:11212")+ , ("baz", "localhost:11212")+ , ("hello", "localhost:11211")+ , ("world", "localhost:11212")+ , ("user:1", "localhost:11212")+ , ("user:2", "localhost:11212")+ , ("user:3", "localhost:11211")+ , ("user:42", "localhost:11212")+ , ("session:abcdef", "localhost:11211")+ , ("counter", "localhost:11211")+ , ("memcache-hs", "localhost:11211")+ , ("key1", "localhost:11212")+ , ("key2", "localhost:11211")+ , ("key3", "localhost:11212")+ , ("key4", "localhost:11212")+ , ("0", "localhost:11212")+ , ("1", "localhost:11212")+ , ("2", "localhost:11211")+ , ("3", "localhost:11211")+ , ("4", "localhost:11211")+ , ("5", "localhost:11211")+ , ("6", "localhost:11211")+ , ("7", "localhost:11211")+ , ("8", "localhost:11212")+ , ("9", "localhost:11211")+ , ("10", "localhost:11211")+ ]+ )+ ]
test/MockServer.hs view
@@ -2,11 +2,21 @@ -- | Mock Memcached server - just enough for testing client. ----- Each mock listens on its own ephemeral port on 127.0.0.1, so tests can run--- several at once and never clash with a real memcached on the default port.+-- Each mock listens on 127.0.0.1 unless given another address, and on an+-- ephemeral port of its own unless given a port, so tests can run several at+-- once and never clash with a real memcached on the default port. A mock can+-- also listen on a Unix socket, at a path from 'withSocketPath'.+--+-- The client asks for the server's version on every new connection before it+-- sends anything else. A mock made by 'withMockServer' and the functions like+-- it answers that request like any other, with its next response, so a test+-- can script the reply. A mock made by 'withMockMemcached' and the functions+-- like it reports itself as memcached of a given version instead. module MockServer (- MockResponse(..), MockServer(..), withMockServer, withMockMemcached,- mockServerSpec+ MockResponse(..), MockServer(..), withMockServer, withMockServerAt,+ withConcurrentMockServer, withMockMemcached, withMockMemcachedAt,+ withConcurrentMockMemcached, withBoundSocket, withBoundSocketReuseAddr,+ withFullListener, withSocketPath, canListenAt, mockServerSpec ) where import Control.Concurrent@@ -14,12 +24,18 @@ import qualified Data.ByteString as B import qualified Data.ByteString.Char8 as C import Data.IORef+import Data.Maybe (isJust) import Database.Memcache.Types (Authentication (NoAuth), ServerSpec (..)) import qualified Network.Socket as N import qualified Network.Socket.ByteString as N+import System.Directory (getTemporaryDirectory, removeFile,+ removePathForcibly)+import System.IO (hClose, openTempFile)+import System.Timeout (timeout) import Text.Read (readMaybe)-import UnliftIO.Exception (SomeException, bracket, finally, handle,- throwIO)+import UnliftIO.Async (concurrently_, withAsync)+import UnliftIO.Exception (IOException, SomeException, bracket, finally,+ handle, onException, throwIO) -- | Actions the mock server can take to a request. data MockResponse@@ -31,14 +47,25 @@ | DelayMS Int MockResponse -- | Read the request but send nothing back. | Noop+ -- | Send these bytes over and over until the client closes the+ -- connection.+ | Repeat B.ByteString+ -- | Reply with these bytes, then reset the connection: close it with+ -- @SO_LINGER@ set to 0, so that once the client has read the bytes, its+ -- next read fails with @ECONNRESET@ rather than finding the connection+ -- closed. TCP only. The 'MVar' is filled once the connection is closed.+ | ReplyThenReset B.ByteString (MVar ()) -- | A running mock server. data MockServer = MockServer- { -- | The port the mock is listening on.+ { -- | The port the mock is listening on, or 0 on a Unix socket. mockPort :: N.PortNumber -- | Every request received so far, oldest first, including any data -- block that followed the command line. , mockRequests :: IO [B.ByteString]+ -- | How many connections the client has closed while the mock was+ -- waiting for its next request.+ , mockClosed :: IO Int } -- | A server specification for connecting to the mock without authentication.@@ -50,86 +77,294 @@ -- in order, one per request, across however many connections the client -- makes; once they run out it closes the connection and stops listening. withMockServer :: [MockResponse] -> (MockServer -> IO a) -> IO a-withMockServer = runMockServer Nothing+withMockServer = withMockServerAt loopback +-- | Like 'withMockServer', but listening on the given address instead of+-- 127.0.0.1. A port of 0 picks a free one.+withMockServerAt :: N.SockAddr -> [MockResponse] -> (MockServer -> IO a) -> IO a+withMockServerAt = serveInTurn Nothing+ -- | Like 'withMockServer', but the mock reports itself as memcached of the--- given version, such as @"1.6.24"@. The client asks for the server's version--- on every new connection before sending anything else, so the mock answers--- the first request on each connection with @VERSION@ and that version,+-- given version, such as @"1.6.24"@: it answers the first request on each+-- connection, the client's version check, with @VERSION@ and that version, -- without using up a response. That request is still recorded.+--+-- The mock answers each connection's version check as soon as it accepts the+-- connection, even while it is serving another one, as the client can't use+-- a connection until then. It still serves the responses to one connection+-- at a time, in the order it accepted them, and once the responses run out,+-- it closes every connection and stops listening. withMockMemcached :: B.ByteString -> [MockResponse] -> (MockServer -> IO a) -> IO a-withMockMemcached version = runMockServer (Just ("VERSION " <> version <> "\r\n"))+withMockMemcached version = withMockMemcachedAt version loopback -runMockServer :: Maybe B.ByteString -> [MockResponse] -> (MockServer -> IO a) -> IO a-runMockServer greeting responses action = do+-- | Like 'withMockServerAt', but reporting the given version, as+-- 'withMockMemcached' does.+withMockMemcachedAt :: B.ByteString -> N.SockAddr -> [MockResponse] -> (MockServer -> IO a) -> IO a+withMockMemcachedAt version = serveInTurn (Just (versionReply version))++-- | Like 'withConcurrentMockServer', but reporting the given version, as+-- 'withMockMemcached' does.+withConcurrentMockMemcached :: B.ByteString -> [[MockResponse]] -> (MockServer -> IO a) -> IO a+withConcurrentMockMemcached version = serveConcurrently (Just (versionReply version))++versionReply :: B.ByteString -> B.ByteString+versionReply version = "VERSION " <> version <> "\r\n"++-- | Serve one connection at a time on the given address, answering the first+-- request on each connection with the greeting, if there is one, and then+-- each request with the next of the responses.+serveInTurn+ :: Maybe B.ByteString -> N.SockAddr -> [MockResponse] -> (MockServer -> IO a) -> IO a+serveInTurn greeting address responses = withListener address $ \sock received closed -> do+ remaining <- newIORef responses+ -- Once the responses run out the listening socket closes, so later+ -- connection attempts are refused straight away.+ case greeting of+ Nothing -> acceptLoop sock remaining received closed+ Just reply -> greetingEach reply sock remaining received closed+ where+ -- Serve one connection at a time.+ acceptLoop sock remaining received closed = do+ pending <- readIORef remaining+ unless (null pending) $ do+ (client, _) <- N.accept sock+ serveConnection Nothing client remaining received closed+ acceptLoop sock remaining received closed++ -- Greet each connection in a thread of its own as soon as it is+ -- accepted, and queue it, then serve the queued connections one at a+ -- time. Every connection accepted is closed once the responses run out.+ greetingEach reply sock remaining received closed = do+ queue <- newChan+ accepted <- newIORef []+ let acceptor = forever $ do+ (client, _) <- N.accept sock+ atomicModifyIORef' accepted (\clients -> (client : clients, ()))+ void $ forkIO $ handle ignore $ do+ greet reply client received closed+ writeChan queue client+ serveQueued = do+ pending <- readIORef remaining+ unless (null pending) $ do+ client <- readChan queue+ serveConnection Nothing client remaining received closed+ serveQueued+ withAsync acceptor (const serveQueued)+ `finally` (readIORef accepted >>= mapM_ N.close)++ ignore :: SomeException -> IO ()+ ignore = const (return ())++-- | Like 'withMockServer', but each connection gets its own responses: the+-- n-th connection the server accepts gets the n-th list. The server answers+-- each connection in a thread of its own from the moment it accepts it, so+-- it answers one connection while another is still open. It closes a+-- connection once its responses run out, and stops listening once every+-- list has its connection. 'mockRequests' gives the requests of every+-- connection, in the order they arrived.+withConcurrentMockServer :: [[MockResponse]] -> (MockServer -> IO a) -> IO a+withConcurrentMockServer = serveConcurrently Nothing++serveConcurrently :: Maybe B.ByteString -> [[MockResponse]] -> (MockServer -> IO a) -> IO a+serveConcurrently greeting scripts = withListener loopback $ \sock received closed -> do+ -- Closing the listening socket here refuses later connection attempts+ -- straight away; closing it again on the way out does nothing.+ let serveFrom [] = N.close sock+ serveFrom (responses:rest) = do+ (client, _) <- N.accept sock+ remaining <- newIORef responses+ concurrently_ (serveConnection greeting client remaining received closed) (serveFrom rest)+ serveFrom scripts++-- | Listen on the given address and serve it with the given function in the+-- background, killing it once the action is done. The function is given the+-- listening socket, the IORef in which it records each request it reads,+-- newest first, and the IORef in which it counts the connections the client+-- closed while it waited for a request.+withListener+ :: N.SockAddr -> (N.Socket -> IORef [B.ByteString] -> IORef Int -> IO ())+ -> (MockServer -> IO a) -> IO a+withListener address serveClients action = do ready <- newEmptyMVar received <- newIORef []+ closed <- newIORef 0 bracket- (forkIO (serve ready received))+ (forkIO (serve ready received closed)) killThread (const $ do port <- takeMVar ready action MockServer { mockPort = port , mockRequests = reverse <$> readIORef received+ , mockClosed = readIORef closed }) where- serve ready received = bracket- (N.socket N.AF_INET N.Stream N.defaultProtocol)+ serve ready received closed = bracket+ (newSocket address) N.close $ \sock -> do N.setSocketOption sock N.ReuseAddr 1- N.bind sock (N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1)))+ N.bind sock address N.listen sock 10- N.socketPort sock >>= putMVar ready- remaining <- newIORef responses- -- Once the responses run out the listening socket closes, so later- -- connection attempts are refused straight away.- acceptLoop sock remaining received+ listeningPort sock >>= putMVar ready+ serveClients sock received closed+ -- A Unix socket has no port.+ listeningPort sock = case address of+ N.SockAddrUnix{} -> return 0+ _ -> N.socketPort sock - acceptLoop sock remaining received = do- pending <- readIORef remaining- unless (null pending) $ do- (client, _) <- N.accept sock- handle ignore (greet client received >> clientHandler client remaining received)- `finally` N.close client- acceptLoop sock remaining received+-- | An ephemeral port on 127.0.0.1.+loopback :: N.SockAddr+loopback = N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1)) - -- Answer the version check that starts each connection, if the mock- -- reports a version.- greet client received = forM_ greeting $ \reply -> do- request <- recvReq client- modifyIORef' received (request :)- N.sendAll client reply+-- | Whether 'withMockServerAt' can listen on the given address. It cannot on+-- an IPv6 address where IPv6 is disabled, for example.+canListenAt :: N.SockAddr -> IO Bool+canListenAt address = handle unavailable $ bracket+ (newSocket address)+ N.close+ (\sock -> N.bind sock address >> return True)+ where+ unavailable :: IOException -> IO Bool+ unavailable = const (return False) +-- | Run an action with a socket bound to the given address but not+-- listening, so that connection attempts there are refused, and nothing else+-- can listen there in the meantime. The action is given the port, a free one+-- if the address's port is 0.+withBoundSocket :: N.SockAddr -> (N.PortNumber -> IO a) -> IO a+withBoundSocket = bindNotListening False++-- | Like 'withBoundSocket', but the socket sets SO_REUSEADDR before it binds.+-- Then the bind succeeds where an earlier mock's connections at that address+-- and port are still in TIME_WAIT, but another socket that sets SO_REUSEADDR,+-- as the mocks do, can listen there in the meantime.+withBoundSocketReuseAddr :: N.SockAddr -> (N.PortNumber -> IO a) -> IO a+withBoundSocketReuseAddr = bindNotListening True++-- | Bind a socket to the given address without listening, setting+-- SO_REUSEADDR first if asked, and run the action with its port.+bindNotListening :: Bool -> N.SockAddr -> (N.PortNumber -> IO a) -> IO a+bindNotListening reuse address action = bracket (newSocket address) N.close $ \sock -> do+ when reuse $ N.setSocketOption sock N.ReuseAddr 1+ N.bind sock address+ N.socketPort sock >>= action++-- | Run an action with a socket listening on the given address that never+-- accepts a connection, and whose queue of connections waiting to be+-- accepted is full, so that connection attempts there get no answer. The+-- action is given the port, a free one if the address's port is 0, and an+-- action that closes the socket, after which connection attempts are refused.+-- Throws instead if a connection attempt gets an answer within 100 ms.+withFullListener :: N.SockAddr -> (N.PortNumber -> IO () -> IO a) -> IO a+withFullListener address action =+ bracket (newSocket address) N.close $ \sock ->+ bracket (newSocket address) N.close $ \first ->+ bracket (newSocket address) N.close $ \second -> do+ N.bind sock address+ -- Linux queues one connection more than the backlog, so two waiting+ -- connections fill a queue of backlog 1. Linux then ignores connection+ -- attempts until the queue has room. Older kernels take a connection on+ -- a queue of backlog 0 only with SYN cookies.+ N.listen sock 1+ listening <- N.getSocketName sock+ mapM_ (`N.connect` listening) [first, second]+ -- A system that queued more would answer this attempt. Fail then, rather+ -- than let a test check something else.+ answered <- bracket (newSocket address) N.close $ \probe ->+ timeout (100 * 1000) (N.connect probe listening)+ when (isJust answered) $ throwIO (userError "the listener's queue is not full")+ port <- N.socketPort sock+ action port (N.close sock)++-- | Run an action with the path of a Unix socket in the temporary directory,+-- at which nothing exists yet, so that a mock can listen there. Whatever is+-- at the path once the action is done, such as the mock's socket, is+-- removed.+withSocketPath :: (FilePath -> IO a) -> IO a+withSocketPath = bracket freshPath removePathForcibly+ where+ -- A temporary file's name is one no other file has. Remove the file, so+ -- that a socket can be bound in its place.+ freshPath = do+ tmp <- getTemporaryDirectory+ (path, file) <- openTempFile tmp "memcache-hs.sock"+ hClose file+ removeFile path+ return path++-- | A stream socket for the given address's family: TCP, or a Unix socket.+newSocket :: N.SockAddr -> IO N.Socket+newSocket address = N.socket (addressFamily address) N.Stream N.defaultProtocol++addressFamily :: N.SockAddr -> N.Family+addressFamily N.SockAddrInet6{} = N.AF_INET6+addressFamily N.SockAddrUnix{} = N.AF_UNIX+addressFamily _ = N.AF_INET++-- | Answer requests on one connection until the responses run out or the+-- script closes it, then close it. If there is a greeting, the first request+-- is answered with it, without using up a response.+serveConnection+ :: Maybe B.ByteString -> N.Socket -> IORef [MockResponse] -> IORef [B.ByteString] -> IORef Int+ -> IO ()+serveConnection greeting client remaining received closed =+ handle ignore (mapM_ (\reply -> greet reply client received closed) greeting >> clientHandler)+ `finally` N.close client+ where ignore :: SomeException -> IO () ignore = const (return ()) - -- Answer requests on one connection until the responses run out or the- -- script closes it.- clientHandler client remaining received = do+ clientHandler = do pending <- readIORef remaining case pending of [] -> return () (r:rest) -> do- request <- recvReq client- modifyIORef' received (request :)+ recordRequest client received closed writeIORef remaining rest- continue <- respond client r- when continue $ clientHandler client remaining received+ continue <- respond r+ when continue clientHandler - respond client r = case r of+ respond r = case r of Noop -> return True MR bytes -> N.sendAll client bytes >> return True- DelayMS ms r' -> threadDelay (ms * 1000) >> respond client r'+ DelayMS ms r' -> threadDelay (ms * 1000) >> respond r' CloseConnection -> return False+ Repeat bytes -> forever (N.sendAll client bytes)+ -- The connection is closed here, so that the MVar is filled only once+ -- it is closed. Closing it again, as serveConnection does, does+ -- nothing.+ ReplyThenReset bytes reset -> do+ N.sendAll client bytes+ N.setSockOpt client N.Linger (N.StructLinger 1 0)+ N.close client+ void (tryPutMVar reset ())+ return False +-- | Read the first request on a connection, the client's version check, and+-- answer it with the reply.+greet :: B.ByteString -> N.Socket -> IORef [B.ByteString] -> IORef Int -> IO ()+greet reply client received closed = do+ recordRequest client received closed+ N.sendAll client reply++-- | Read the next request and record it, or count the connection as closed if+-- the client has closed it. Requests and closed connections can be recorded+-- from several threads at once.+recordRequest :: N.Socket -> IORef [B.ByteString] -> IORef Int -> IO ()+recordRequest client received closed = do+ request <- recvReq client+ `onException` atomicModifyIORef' closed (\n -> (n + 1, ()))+ atomicModifyIORef' received (\requests -> (request : requests, ()))+ -- | Read one request: its command line and, for storage commands, the data -- block that follows it. recvReq :: N.Socket -> IO B.ByteString recvReq s = do line <- recvLine s []- case payloadLen (C.words line) of+ -- Split on spaces only, as memcached does: a raw key can contain other+ -- whitespace bytes, such as 0xa0.+ case payloadLen (filter (not . B.null) (C.split ' ' line)) of Just n -> do payload <- recvAll s n trailer <- recvLine s []
+ test/RoutingSpec.hs view
@@ -0,0 +1,437 @@+{-# LANGUAGE OverloadedStrings #-}+-- The tests of legacyServerForKey use it, although it is deprecated.+{-# OPTIONS_GHC -Wno-deprecations #-}++-- | Key routing: the ketama ring checked against libmemcached's choices, and+-- the cluster's use of it, including servers marked failed; and the router of+-- 0.4, checked against a copy of its code.+module RoutingSpec (routingSpec) where++import Control.Concurrent.MVar (newMVar, swapMVar)+import Control.Monad (void)+import qualified Data.ByteString.Char8 as C+import Data.Foldable (for_, toList)+import qualified Data.Hashable as Hashable+import Data.IORef (readIORef, writeIORef)+import Data.List (find, findIndex, group, nub, sort)+import Data.Maybe (fromMaybe, listToMaybe, maybeToList)+import Data.Time.Clock.POSIX (getPOSIXTime)+import Data.Word (Word32)+import qualified Database.Memcache.Client as Client+import Database.Memcache.Cluster (Cluster, Options (..), getServers, isAlive,+ ketamaServerForKey, legacyServerForKey,+ newCluster, setServers)+import Database.Memcache.Errors+import qualified Database.Memcache.Ketama as K+import Database.Memcache.Server (Server, failed, serverHost, serverPort, sid)+import Database.Memcache.Types (Authentication (NoAuth), Key, ServerSpec (..))+import KetamaVectors+import Test.Hspec+import Test.Hspec.QuickCheck (prop)+import Test.QuickCheck++routingSpec :: Spec+routingSpec = do+ describe "ketama ring" $ do+ it "hashes a key to the first four bytes of its MD5 digest, little-endian" $ do+ K.keyHash "foo" `shouldBe` 0xdb18bdac+ K.keyHash "key1" `shouldBe` 0x94d6adc2+ K.keyHash "SVa_]_V41)" `shouldBe` 0x1a7231c5+ K.keyHash "VDEAAAAA" `shouldBe` 0xfffcd1b5++ it "names a server by its host on port 11211 and host:port otherwise" $ do+ K.serverName "10.0.0.1" "11211" `shouldBe` "10.0.0.1"+ K.serverName "10.0.0.3" "11212" `shouldBe` "10.0.0.3:11212"+ K.serverName "localhost" "11212" `shouldBe` "localhost:11212"++ it "takes four little-endian points from the MD5 digest of each point name" $ do+ -- The digests of "10.0.0.1-0" and "10.0.0.3:11212-0".+ missingPoints "10.0.0.1" "11211" [0x2194783c, 0x36d6a85b, 0xf1edf592, 0x40a9b8cf]+ `shouldBe` []+ missingPoints "10.0.0.3" "11212" [0x0038906b, 0x901937a6, 0x4b190c65, 0x2e809329]+ `shouldBe` []++ it "shares out 160 points per server by weight" $ do+ pointCounts (ringOf (scenarioServers "fixture8_weighted"))+ `shouldBe` [176, 88, 56, 104, 296, 236, 280, 28]+ pointCounts (K.ring [("10.0.0.1", 1)]) `shouldBe` [160]++ it "rounds in single precision, giving 25 equal servers 156 points each" $ do+ pointCounts (ringOf (scenarioServers "eq25")) `shouldBe` replicate 25 156+ pointCounts (equalRing 24) `shouldBe` replicate 24 160+ pointCounts (equalRing 26) `shouldBe` replicate 26 160++ it "counts a weight below 1 as 1" $+ K.ringPoints (K.ring [("a", 0), ("b", -3)])+ `shouldBe` K.ringPoints (K.ring [("a", 1), ("b", 1)])++ for_ scenarios $ \(name, servers, keys) ->+ it ("routes keys as libmemcached does: " <> name) $ do+ let ring = ringOf servers+ hostPort i = let (host, port, _) = servers !! i in host <> ":" <> show port+ [(key, hostPort <$> K.owner ring (K.keyHash (C.pack key))) | (key, _) <- keys]+ `shouldBe` [(key, Just server) | (key, server) <- keys]++ it "gives a point two servers share to the server listed first" $ do+ -- Point 0 of "h8.example-20" and point 3 of "h256.example-34" are both+ -- 0xf53a3e63, and libmemcached gives it to whichever is listed first.+ let tiedOwners r = [o | (p, o) <- K.ringPoints r, p == 0xf53a3e63]+ h8First = ringOf [("h8.example", 11211, 1), ("h256.example", 11211, 1)]+ h256First = ringOf [("h256.example", 11211, 1), ("h8.example", 11211, 1)]+ tiedOwners h8First `shouldBe` [0, 1]+ tiedOwners h256First `shouldBe` [0, 1]+ K.owner h8First (K.keyHash "tie-10931") `shouldBe` Just 0+ K.owner h256First (K.keyHash "tie-10931") `shouldBe` Just 0++ it "wraps a hash after the last point round to the first point" $ do+ let ring = ringOf (scenarioServers "fixture8_weighted")+ firstOwner = snd <$> listToMaybe (K.ringPoints ring)+ K.keyHash "VDEAAAAA" `shouldSatisfy` (> maximum (map fst (K.ringPoints ring)))+ K.owner ring (K.keyHash "VDEAAAAA") `shouldBe` firstOwner+ K.owner ring maxBound `shouldBe` firstOwner++ it "moves only the removed server's keys when rebuilt without it" $ do+ -- libmemcached's choices with and without cache3 differ only for the+ -- keys that were on cache3.+ let withCache3 = scenarioKeys "host4"+ withoutCache3 = scenarioKeys "host4_minus_cache3"+ moved = [key | (key, server) <- withCache3, lookup key withoutCache3 /= Just server]+ moved `shouldBe` [key | (key, "cache3.example.com:11211") <- withCache3]+ -- The same holds for many more keys.+ let hostWith = hostFor (scenarioServers "host4")+ hostWithout = hostFor (scenarioServers "host4_minus_cache3")+ changes = filter (uncurry (/=)) [(hostWith key, hostWithout key) | key <- manyKeys]+ changes `shouldSatisfy` (not . null)+ map fst changes `shouldSatisfy` all (== Just "cache3.example.com")++ it "has no owner on an empty ring" $ do+ K.owner (K.ring []) 0 `shouldBe` Nothing+ K.owners (K.ring []) 0 `shouldBe` []++ let weighted = ringOf (scenarioServers "fixture8_weighted")+ it "gives a hash equal to a point to that point's owner" $ do+ let points = K.ringPoints weighted+ next = drop 1 points ++ take 1 points+ -- No two points share a value, so each point has a single owner.+ length (group (map fst points)) `shouldBe` length points+ [K.owner weighted p | (p, _) <- points] `shouldBe` [Just o | (_, o) <- points]+ [K.owner weighted (p + 1) | (p, _) <- points] `shouldBe` [Just o | (_, o) <- next]++ prop "falls back through every server once, starting with the owner" $ \hash ->+ let order = K.owners weighted hash+ in sort order === [0 .. 7] .&&. take 1 order === maybeToList (K.owner weighted hash)++ -- What libmemcached's auto-eject does, as long as the number of points+ -- per server stays the same (it does from 8 servers to 7).+ let names = map C.pack (equalHosts 8)+ withoutServer down = [name | (i, name) <- zip [0 ..] names, i /= down]+ ringsWithout = [K.ring [(name, 1) | name <- withoutServer down] | down <- [0 .. 7 :: Int]]+ prop "falls back to the server a ring rebuilt without the owner chooses" $ \hash ->+ case K.owners (equalRing 8) hash of+ down : next : _ ->+ ((withoutServer down !!) <$> K.owner (ringsWithout !! down) hash)+ === Just (names !! next)+ _ -> property False++ describe "cluster routing" $ do+ for_ [s | s@(_, servers, _) <- scenarios, all (\(_, _, w) -> w == 1) servers] $+ \(name, servers, keys) ->+ it ("routes keys as libmemcached does: " <> name) $ do+ cluster <- clusterOf servers+ routed <- mapM (route cluster . C.pack . fst) keys+ zip (map fst keys) routed `shouldBe` [(key, Just server) | (key, server) <- keys]++ it "gives a point two servers share to the server configured first" $ do+ -- The keys just below the tied point 0xf53a3e63 (see above).+ let keys = ["tie-10931", "tie-18872", "tie-235249", "tie-567986", "tie-706509"]+ h8First <- clusterOf [("h8.example", 11211, 1), ("h256.example", 11211, 1)]+ h256First <- clusterOf [("h256.example", 11211, 1), ("h8.example", 11211, 1)]+ mapM (route h8First) keys `shouldReturn` map (const (Just "h8.example:11211")) keys+ mapM (route h256First) keys `shouldReturn` map (const (Just "h256.example:11211")) keys++ it "names a server on the ring by its host in UTF-8" $ do+ -- Hosts with U+00E9 and U+20AC in them, and the UTF-8 bytes of their+ -- names written out.+ let servers = [ ("cach\233.example", "cach\195\169.example")+ , ("cache\8364.example", "cache\226\130\172.example")+ , ("10.0.1.3", "10.0.1.3") ]+ ring = K.ring [(C.pack utf8, 1) | (_, utf8) <- servers]+ expected key = (\i -> fst (servers !! i) <> ":11211") <$> K.owner ring (K.keyHash key)+ cluster <- clusterOf [(host, 11211, 1) | (host, _) <- servers]+ mapM (route cluster) manyKeys `shouldReturn` map expected manyKeys++ it "names a Unix socket on the ring by the bytes of its path" $ do+ -- Paths held a byte per Char, as the client connects to them and as+ -- parseServerSpec decodes memcached:///run/caf%C3%A9-a.sock: \195\169+ -- is U+00E9 in UTF-8. libmemcached names a socket by these bytes, so+ -- they mustn't be encoded in UTF-8 again, as a host's characters are.+ -- serverFor names each server by C.pack of its host, which keeps them.+ let sockets = [ ("/run/caf\195\169-a.sock", 11211, 1)+ , ("/run/caf\195\169-b.sock", 11211, 1)+ , ("/run/caf\195\169-c.sock", 0, 1) ]+ cluster <- clusterOf sockets+ mapM (route cluster) manyKeys `shouldReturn` map (serverFor sockets) manyKeys++ it "sends a failed server's keys to the next live server, and keeps the rest" $ do+ cluster <- clusterOf (scenarioServers "host4")+ cache3 <- serverNamed cluster "cache3.example.com:11211"+ let original = scenarioKeys "host4"+ routes = mapM (route cluster . C.pack . fst) original+ onCache3 (_, server) = server == "cache3.example.com:11211"+ markFailed cache3+ routed <- routes+ -- Where libmemcached sends them after ejecting cache3...+ routed `shouldBe` map (Just . snd) (scenarioKeys "host4_minus_cache3")+ -- ...which leaves every other key where it was.+ [r | (r, k) <- zip routed original, not (onCache3 k)]+ `shouldBe` [Just server | k@(_, server) <- original, not (onCache3 k)]+ writeIORef (failed cache3) 0+ routes `shouldReturn` map (Just . snd) original++ it "routes as a ring of only the live servers would, for many keys" $ do+ cluster <- clusterOf [(host, 11211, 1) | host <- equalHosts 8]+ down <- serverNamed cluster "10.0.1.6:11211"+ markFailed down+ let live = filter (/= "10.0.1.6") (equalHosts 8)+ ring = K.ring [(C.pack host, 1) | host <- live]+ expected key = (\i -> (live !! i) <> ":11211") <$> K.owner ring (K.keyHash key)+ routed <- mapM (route cluster) manyKeys+ routed `shouldBe` map expected manyKeys++ it "returns no server once every server is marked failed" $ do+ cluster <- clusterOf (scenarioServers "host4")+ mapM_ markFailed . toList =<< getServers cluster+ route cluster "key" `shouldReturn` Nothing+ Client.get cluster "key" `shouldThrow` (== ClientError NoServersReady)++ it "routes with ketamaServerForKey by default" $ do+ cluster <- clusterOf [(host, 11211, 1) | host <- equalHosts 8]+ markFailed =<< serverNamed cluster "10.0.1.6:11211"+ byName <- mapM (fmap (fmap serverLabel) . ketamaServerForKey cluster) manyKeys+ mapM (route cluster) manyKeys `shouldReturn` byName+ -- Every live server gets keys.+ length (nub byName) `shouldBe` 7++ it "isAlive counts a failed server as down until optsDeadRetryDelay has passed" $ do+ -- The cluster's delay is an hour, and the default 1.5 s.+ cluster <- clusterOf (scenarioServers "host4")+ server <- serverNamed cluster "cache3.example.com:11211"+ isAlive cluster server `shouldReturn` True+ now <- getPOSIXTime+ writeIORef (failed server) (now - 60)+ isAlive cluster server `shouldReturn` False+ readIORef (failed server) `shouldReturn` (now - 60)+ -- Once the delay has passed, the server is live again and the mark is+ -- cleared.+ writeIORef (failed server) (now - 7200)+ isAlive cluster server `shouldReturn` True+ readIORef (failed server) `shouldReturn` 0++ it "rebuilds its ring when the server set changes" $ do+ four <- clusterOf (scenarioServers "host4")+ three <- getServers =<< clusterOf (scenarioServers "host4_minus_cache3")+ servers <- newMVar =<< getServers four+ -- ElastiCache discovery replaces the servers in the MVar like this.+ let discovered = setServers four (Left servers)+ routes cluster = mapM (route cluster . C.pack . fst) (scenarioKeys "host4")+ routes discovered `shouldReturn` map (Just . snd) (scenarioKeys "host4")+ void (swapMVar servers three)+ routes discovered `shouldReturn` map (Just . snd) (scenarioKeys "host4_minus_cache3")+ -- The original cluster keeps its own servers and ring.+ routes four `shouldReturn` map (Just . snd) (scenarioKeys "host4")+ routes discovered `shouldReturn` map (Just . snd) (scenarioKeys "host4_minus_cache3")++ it "routes by the fixed servers that setServers gives it" $ do+ four <- clusterOf (scenarioServers "host4")+ three <- getServers =<< clusterOf (scenarioServers "host4_minus_cache3")+ let routes cluster = mapM (route cluster . C.pack . fst) (scenarioKeys "host4")+ routes (setServers four (Right three))+ `shouldReturn` map (Just . snd) (scenarioKeys "host4_minus_cache3")+ routes four `shouldReturn` map (Just . snd) (scenarioKeys "host4")++ it "rebuilds its ring when only a port changes" $ do+ -- Two servers on one host, then the second replaced by one on another+ -- port.+ let pair = [("localhost", 11211, 1), ("localhost", 11212, 1)]+ replaced = [("localhost", 11211, 1), ("localhost", 11213, 1)]+ original <- clusterOf pair+ servers <- newMVar =<< getServers original+ let discovered = setServers original (Left servers)+ mapM (route discovered) manyKeys `shouldReturn` map (serverFor pair) manyKeys+ void (swapMVar servers =<< getServers =<< clusterOf replaced)+ mapM (route discovered) manyKeys `shouldReturn` map (serverFor replaced) manyKeys++ it "routes by host when discovery returns new or reordered servers" $ do+ -- This checks the host and port of each server chosen. The next test+ -- checks that they are the Server values in the MVar.+ four <- clusterOf (scenarioServers "host4")+ -- New Server values for the same hosts...+ fresh <- getServers =<< clusterOf (scenarioServers "host4")+ -- ...and the same hosts listed the other way round.+ reordered <- getServers =<< clusterOf (reverse (scenarioServers "host4"))+ servers <- newMVar =<< getServers four+ let discovered = setServers four (Left servers)+ routes = mapM (route discovered . C.pack . fst) (scenarioKeys "host4")+ expected = map (Just . snd) (scenarioKeys "host4")+ routes `shouldReturn` expected+ void (swapMVar servers fresh)+ routes `shouldReturn` expected+ void (swapMVar servers reordered)+ routes `shouldReturn` expected++ it "routes to the servers in the MVar when new ones have the same hosts" $ do+ four <- clusterOf (scenarioServers "host4")+ servers <- newMVar =<< getServers four+ let discovered = setServers four (Left servers)+ routes = mapM (route discovered . C.pack . fst) (scenarioKeys "host4")+ routes `shouldReturn` map (Just . snd) (scenarioKeys "host4")+ -- The ring can be reused, but the router must check, and return, the+ -- new Server values: once the new cache3 is marked failed, its keys+ -- move.+ fresh <- clusterOf (scenarioServers "host4")+ void (swapMVar servers =<< getServers fresh)+ markFailed =<< serverNamed fresh "cache3.example.com:11211"+ routes `shouldReturn` map (Just . snd) (scenarioKeys "host4_minus_cache3")++ it "rebuilds its ring when a server is added at the end" $ do+ -- As when a refresh finds a new node listed after the others, and keeps+ -- the Server values of the nodes it knows.+ let four = scenarioServers "host4"+ three = take 3 four+ original <- clusterOf three+ known <- getServers original+ servers <- newMVar known+ let discovered = setServers original (Left servers)+ mapM (route discovered) manyKeys `shouldReturn` map (serverFor three) manyKeys+ added <- getServers =<< clusterOf (drop 3 four)+ void (swapMVar servers (known <> added))+ mapM (route discovered) manyKeys `shouldReturn` map (serverFor four) manyKeys++ it "spreads keys evenly over eight equal servers" $ do+ cluster <- clusterOf [(host, 11211, 1) | host <- equalHosts 8]+ routed <- mapM (route cluster) manyKeys+ let counts = map length (group (sort routed))+ share count = fromIntegral count / fromIntegral (length manyKeys) :: Double+ length counts `shouldBe` 8+ map share counts `shouldSatisfy` all (\s -> s >= 0.09 && s <= 0.16)++ -- 0.4 sorted the servers by sid when it made a cluster and when ElastiCache+ -- discovery found them. Neither sorts them now, and legacyServerForKey must+ -- still pick the server 0.4 did.+ describe "legacyServerForKey" $ do+ prop "routes keys as 0.4 did" $+ forAll legacyServers $ \servers -> ioProperty $ do+ cluster <- clusterOf [(host, port, 1) | ((host, port), _) <- servers]+ configured <- toList <$> getServers cluster+ sequence_ [markFailed server | (server, (_, True)) <- zip configured servers]+ let live = [server | (server, (_, False)) <- zip configured servers]+ -- Servers named twice are told apart by position.+ position server = findIndex ((== failed server) . failed) configured+ legacyKeys = take 1000 manyKeys+ routed <- mapM (legacyServerForKey cluster) legacyKeys+ return $ map (>>= position) routed+ === [router04 live key >>= position | key <- legacyKeys]++ it "sends every key to one of two servers" $ do+ cluster <- clusterOf [(host, 11211, 1) | host <- equalHosts 8]+ routed <- mapM (fmap (fmap serverLabel) . legacyServerForKey cluster) manyKeys+ length (nub routed) `shouldSatisfy` (<= 2)++ it "returns no server once every server is marked failed" $ do+ cluster <- clusterOf (scenarioServers "host4")+ mapM_ markFailed . toList =<< getServers cluster+ legacyServerForKey cluster "key" `shouldReturn` Nothing++-- | Look up a scenario's servers or expected routes by name.+scenarioServers :: String -> [(String, Int, Int)]+scenarioServers name = concat [servers | (n, servers, _) <- scenarios, n == name]++scenarioKeys :: String -> [(String, String)]+scenarioKeys name = concat [keys | (n, _, keys) <- scenarios, n == name]++-- | A ring of servers given as (host, port, weight), in that order.+ringOf :: [(String, Int, Int)] -> K.Ring+ringOf servers =+ K.ring [(K.serverName (C.pack host) (C.pack (show port)), weight) | (host, port, weight) <- servers]++-- | The host each key belongs to on a ring of these servers.+hostFor :: [(String, Int, Int)] -> Key -> Maybe String+hostFor servers = \key -> (hosts !!) <$> K.owner ring (K.keyHash key)+ where+ ring = ringOf servers+ hosts = [host | (host, _, _) <- servers]++-- | The server each key belongs to on a ring of these servers, as host:port.+serverFor :: [(String, Int, Int)] -> Key -> Maybe String+serverFor servers = \key -> (names !!) <$> K.owner ring (K.keyHash key)+ where+ ring = ringOf servers+ names = [host <> ":" <> show port | (host, port, _) <- servers]++-- | Hosts 10.0.1.1, 10.0.1.2, and so on.+equalHosts :: Int -> [String]+equalHosts n = ["10.0.1." <> show i | i <- [1 .. n]]++-- | A ring of @n@ hosts on port 11211, all with weight 1.+equalRing :: Int -> K.Ring+equalRing n = K.ring [(C.pack host, 1) | host <- equalHosts n]++-- | How many points each server owns, in server order.+pointCounts :: K.Ring -> [Int]+pointCounts = map length . group . sort . map snd . K.ringPoints++-- | Which of the given points are missing from the ring of a single server.+missingPoints :: C.ByteString -> C.ByteString -> [Word32] -> [Word32]+missingPoints host port = filter (`notElem` points)+ where+ points = map fst (K.ringPoints (K.ring [(K.serverName host port, 1)]))++manyKeys :: [Key]+manyKeys = [C.pack ("key:" <> show i) | i <- [1 .. 20000 :: Int]]++-- | 0.4's router, getServerForKeyDefault, with lists for vectors: of the live+-- servers, sorted by sid as 0.4 kept them, the first whose sid is below the+-- key's hash, or else the last.+router04 :: [Server] -> Key -> Maybe Server+router04 live k =+ let hashedKey = Hashable.hash k+ searchF s = sid s < hashedKey+ servers' = sort live+ in if null servers'+ then Nothing+ else Just $ fromMaybe (last servers') (find searchF servers')++-- | One to eight servers, each marked failed or not, from a few hosts and+-- ports, so that some lists name a server twice.+legacyServers :: Gen [((String, Int), Bool)]+legacyServers = do+ n <- choose (1, 8)+ vectorOf n ((,) <$> server <*> frequency [(3, pure False), (1, pure True)])+ where+ server = (,) <$> elements ["cache" <> show i <> ".example" | i <- [1 .. 10 :: Int]]+ <*> elements [11211, 11212]++-- | A cluster of servers given as (host, port, weight); weights are ignored.+-- No connections are made until a request is sent.+clusterOf :: [(String, Int, Int)] -> IO Cluster+clusterOf servers =+ newCluster [ServerSpec host (show port) NoAuth | (host, port, _) <- servers]+ Client.def { optsDeadRetryDelay = 3600000 }++-- | The server the cluster's default router picks for a key, as host:port.+route :: Cluster -> Key -> IO (Maybe String)+route cluster key = fmap serverLabel <$> optsGetServerForKey Client.def cluster key++serverLabel :: Server -> String+serverLabel server = serverHost server <> ":" <> serverPort server++serverNamed :: Cluster -> String -> IO Server+serverNamed cluster name = do+ servers <- toList <$> getServers cluster+ case filter ((== name) . serverLabel) servers of+ [server] -> return server+ found -> fail ("expected one server " <> name <> ", found " <> show (length found))++-- | Mark a server failed now, as running out of retries does.+markFailed :: Server -> IO ()+markFailed server = writeIORef (failed server) =<< getPOSIXTime
test/Spec.hs view
@@ -2,937 +2,4125 @@ module Main where -import MockServer-import qualified Database.Memcache.Auth as Auth-import qualified Database.Memcache.Client as Client-import qualified Database.Memcache.ElastiCacheClient as EC-import Database.Memcache.Errors-import qualified Database.Memcache.Errors as E-import Database.Memcache.Socket-import Database.Memcache.Types (ServerSpec (..), parseServerSpec)-import Test.Hspec-import Test.Hspec.QuickCheck (modifyMaxSuccess, prop)-import Test.QuickCheck-import qualified Data.ByteString as B-import qualified Data.ByteString.Base64 as Base64-import qualified Data.ByteString.Char8 as C-import Data.ByteString.Unsafe (unsafePackMallocCStringLen)-import qualified Network.Socket as N-import qualified Network.Socket.ByteString as N-import Control.Concurrent (forkIO, threadDelay)-import Control.Exception (bracket)-import Control.Monad (forM_, void)-import Data.Either (isLeft)-import Data.List (isInfixOf)-import Foreign.Marshal.Alloc (mallocBytes)-import System.Timeout (timeout)--main :: IO ()-main = hspec $ do- describe "meta command encoding" $ do- it "leaves safe keys unchanged and encodes unsafe keys" $ do- fmap wireKey (encodeKey "safe-key") `shouldBe` Right "safe-key"- fmap wireKey (encodeKey "a b") `shouldBe` Right (Base64.encode "a b")-- it "terminates arithmetic commands" $- arithmeticRequest Incr (wire "counter") 0 1 maxBound 0 `shouldSatisfy` C.isSuffixOf "\r\n"-- it "validates encoded key length" $- encodeKey (C.replicate 251 'a') `shouldSatisfy` isLeft-- it "rejects empty keys" $- encodeKey "" `shouldSatisfy` isLeft-- it "accepts a key of exactly 250 bytes unchanged" $- encodeKey (C.replicate 250 'a') `shouldBe` Right (WireKey (C.replicate 250 'a') False)-- it "base64-encodes keys containing control bytes or DEL" $ do- fmap wireB64 (encodeKey "tab\tkey") `shouldBe` Right True- fmap wireB64 (encodeKey "line\nkey") `shouldBe` Right True- fmap wireB64 (encodeKey "del\DELkey") `shouldBe` Right True-- it "sends bytes above 0x7f unencoded" $- encodeKey "caf\xc3\xa9" `shouldBe` Right (WireKey "caf\xc3\xa9" False)-- it "rejects a key whose base64 form exceeds 250 bytes" $ do- -- 186 bytes encode to 248 base64 bytes; 187 bytes encode to 252.- fmap (B.length . wireKey) (encodeKey (C.replicate 186 ' ')) `shouldBe` Right 248- encodeKey (C.replicate 187 ' ') `shouldSatisfy` isLeft-- prop "round-trips every key it accepts" $- forAll keyGen $ \key -> case encodeKey key of- Right (WireKey bytes True) ->- Base64.decode bytes === Right key .&&. B.length bytes <= 250- Right (WireKey bytes False) ->- bytes === key .&&. not (B.any unsafeByte key)- Left _ ->- property (B.length key > 250 || (B.any unsafeByte key && base64Length key > 250))-- prop "only ever sends safe bytes as a key" $- forAll keyGen $ \key -> case encodeKey key of- Right (WireKey bytes _) -> not (B.any unsafeByte bytes)- Left _ -> True-- prop "encodes each non-storage request as exactly one line" $- forAll keyGen $ \key -> case encodeKey key of- Left _ -> property True- Right w -> conjoin- [ oneLine (getRequest w)- , oneLine (getManyRequest w)- , oneLine (gatRequest w 30)- , oneLine (touchRequest w 30)- , oneLine (deleteRequest w 7)- , oneLine (arithmeticRequest Incr w 1 2 30 7)- ]-- it "rejects over-long keys without contacting the server" $- withClient [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \client -> do- Client.get client (C.replicate 251 'a') `shouldThrow` isKeyTooLong- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)-- it "rejects unsafe stats keys before sending them" $ do- client <- Client.newClient [Client.def] Client.def- Client.stats client (Just "items\r\nset injected 0 0 1\r\nx")- `shouldThrow` isKeyTooLong-- it "encodes all core request forms" $ do- getRequest (wire "key") `shouldBe` "mg key v f c\r\n"- getManyRequest (wire "key") `shouldBe` "mg key v f c k q\r\n"- gatRequest (wire "key") 30 `shouldBe` "mg key v f c T30\r\n"- touchRequest (wire "key") 30 `shouldBe` "mg key T30 c\r\n"- deleteRequest (wire "key") 9 `shouldBe` "md key C9\r\n"- flushRequest Nothing `shouldBe` "flush_all\r\n"- flushRequest (Just 30) `shouldBe` "flush_all 30\r\n"- versionRequest `shouldBe` "version\r\n"- statsRequest Nothing `shouldBe` "stats\r\n"- statsRequest (Just "key") `shouldBe` "stats key\r\n"- quitRequest `shouldBe` "quit\r\n"- noOpRequest `shouldBe` "mn\r\n"- authRequest "user pass" `shouldBe` "set auth 0 0 9\r\nuser pass\r\n"-- it "flags encoded keys on every keyed request" $ do- let encoded = Base64.encode "a b"- getRequest (wire "a b") `shouldBe` "mg " <> encoded <> " b v f c\r\n"- deleteRequest (wire "a b") 0 `shouldBe` "md " <> encoded <> " b\r\n"- modifyRequest Append (wire "a b") "x" 0 `shouldBe` "ms " <> encoded <> " 1 MA b c\r\nx\r\n"-- it "places the binary-key flag after the data length" $- storeRequest Set (wire "a b") "value" 3 20 0- `shouldBe` "ms " <> Base64.encode "a b" <> " 5 F3 T20 MS b c\r\nvalue\r\n"-- it "encodes store modes and CAS" $ do- storeRequest Add (wire "key") "v" 1 2 0 `shouldBe` "ms key 1 F1 T2 ME c\r\nv\r\n"- storeRequest Replace (wire "key") "v" 1 2 5 `shouldBe` "ms key 1 F1 T2 C5 MR c\r\nv\r\n"- modifyRequest Append (wire "key") "xyz" 5 `shouldBe` "ms key 3 MA C5 c\r\nxyz\r\n"- modifyRequest Prepend (wire "key") "xyz" 0 `shouldBe` "ms key 3 MP c\r\nxyz\r\n"-- it "encodes arithmetic with auto-creation" $- arithmeticRequest Decr (wire "n") 10 2 60 7 `shouldBe` "ma n D2 C7 MD v c N60 J10\r\n"-- it "omits arithmetic defaults at the maximum expiration" $- arithmeticRequest Incr (wire "key") 0 1 maxBound 0- `shouldBe` "ma key D1 MI v c\r\n"-- it "omits zero CAS values" $- storeRequest Set (wire "key") "v" 0 0 0 `shouldBe` "ms key 1 F0 T0 MS c\r\nv\r\n"-- -- Most encoders write into a single buffer for speed; check them all- -- against the plainest possible spelling of each request.- prop "encodes every request as the concatenation of its pieces" $- forAll wireKeyGen $ \key ->- forAll requestValueGen $ \value ->- forAll ((,,) <$> numberGen <*> numberGen <*> numberGen) $ \(flags, expiry, version) ->- forAll ((,) <$> numberGen <*> numberGen) $ \(initial, delta) ->- forAll ((,) <$> elements [Set, Add, Replace, Append, Prepend] <*> elements [Incr, Decr]) $ \(mode, arith) ->- let keyArgs = wireKey key <> binaryFlag key- binaryFlag k = if wireB64 k then " b" else ""- casArg n = if n == 0 then "" else " C" <> decimal n- decimal :: Show a => a -> B.ByteString- decimal = C.pack . show- modeLetter m = case m of- Set -> "S"- Add -> "E"- Replace -> "R"- Append -> "A"- Prepend -> "P"- arithLetter m = case m of- Incr -> "I"- Decr -> "D"- in conjoin- [ getRequest key === B.concat ["mg ", keyArgs, " v f c\r\n"]- , getManyRequest key === B.concat ["mg ", keyArgs, " v f c k q\r\n"]- , gatRequest key expiry === B.concat ["mg ", keyArgs, " v f c T", decimal expiry, "\r\n"]- , touchRequest key expiry === B.concat ["mg ", keyArgs, " T", decimal expiry, " c\r\n"]- , storeRequest mode key value flags expiry version === B.concat- [ "ms ", wireKey key, " ", decimal (B.length value), " F", decimal flags- , " T", decimal expiry, casArg version, " M", modeLetter mode, binaryFlag key- , " c\r\n", value, "\r\n" ]- , modifyRequest mode key value version === B.concat- [ "ms ", wireKey key, " ", decimal (B.length value), " M", modeLetter mode- , binaryFlag key, casArg version, " c\r\n", value, "\r\n" ]- , arithmeticRequest arith key initial delta expiry version === B.concat- [ "ma ", keyArgs, " D", decimal delta, casArg version, " M", arithLetter arith, " v c"- , if expiry == maxBound then "" else " N" <> decimal expiry <> " J" <> decimal initial- , "\r\n" ]- , deleteRequest key version === B.concat ["md ", keyArgs, casArg version, "\r\n"]- , flushRequest (Just expiry) === B.concat ["flush_all ", decimal expiry, "\r\n"]- , flushRequest Nothing === "flush_all\r\n"- , statsRequest Nothing === "stats\r\n"- , statsRequest (Just (wireKey key)) === B.concat ["stats ", wireKey key, "\r\n"]- , versionRequest === "version\r\n"- , quitRequest === "quit\r\n"- , noOpRequest === "mn\r\n"- , authRequest value === B.concat ["set auth 0 0 ", decimal (B.length value), "\r\n", value, "\r\n"]- ]-- describe "response parsing" $ do- it "parses values and protocol errors" $ do- recvResponseFrom "VA 3 f7 c9 O2\r\nabc\r\n" `shouldReturn` Response "VA" ["3", "f7", "c9", "O2"] "abc"- (recvResponseFrom "SERVER_ERROR broken\r\n" >>= responseStatus) `shouldThrow` isServerError- (recvResponseFrom "CLIENT_ERROR bad\r\n" >>= responseStatus) `shouldThrow` isBadCommand-- it "maps protocol statuses" $ do- (recvResponseFrom "VA 0 f1 c2\r\n\r\n" >>= responseStatus) `shouldReturn` NoError- (recvResponseFrom "EN\r\n" >>= responseStatus) `shouldReturn` ErrKeyNotFound- (recvResponseFrom "NF\r\n" >>= responseStatus) `shouldReturn` ErrKeyNotFound- (recvResponseFrom "HD\r\n" >>= responseStatus) `shouldReturn` NoError- (recvResponseFrom "EX\r\n" >>= responseStatus) `shouldReturn` ErrKeyExists- (recvResponseFrom "NS\r\n" >>= responseStatus) `shouldReturn` ErrItemNotStored- (recvResponseFrom "MN\r\n" >>= responseStatus) `shouldReturn` NoError- (recvResponseFrom "VERSION 1.6.0\r\n" >>= responseStatus) `shouldReturn` NoError- (recvResponseFrom "STAT curr_items 2\r\n" >>= responseStatus) `shouldReturn` NoError-- it "maps server error text back to operation statuses" $ do- (recvResponseFrom "SERVER_ERROR object too large for cache\r\n" >>= responseStatus)- `shouldThrow` (== OpError ErrValueTooLarge)- (recvResponseFrom "SERVER_ERROR out of memory storing object\r\n" >>= responseStatus)- `shouldThrow` (== OpError ErrOutOfMemory)- (recvResponseFrom "CLIENT_ERROR cannot increment or decrement non-numeric value\r\n" >>= responseStatus)- `shouldThrow` (== OpError ErrValueNonNumeric)-- it "rejects unknown and empty response lines" $ do- (recvResponseFrom "ERROR\r\n" >>= responseStatus) `shouldThrow` isUnknownOp- (recvResponseFrom "XX\r\n" >>= responseStatus) `shouldThrow` isUnknownStatus- recvResponseFrom "\r\n" `shouldThrow` isUnknownPkt-- it "rejects a value without its trailing line break" $- recvResponseFrom "VA 3\r\nabcX\r\n" `shouldThrow` isBadLength-- it "reads flags, CAS and numbers from responses" $ do- responseCas (Response "HD" ["c42"] "") `shouldBe` 42- responseCas (Response "HD" [] "") `shouldBe` 0- responseCas (Response "HD" ["cxyz"] "") `shouldBe` 0- responseFlags (Response "VA" ["5", "f7", "c9"] "") `shouldBe` 7- responseFlags (Response "VA" ["5"] "") `shouldBe` 0- responseNumber (Response "VA" ["2"] "42") `shouldBe` 42- responseNumber (Response "VA" ["3"] "abc") `shouldBe` 0-- describe "connection buffering" $ do- it "keeps bytes after a parsed response" $- bracket- (N.socketPair N.AF_UNIX N.Stream 0)- (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "VA 3 f1 c2\r\nabc\r\nMN\r\n"- recvResponse connection `shouldReturn` Response "VA" ["3", "f1", "c2"] "abc"- recvResponse connection `shouldReturn` Response "MN" [] ""-- it "handles a value split across packets" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "VA 5 f1 c2\r\nhe" >> N.sendAll right "llo\r\n"- recvResponse connection `shouldReturn` Response "VA" ["5", "f1", "c2"] "hello"-- modifyMaxSuccess (const 50) $- prop "parses responses however they are split across packets" $- forAll ((,) <$> valueGen <*> listOf (choose (0, 1.0 :: Double))) $ \(value, cuts) ->- ioProperty $ do- let bytes = "VA " <> C.pack (show (B.length value)) <> " f1 c2\r\n"- <> value <> "\r\nHD c3\r\n"- pieces = splitAtFractions cuts bytes- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ mapM_ (\piece -> N.sendAll right piece >> threadDelay 200) pieces- first <- recvResponse connection- second <- recvResponse connection- return $ first === Response "VA" [C.pack (show (B.length value)), "f1", "c2"] value- .&&. second === Response "HD" ["c3"] ""-- it "reads through a delimiter split across packets" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "CONFIG x\r\nEN" >> threadDelay 1000 >> N.sendAll right "D\r\nrest"- recvRawUntil "END\r\n" connection `shouldReturn` "CONFIG x\r\nEND\r\n"- recvExact connection 4 `shouldReturn` "rest"-- it "reports EOF in a value" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "VA 5\r\nhe" >> N.close right- recvResponse connection `shouldThrow` isUnexpectedEOF-- it "rejects unreasonably large value lengths" $- recvResponseFrom "VA 134217729\r\n" `shouldThrow` isBadLength-- it "rejects an unterminated response line" $- recvResponseFrom (C.replicate (16 * 1024) 'x') `shouldThrow` isBadLength-- describe "getMany" $ do- it "matches echoed keys and skips misses" $- withServerAndClient [MR "VA 5 f1 c11 kone\r\nfirst\r\n", Noop, MR "MN\r\n"] $ \server client -> do- Client.getMany client ["one", "two"] `shouldReturn` [("one", ("first", 1, 11))]- mockRequests server `shouldReturn`- ["version\r\n", "mg one v f c k q\r\n", "mg two v f c k q\r\n", "mn\r\n"]-- it "maps encoded keys back to the caller's key" $ do- let encoded = Base64.encode "a b"- withClient [MR ("VA 3 f0 c4 k" <> encoded <> " b\r\nabc\r\n"), MR "MN\r\n"] $ \client ->- Client.getMany client ["a b"] `shouldReturn` [("a b", ("abc", 0, 4))]-- it "tells a plain key from another key's base64 form" $- -- " " is sent, and echoed, as IA== with the b flag.- withClient [MR "VA 1 f0 c1 kIA==\r\na\r\n", MR "VA 1 f0 c2 kIA== b\r\nb\r\n", MR "MN\r\n"] $ \client ->- Client.getMany client ["IA==", " "] `shouldReturn` [("IA==", ("a", 0, 1)), (" ", ("b", 0, 2))]-- describe "client operations" $ do- it "gets a hit" $- withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)- mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n"]-- it "returns Nothing for a miss" $- withClient [MR "EN\r\n"] $ \client ->- Client.get client "key" `shouldReturn` Nothing-- it "rejects a reply meant for a different command" $- withClient [MR "HD\r\n"] $ \client ->- Client.get client "key" `shouldThrow` isUnexpectedResponse-- it "rejects an unknown reply" $- withClient [MR "XX\r\n"] $ \client ->- Client.get client "key" `shouldThrow` isUnknownStatus-- it "gets and touches a value" $- withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n", MR "EN\r\n"] $ \server client -> do- Client.gat client "key" 60 `shouldReturn` Just ("foo", 2, 7)- Client.gat client "key" 60 `shouldReturn` Nothing- mockRequests server `shouldReturn`- ["version\r\n", "mg key v f c T60\r\n", "mg key v f c T60\r\n"]-- it "touches a value" $- withClient [MR "HD c3\r\n", MR "EN\r\n"] $ \client -> do- Client.touch client "key" 60 `shouldReturn` Just 3- Client.touch client "key" 60 `shouldReturn` Nothing-- it "stores and returns the server CAS" $- withServerAndClient [MR "HD c9\r\n"] $ \server client -> do- Client.set client "key" "foo" 2 30 `shouldReturn` 9- mockRequests server `shouldReturn` ["version\r\n", "ms key 3 F2 T30 MS c\r\nfoo\r\n"]-- it "reports a set the server did not store" $- withClient [MR "NS\r\n"] $ \client ->- Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrItemNotStored)-- it "adds only when the key is new" $- withClient [MR "HD c5\r\n", MR "NS\r\n"] $ \client -> do- Client.add client "key" "v" 0 0 `shouldReturn` Just 5- Client.add client "key" "v" 0 0 `shouldReturn` Nothing-- it "replaces only when the key exists and the CAS matches" $- withClient [MR "HD c8\r\n", MR "NS\r\n", MR "NF\r\n", MR "EX\r\n"] $ \client -> do- Client.replace client "key" "v" 0 0 0 `shouldReturn` Just 8- Client.replace client "key" "v" 0 0 0 `shouldReturn` Nothing- Client.replace client "key" "v" 0 0 4 `shouldReturn` Nothing- Client.replace client "key" "v" 0 0 4 `shouldReturn` Nothing-- it "compares and swaps" $- withServerAndClient [MR "HD c3\r\n", MR "EX\r\n", MR "NF\r\n"] $ \server client -> do- Client.cas client "key" "v" 0 0 2 `shouldReturn` Just 3- Client.cas client "key" "v" 0 0 2 `shouldReturn` Nothing- Client.cas client "key" "v" 0 0 2 `shouldReturn` Nothing- mockRequests server `shouldReturn`- "version\r\n" : replicate 3 "ms key 1 F0 T0 C2 MS c\r\nv\r\n"-- it "deletes a present key" $- withClient [MR "HD\r\n"] $ \client ->- Client.delete client "key" 0 `shouldReturn` True-- it "reports a delete of a missing key or with a stale CAS" $- withClient [MR "NF\r\n", MR "EX\r\n"] $ \client -> do- Client.delete client "key" 0 `shouldReturn` False- Client.delete client "key" 5 `shouldReturn` False-- it "does not retry a command the server rejected" $- withClient [MR "CLIENT_ERROR bad\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \client ->- Client.get client "key" `shouldThrow` isBadCommand-- it "reads version" $- withClient [MR ("VERSION " <> supportedVersion <> "\r\n")] $ \client ->- Client.version client `shouldReturn` supportedVersion-- it "collects stats until END, keeping values that contain spaces" $- withClient [MR "STAT curr_items 2\r\nSTAT rusage_user 0.1 0.2\r\nEND\r\n"] $ \client -> do- result <- Client.stats client Nothing- result `shouldSatisfy` allStats-- it "allows stats arguments containing spaces" $- withServerAndClient [MR "END\r\n"] $ \server client -> do- result <- Client.stats client (Just "cachedump 1 100")- map snd result `shouldBe` [[]]- mockRequests server `shouldReturn` ["version\r\n", "stats cachedump 1 100\r\n"]-- it "keeps non-STAT stats lines" $- withClient [MR "ITEM key [3 b; 0 s]\r\nEND\r\n"] $ \client -> do- result <- Client.stats client (Just "cachedump 1 1")- map snd result `shouldBe` [[("ITEM", "key [3 b; 0 s]")]]-- it "returns Nothing when arithmetic cannot create the item" $- withClient [MR "NS\r\n"] $ \client ->- Client.increment client "counter" 0 1 30 0 `shouldReturn` Nothing-- it "returns the new counter value and CAS" $- withServerAndClient [MR "VA 1 c2\r\n0\r\n"] $ \server client -> do- Client.decrement client "counter" 10 1 60 0 `shouldReturn` Just (0, 2)- mockRequests server `shouldReturn` ["version\r\n", "ma counter D1 MD v c N60 J10\r\n"]-- it "reports arithmetic on a non-numeric value" $- withClient [MR "CLIENT_ERROR cannot increment or decrement non-numeric value\r\n"] $ \client ->- Client.increment client "key" 0 1 0 0 `shouldThrow` (== OpError ErrValueNonNumeric)-- it "returns Nothing appending to a missing key" $- withClient [MR "NS\r\n"] $ \client ->- Client.append client "key" "more" 0 `shouldReturn` Nothing-- it "prepends to an existing key" $- withServerAndClient [MR "HD c4\r\n"] $ \server client -> do- Client.prepend client "key" "pre" 0 `shouldReturn` Just 4- mockRequests server `shouldReturn` ["version\r\n", "ms key 3 MP c\r\npre\r\n"]-- it "flushes" $- withServerAndClient [MR "OK\r\n"] $ \server client -> do- Client.flush client (Just 10) `shouldReturn` ()- mockRequests server `shouldReturn` ["version\r\n", "flush_all 10\r\n"]-- it "reports an oversized value as a status, not a protocol failure" $- withClient [MR "SERVER_ERROR object too large for cache\r\n"] $ \client ->- Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrValueTooLarge)-- it "reports a server out of memory as a status" $- withClient [MR "SERVER_ERROR out of memory storing object\r\n"] $ \client ->- Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrOutOfMemory)-- it "returns no stats on a reset reply" $- withClient [MR "RESET\r\n"] $ \client -> do- result <- Client.stats client Nothing- map snd result `shouldBe` [[]]-- it "sends quit" $- withServerAndClient [Noop] $ \server client -> do- Client.quit client- requests <- waitForRequests server 2- requests `shouldBe` ["version\r\n", "quit\r\n"]-- -- The value is one byte over the 1 GiB limit. In the test of each- -- operation, the client's server is a port that is bound but not listening,- -- so a client that tried to send the value would throw the refused- -- connection instead: the check has to come before the client connects.- describe "oversized values" $ beforeAll overLimitValue $ do- it "rejects an over-limit value from set without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.set client "key" value 0 0 `shouldThrow` (== OpError ErrValueTooLarge)-- it "rejects an over-limit value from add without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.add client "key" value 0 0 `shouldThrow` (== OpError ErrValueTooLarge)-- it "rejects an over-limit value from replace without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.replace client "key" value 0 0 0 `shouldThrow` (== OpError ErrValueTooLarge)-- it "rejects an over-limit value from cas without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.cas client "key" value 0 0 1 `shouldThrow` (== OpError ErrValueTooLarge)-- it "rejects an over-limit value from append without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.append client "key" value 0 `shouldThrow` (== OpError ErrValueTooLarge)-- it "rejects an over-limit value from prepend without sending anything" $ \value ->- withRefusingClient $ \client ->- Client.prepend client "key" value 0 `shouldThrow` (== OpError ErrValueTooLarge)-- -- This client routes every key to no server, so a value that it lets- -- through fails with NoServersReady and is never sent. It still rejects- -- the over-limit value, which shows that it checks values first.- it "does not reject a value of exactly the limit" $ \value -> do- client <- Client.newClient [] Client.def { Client.optsGetServerForKey = \_ _ -> return Nothing }- Client.set client "key" value 0 0 `shouldThrow` (== OpError ErrValueTooLarge)- Client.set client "key" (B.take valueLimit value) 0 0 `shouldThrow` (== ClientError NoServersReady)-- describe "retries and failover" $ do- let fast = Client.def- { Client.optsServerTimeout = 300- , Client.optsFailRetryDelay = 10- , Client.optsDeadRetryDelay = 300- }-- it "retries after the server drops the connection" $- withServerAndClientOpts fast [CloseConnection, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)- -- The retry's new connection checks the version again.- mockRequests server `shouldReturn`- ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]-- it "retries after an operation times out" $- withServerAndClientOpts fast [DelayMS 350 Noop, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \_ client ->- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)-- it "marks a server dead once its retries run out, then revives it" $- withServerAndClientOpts fast [CloseConnection, CloseConnection, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do- -- The default of two retries means two attempts in total.- Client.get client "key" `shouldThrow` isUnexpectedEOF- mockRequests server `shouldReturn`- ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]- Client.get client "key" `shouldThrow` (== ClientError NoServersReady)- threadDelay 400000- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)-- it "checks the version again on the connection a retry opens" $- withMockServer [supported, CloseConnection, MR "VERSION 1.6.9\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] fast- Client.get client "key" `shouldThrow` refused "VERSION 1.6.9"- mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n", "version\r\n"]-- describe "authenticate" $ do- it "accepts credentials once the server confirms auth-file mode" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "CLIENT_ERROR unauthenticated\r\nSTORED\r\n"- Auth.authenticate connection (Auth.Auth "user" "pass") `shouldReturn` ()- recvExactly right (B.length expectedAuthRequests) `shouldReturn` expectedAuthRequests-- it "reports rejected credentials" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "CLIENT_ERROR unauthenticated\r\nCLIENT_ERROR authentication failure\r\n"- Auth.authenticate connection (Auth.Auth "user" "pass") `shouldThrow` (== OpError TextAuthFail)-- it "refuses to send credentials when the server is not auth-gated" $- bracket- (N.socketPair N.AF_UNIX N.Stream 0)- (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right "VERSION 1.6\r\n"- Auth.authenticate connection (Auth.Auth "user" "pass") `shouldThrow` isInvalidAuthentication-- it "rejects credentials containing delimiters before contacting the server" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, _) -> do- connection <- newConnection left- Auth.authenticate connection (Auth.Auth "us er" "pass") `shouldThrow` isInvalidAuthentication-- it "does nothing without credentials" $- bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, _) -> do- connection <- newConnection left- Auth.authenticate connection Auth.NoAuth `shouldReturn` ()-- it "validates credential delimiters" $ do- Auth.validateCredentials "user" "pass" `shouldBe` Right ()- mapM_ (\c -> Auth.validateCredentials ("us" <> c <> "er") "pass" `shouldSatisfy` isLeft)- [" ", "\t", "\r", "\n", ":"]- mapM_ (\c -> Auth.validateCredentials "user" ("pa" <> c <> "ss") `shouldSatisfy` isLeft)- [" ", "\t", "\r", "\n"]-- it "authenticates each new client connection" $- withMockServer (authExchange ++ [supported, MR "VA 3 f2 c7\r\nfoo\r\n"]) $ \server -> do- client <- Client.newClient [withCredentials server] Client.def- Client.get client "key" `shouldReturn` Just ("foo", 2, 7)- mockRequests server `shouldReturn` authRequests ++ ["version\r\n", "mg key v f c\r\n"]-- -- Older servers don't read the data block of the meta set this client- -- sends, so they run the value's bytes as commands. The client must not- -- send them anything but its version check (and, with credentials, the- -- authentication exchange before it). Each mock has a reply left over for- -- a request that should never come, so that one would be recorded.- describe "server version check" $ do- forM_ oldVersions $ \v ->- it ("refuses memcached " <> C.unpack v <> ", sending it nothing but version") $- withMockServer [MR ("VERSION " <> v <> "\r\n"), MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` refused ("VERSION " <> v)- mockRequests server `shouldReturn` ["version\r\n"]-- -- Every operation that sends a value, not only set, must be refused- -- before the value reaches the server.- forM_ valueOps $ \(name, op) ->- it ("refuses memcached 1.6.9 for " <> name <> ", sending it nothing but version") $- withMockServer [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- op client `shouldThrow` refused "VERSION 1.6.9"- mockRequests server `shouldReturn` ["version\r\n"]-- forM_ newVersions $ \v ->- it ("accepts memcached " <> C.unpack v) $- withMockServer [MR ("VERSION " <> v <> "\r\n"), MR "HD c9\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0 `shouldReturn` 9- mockRequests server `shouldReturn` ["version\r\n", "ms key 5 F0 T0 MS c\r\nvalue\r\n"]-- forM_ unreadableReplies $ \reply ->- it ("refuses a server that answers version with " <> show reply) $- withMockServer [MR (reply <> "\r\n"), MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` refused reply- mockRequests server `shouldReturn` ["version\r\n"]-- -- Such as a server in auth-file mode, when the client has no credentials.- it "throws a CLIENT_ERROR reply to version as it would for any request" $- withMockServer [MR "CLIENT_ERROR unauthenticated\r\n", MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0- `shouldThrow` (== ProtocolError (E.BadCommand "unauthenticated"))- mockRequests server `shouldReturn` ["version\r\n"]-- it "throws an ERROR reply to version as it would for any request" $- withMockServer [MR "ERROR\r\n", MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` (== ProtocolError (UnknownOp "ERROR"))- mockRequests server `shouldReturn` ["version\r\n"]-- -- A server failure is retried, as it would be for any request, but each- -- attempt still sends nothing but version.- it "throws a SERVER_ERROR reply to version as it would for any request" $- withMockServer [MR "SERVER_ERROR busy\r\n", MR "SERVER_ERROR busy\r\n", MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def { Client.optsFailRetryDelay = 10 }- Client.set client "key" "value" 0 0 `shouldThrow` (== ProtocolError (E.ServerError "busy"))- mockRequests server `shouldReturn` ["version\r\n", "version\r\n"]-- -- The server's version won't change on a retry: a refusal is a client- -- fault, so the operation fails at once and the server stays alive.- it "neither retries a refused server nor marks it dead" $- withMockServer [MR "VERSION 1.6.9\r\n", MR "VERSION 1.6.9\r\n", MR "HD c1\r\n", MR "HD c1\r\n"] $ \server -> do- client <- Client.newClient [mockServerSpec server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` refused "VERSION 1.6.9"- mockRequests server `shouldReturn` ["version\r\n"]- -- A server marked dead would fail with NoServersReady without being- -- asked again.- Client.get client "key" `shouldThrow` refused "VERSION 1.6.9"- mockRequests server `shouldReturn` ["version\r\n", "version\r\n"]-- describe "with credentials" $ do- forM_ oldVersions $ \v ->- it ("asks again after authenticating, and refuses memcached " <> C.unpack v) $- withMockServer (authExchange ++ [MR ("VERSION " <> v <> "\r\n"), MR "HD c1\r\n"]) $ \server -> do- client <- Client.newClient [withCredentials server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` refused ("VERSION " <> v)- mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]-- forM_ newVersions $ \v ->- it ("asks again after authenticating, and accepts memcached " <> C.unpack v) $- withMockServer (authExchange ++ [MR ("VERSION " <> v <> "\r\n"), MR "HD c9\r\n"]) $ \server -> do- client <- Client.newClient [withCredentials server] Client.def- Client.set client "key" "value" 0 0 `shouldReturn` 9- mockRequests server `shouldReturn`- authRequests ++ ["version\r\n", "ms key 5 F0 T0 MS c\r\nvalue\r\n"]-- forM_ unreadableReplies $ \reply ->- it ("refuses a server that answers version with " <> show reply <> " after authenticating") $- withMockServer (authExchange ++ [MR (reply <> "\r\n"), MR "HD c1\r\n"]) $ \server -> do- client <- Client.newClient [withCredentials server] Client.def- Client.set client "key" "value" 0 0 `shouldThrow` refused reply- mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]-- describe "server URIs" $ do- it "parses host and port" $- parseServerSpec "memcached://cache.example:1234" `shouldBe`- Right (ServerSpec "cache.example" "1234" Auth.NoAuth)-- it "defaults the port and host" $ do- parseServerSpec "memcached://cache.example" `shouldBe`- Right (ServerSpec "cache.example" "11211" Auth.NoAuth)- parseServerSpec "memcached://" `shouldBe`- Right (ServerSpec "127.0.0.1" "11211" Auth.NoAuth)-- it "parses credentials" $- parseServerSpec "memcached://user:secret@cache.example:1234" `shouldBe`- Right (ServerSpec "cache.example" "1234" (Auth.Auth "user" "secret"))-- it "ignores incomplete credentials" $ do- parseServerSpec "memcached://user@cache.example" `shouldBe`- Right (ServerSpec "cache.example" "11211" Auth.NoAuth)- parseServerSpec "memcached://:secret@cache.example" `shouldBe`- Right (ServerSpec "cache.example" "11211" Auth.NoAuth)-- it "rejects other schemes and invalid URIs" $ do- parseServerSpec "http://cache.example" `shouldSatisfy` isLeft- parseServerSpec "not a uri" `shouldSatisfy` isLeft-- describe "clusters" $ do- it "fails over to another server once one is marked dead" $- withMockMemcached supportedVersion [MR "VA 1 f0 c1\r\nA\r\n", CloseConnection, CloseConnection] $ \a ->- withMockMemcached supportedVersion [MR "VA 1 f0 c1\r\nB\r\n", CloseConnection, CloseConnection] $ \b -> do- client <- Client.newClient [mockServerSpec a, mockServerSpec b] fastFailover- first <- Client.get client "key"- -- The server that answered now drops two connections in a row, which- -- uses up its retries and marks it dead...- Client.get client "key" `shouldThrow` isUnexpectedEOF- -- ...so the same key is now served by the other server.- second <- Client.get client "key"- map (fmap fst3) [first, second] `shouldSatisfy` (`elem` [[Just "A", Just "B"], [Just "B", Just "A"]])-- it "throws NoServersReady once every server is dead" $- withMockMemcached supportedVersion [CloseConnection, CloseConnection] $ \a -> do- client <- Client.newClient [mockServerSpec a] fastFailover- Client.get client "key" `shouldThrow` isUnexpectedEOF- Client.get client "key" `shouldThrow` (== ClientError NoServersReady)- Client.flush client Nothing `shouldThrow` (== ClientError NoServersReady)- Client.version client `shouldThrow` (== ClientError NoServersReady)-- it "flushes every server" $- withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->- withMockMemcached supportedVersion [MR "OK\r\n"] $ \b -> do- client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def- Client.flush client Nothing- mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]- mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]-- it "collects stats from every server" $- withMockMemcached supportedVersion [MR "STAT pid 1\r\nEND\r\n"] $ \a ->- withMockMemcached supportedVersion [MR "STAT pid 2\r\nEND\r\n"] $ \b -> do- client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def- results <- Client.stats client Nothing- concatMap snd results `shouldMatchList` [("pid", "1"), ("pid", "2")]-- it "asks a single server for the version" $- withMockMemcached supportedVersion [supported] $ \a ->- withMockMemcached supportedVersion [supported] $ \b -> do- client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def- Client.version client `shouldReturn` supportedVersion- -- The connection's own version check, then the operation.- requests <- sequence [mockRequests a, mockRequests b]- requests `shouldMatchList` [["version\r\n", "version\r\n"], []]-- describe "ElastiCache discovery" $ do- it "rejects an endpoint that is not a configuration endpoint" $- isLeft (EC.parseConfigurationEndpoint "memcached://cache.example:11211") `shouldBe` True-- it "discovers nodes with the config command" $- withMockMemcached supportedVersion [MR "VA 3 f0 c1\r\nfoo\r\n"] $ \node -> do- let config = "CONFIG cluster 0 " <> nodeList node <> "END\r\n"- withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config, Noop] $ \cfg -> do- client <- inTime (EC.newClient Client.def hour (endpoint cfg))- Client.get client "key" `shouldReturn` Just ("foo", 0, 1)- -- The connection's own version check, then discovery's.- waitForRequests cfg 4 `shouldReturn`- ["version\r\n", "version\r\n", "config get cluster\r\n", "quit\r\n"]-- -- Engines before 1.4.14 keep the node list in a key, which discovery- -- used to read with a plain get. Like any server older than memcached- -- 1.6.10, they are now refused before discovery sends anything.- it "refuses a configuration endpoint older than memcached 1.6.10" $- withMockServer [MR "VA 3 f0 c1\r\nfoo\r\n"] $ \node -> do- let value = "VALUE AmazonElastiCache:cluster 0 " <> nodeList node <> "END\r\n"- withMockServer [MR "VERSION 1.4.5\r\n", MR value, Noop] $ \cfg -> do- inTime (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow` refused "VERSION 1.4.5"- -- Discovery's connection is refused after asking for the version.- -- Quitting its client then opens another, which the mock answers- -- with the node list; that is no version either, so it is refused- -- too.- mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n"]-- where- recvResponseFrom bytes =- bracket- (N.socketPair N.AF_UNIX N.Stream 0)- (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do- connection <- newConnection left- _ <- forkIO $ N.sendAll right bytes- recvResponse connection- wire key = either (error . show) id (encodeKey key)- isKeyTooLong (ClientError (KeyTooLong _)) = True- isKeyTooLong _ = False- isInvalidAuthentication (ClientError (InvalidAuthentication _)) = True- isInvalidAuthentication _ = False- isServerError (ProtocolError (E.ServerError _)) = True- isServerError _ = False- isBadCommand (ProtocolError (E.BadCommand _)) = True- isBadCommand _ = False- isUnexpectedEOF (ProtocolError (UnexpectedEOF _)) = True- isUnexpectedEOF _ = False- isBadLength (ProtocolError (BadLength _)) = True- isBadLength _ = False- isUnknownOp (ProtocolError (UnknownOp _)) = True- isUnknownOp _ = False- isUnknownStatus (ProtocolError (UnknownStatus _)) = True- isUnknownStatus _ = False- isUnknownPkt (ProtocolError (UnknownPkt _)) = True- isUnknownPkt _ = False- isUnexpectedResponse (ProtocolError (UnexpectedResponse _)) = True- isUnexpectedResponse _ = False- closePair (left, right) = N.close left >> N.close right- allStats [(_, [ ("curr_items", "2"), ("rusage_user", "0.1 0.2") ])] = True- allStats _ = False- expectedAuthRequests = "version\r\nset auth 0 0 9\r\nuser pass\r\n"- -- A server in auth-file mode rejects the client's first version request,- -- then accepts its credentials.- authExchange = [MR "CLIENT_ERROR unauthenticated\r\n", MR "STORED\r\n"]- authRequests = ["version\r\n", "set auth 0 0 9\r\nuser pass\r\n"]- withCredentials server = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }- supported = MR ("VERSION " <> supportedVersion <> "\r\n")- -- Versions as memcached reports them, with Ubuntu's suffix and a- -- release candidate's; 1.10.0 and 1.6.9 catch comparing them as text.- oldVersions =- ["1.6.9", "1.5.22", "1.4.25", "1.6.0", "1.4.25 Ubuntu", "1.6.9-rc1", "0.10.10"]- newVersions =- ["1.6.10", "1.6.14", "1.6.24", "1.6.100", "1.7.0", "1.10.0", "2.0.0", "1.6.10-rc1", "1.6.24 Ubuntu"]- unreadableReplies =- [ "VERSION", "VERSION 1.6", "VERSION 1.7", "VERSION 1.6.x", "VERSION v1.6.24"- , "VERSION 1..6.24", "VERSION 1.6.+10", "VERSION -1.6.24", "HD", "END", "HD 1.6.24" ]- -- The operations that send a value, each storing one that would run as- -- a command on a server older than 1.6.10.- valueOps :: [(String, Client.Client -> IO ())]- valueOps =- [ ("set", \c -> void $ Client.set c "key" injection 0 0)- , ("add", \c -> void $ Client.add c "key" injection 0 0)- , ("replace", \c -> void $ Client.replace c "key" injection 0 0 0)- , ("cas", \c -> void $ Client.cas c "key" injection 0 0 1)- , ("append", \c -> void $ Client.append c "key" injection 0)- , ("prepend", \c -> void $ Client.prepend c "key" injection 0)- ]- injection :: B.ByteString- injection = "set injected 0 0 3\r\nbad"- fastFailover = Client.def { Client.optsFailRetryDelay = 10, Client.optsDeadRetryDelay = 60000 }- -- Discovery has no timeout of its own, so a client that sent the mock- -- something it has no reply for would wait for one forever.- inTime action = timeout 2000000 action >>= maybe (fail "timed out") return- fst3 (value, _, _) = value- hour = 3600 * 1000000- -- The endpoint check only looks for ".cfg" somewhere in the URI, so put it- -- in the path and keep the host a plain loopback address.- endpoint cfg = either error id $- EC.parseConfigurationEndpoint ("memcached://127.0.0.1:" <> show (mockPort cfg) <> "/x.cfg")- -- A config reply body listing one node, from its length line onwards.- nodeList node =- let body = "1\n127.0.0.1|127.0.0.1|" <> C.pack (show (mockPort node)) <> "\n"- in C.pack (show (B.length body)) <> "\r\n" <> body <> "\r\n"---- | Run an action with a client connected to a fresh mock server, which--- reports 'supportedVersion' to each connection's version check.-withClient :: [MockResponse] -> (Client.Client -> IO a) -> IO a-withClient responses action = withServerAndClient responses (const action)--withServerAndClient :: [MockResponse] -> (MockServer -> Client.Client -> IO a) -> IO a-withServerAndClient = withServerAndClientOpts Client.def--withServerAndClientOpts- :: Client.Options -> [MockResponse] -> (MockServer -> Client.Client -> IO a) -> IO a-withServerAndClientOpts opts responses action = withMockMemcached supportedVersion responses $ \server -> do- client <- Client.newClient [mockServerSpec server] opts- action server client---- | The version that mock servers report unless a test says otherwise: one--- the client accepts.-supportedVersion :: B.ByteString-supportedVersion = "1.6.24"---- | Whether an exception is the client refusing a server that answered its--- version check with the given reply: 'UnknownOp', with a message that quotes--- the reply and names the minimum version.-refused :: B.ByteString -> MemcacheError -> Bool-refused reply (ProtocolError (UnknownOp message)) =- show reply `isInfixOf` message && "1.6.10 or later" `isInfixOf` message-refused _ _ = False---- | Run an action with a client whose server is a port on 127.0.0.1 that is--- bound but not listening, so every attempt to connect to it is refused.-withRefusingClient :: (Client.Client -> IO a) -> IO a-withRefusingClient action =- bracket (N.socket N.AF_INET N.Stream N.defaultProtocol) N.close $ \sock -> do- N.bind sock (N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1)))- port <- N.socketPort sock- client <- Client.newClient [ServerSpec "127.0.0.1" (show port) Auth.NoAuth] Client.def- action client---- | The largest value memcached can store, 1 GiB.-valueLimit :: Int-valueLimit = 1024 * 1024 * 1024---- | A value one byte longer than 'valueLimit'. It is malloc'd outside the GHC--- heap and never written, and the client reads only its length, so it takes--- next to no memory, unless malloc fills it, as glibc's does when--- MALLOC_PERTURB_ is set. It is freed once nothing refers to it.-overLimitValue :: IO B.ByteString-overLimitValue = do- let len = valueLimit + 1- ptr <- mallocBytes len- unsafePackMallocCStringLen (ptr, len)---- | Wait (up to a second) for the mock to have received at least @n@ requests.-waitForRequests :: MockServer -> Int -> IO [B.ByteString]-waitForRequests server n = do- result <- timeout 1000000 go- maybe (mockRequests server) return result- where- go = do- requests <- mockRequests server- if length requests >= n then return requests else threadDelay 1000 >> go---- | Receive exactly @n@ bytes from a socket.-recvExactly :: N.Socket -> Int -> IO B.ByteString-recvExactly _ 0 = return B.empty-recvExactly socket n = do- chunk <- N.recv socket n- if B.null chunk- then return B.empty- else (chunk <>) <$> recvExactly socket (n - B.length chunk)---- | Keys of up to 260 arbitrary bytes, biased towards safe ones.-keyGen :: Gen B.ByteString-keyGen = B.pack <$> do- len <- frequency [(4, choose (1, 64)), (1, choose (180, 260))]- vectorOf len (frequency [(8, choose (0x21, 0x7e)), (1, arbitrary)])--valueGen :: Gen B.ByteString-valueGen = B.pack <$> do- len <- choose (0, 3000)- vectorOf len arbitrary---- | Keys the encoder accepts, plain and base64.-wireKeyGen :: Gen WireKey-wireKeyGen = keyGen `suchThatMap` (either (const Nothing) Just . encodeKey)---- | Request values, sometimes over the size above which--- 'Data.ByteString.Builder.byteString' stops copying.-requestValueGen :: Gen B.ByteString-requestValueGen = frequency- [ (4, valueGen)- , (1, B.replicate <$> choose (8 * 1024, 20 * 1024) <*> arbitrary)- ]---- | Numbers of any size, with the edges of their range picked often.-numberGen :: (Arbitrary a, Bounded a, Num a) => Gen a-numberGen = frequency [(1, elements [0, 1, maxBound]), (4, arbitrary)]--unsafeByte :: (Ord a, Num a) => a -> Bool-unsafeByte c = c <= 0x20 || c == 0x7f--base64Length :: B.ByteString -> Int-base64Length key = 4 * ((B.length key + 2) `div` 3)---- | A request line: ends in CRLF and contains no other line break.-oneLine :: B.ByteString -> Property-oneLine request =- counterexample (show request) $- C.count '\n' request === 1 .&&. C.count '\r' request === 1 .&&.- property ("\r\n" `B.isSuffixOf` request)---- | Split bytes at the given fractions of their length.-splitAtFractions :: [Double] -> B.ByteString -> [B.ByteString]-splitAtFractions cuts bytes = go 0 (map toIndex cuts) bytes- where- toIndex f = floor (f * fromIntegral (B.length bytes))- go _ [] rest = [rest]- go offset (i:is) rest- | i <= offset = go offset is rest- | otherwise =- let (now, later) = B.splitAt (i - offset) rest- in now : go i is later+import FileLimit+import MockServer+import RoutingSpec+import qualified Database.Memcache.Auth as Auth+import qualified Database.Memcache.Client as Client+import qualified Database.Memcache.Cluster as Cluster+import qualified Database.Memcache.ElastiCacheClient as EC+import Database.Memcache.Errors+import qualified Database.Memcache.Errors as E+import qualified Database.Memcache.Server as Server+import Database.Memcache.Socket+import Database.Memcache.Types (ServerSpec (..), parseServerSpec)+import Test.Hspec+import Test.Hspec.QuickCheck (modifyMaxSuccess, prop)+import Test.QuickCheck+import qualified Data.ByteString as B+import qualified Data.ByteString.Base64 as Base64+import qualified Data.ByteString.Char8 as C+import Data.ByteString.Unsafe (unsafePackMallocCStringLen)+import Data.Char (isAscii, isPrint)+import qualified Network.Socket as N+import qualified Network.Socket.ByteString as N+import Control.Concurrent (MVar, forkIO, getNumCapabilities, killThread,+ myThreadId, newEmptyMVar, putMVar, readMVar,+ takeMVar, threadDelay, tryPutMVar, yield)+import Control.Exception (MaskingState (..), bracket, displayException, finally,+ getMaskingState, mask_, try)+import Control.Monad (forM_, forever, replicateM_, unless, void, when,+ (>=>))+import Data.Either (isLeft, isRight)+import Data.Foldable (toList)+import Data.IORef (atomicModifyIORef', newIORef, readIORef,+ writeIORef)+import Data.List (find, isInfixOf, isPrefixOf, nub)+import Foreign.Marshal.Alloc (mallocBytes)+import GHC.Clock (getMonotonicTime)+import GHC.Conc (ThreadStatus (..), threadStatus)+import System.IO.Error (isDoesNotExistError)+import System.Timeout (timeout)+import UnliftIO.Async (Async, AsyncCancelled (..), async, asyncOn,+ asyncThreadId, cancel, mapConcurrently, wait)+import UnliftIO.Exception (StringException (..), tryIO)++main :: IO ()+main = hspec $ do+ describe "meta command encoding" $ do+ it "leaves safe keys unchanged and encodes unsafe keys" $ do+ fmap wireKey (encodeKey "safe-key") `shouldBe` Right "safe-key"+ fmap wireKey (encodeKey "a b") `shouldBe` Right (Base64.encode "a b")++ it "terminates arithmetic commands" $+ arithmeticRequest Incr (wire "counter") 0 1 maxBound 0 `shouldSatisfy` C.isSuffixOf "\r\n"++ it "validates encoded key length" $+ encodeKey (C.replicate 251 'a') `shouldSatisfy` isLeft++ it "rejects empty keys" $+ encodeKey "" `shouldSatisfy` isLeft++ it "accepts a key of exactly 250 bytes unchanged" $+ encodeKey (C.replicate 250 'a') `shouldBe` Right (WireKey (C.replicate 250 'a') False)++ it "base64-encodes keys containing control bytes or DEL" $ do+ fmap wireB64 (encodeKey "tab\tkey") `shouldBe` Right True+ fmap wireB64 (encodeKey "line\nkey") `shouldBe` Right True+ fmap wireB64 (encodeKey "del\DELkey") `shouldBe` Right True++ it "sends bytes above 0x7f unencoded" $+ encodeKey "caf\xc3\xa9" `shouldBe` Right (WireKey "caf\xc3\xa9" False)++ it "rejects a key whose base64 form exceeds 250 bytes" $ do+ -- 186 bytes encode to 248 base64 bytes; 187 bytes encode to 252.+ fmap (B.length . wireKey) (encodeKey (C.replicate 186 ' ')) `shouldBe` Right 248+ encodeKey (C.replicate 187 ' ') `shouldSatisfy` isLeft++ prop "round-trips every key it accepts" $+ forAll keyGen $ \key -> case encodeKey key of+ Right (WireKey bytes True) ->+ Base64.decode bytes === Right key .&&. B.length bytes <= 250+ Right (WireKey bytes False) ->+ bytes === key .&&. not (B.any unsafeByte key)+ Left _ ->+ property (B.length key > 250 || (B.any unsafeByte key && base64Length key > 250))++ prop "only ever sends safe bytes as a key" $+ forAll keyGen $ \key -> case encodeKey key of+ Right (WireKey bytes _) -> not (B.any unsafeByte bytes)+ Left _ -> True++ prop "encodes each non-storage request as exactly one line" $+ forAll keyGen $ \key -> case encodeKey key of+ Left _ -> property True+ Right w -> conjoin+ [ oneLine (getRequest w)+ , oneLine (getManyRequest w 7)+ , oneLine (gatRequest w 30)+ , oneLine (touchRequest w 30)+ , oneLine (deleteRequest w 7)+ , oneLine (deleteManyRequest w 7)+ , oneLine (arithmeticRequest Incr w 1 2 30 7)+ ]++ prop "encodes batch requests as the single-key request plus an opaque" $+ forAll ((,,,) <$> keyGen <*> valueGen <*> arbitrary <*> arbitrary) $ \(key, value, flags, expiry) ->+ forAll arbitrary $ \opaque ->+ case encodeKey key of+ Left _ -> property True+ Right w -> conjoin+ [ getManyRequest w opaque === addFlags (" q O" <> C.pack (show opaque)) (getRequest w)+ , setManyRequest w value flags expiry opaque+ === addFlags (" O" <> C.pack (show opaque)) (storeRequest Set w value flags expiry 0)+ , deleteManyRequest w opaque === addFlags (" O" <> C.pack (show opaque)) (deleteRequest w 0)+ ]++ it "rejects over-long keys without contacting the server" $+ withClient [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \client -> do+ Client.get client (C.replicate 251 'a') `shouldThrow` isKeyTooLong+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ it "rejects unsafe stats keys before sending them" $ do+ client <- Client.newClient [Client.def] Client.def+ Client.stats client (Just "items\r\nset injected 0 0 1\r\nx")+ `shouldThrow` isKeyTooLong++ it "encodes all core request forms" $ do+ getRequest (wire "key") `shouldBe` "mg key v f c\r\n"+ getManyRequest (wire "key") 3 `shouldBe` "mg key v f c q O3\r\n"+ gatRequest (wire "key") 30 `shouldBe` "mg key v f c T30\r\n"+ touchRequest (wire "key") 30 `shouldBe` "mg key T30 c\r\n"+ deleteRequest (wire "key") 9 `shouldBe` "md key C9\r\n"+ deleteManyRequest (wire "key") 3 `shouldBe` "md key O3\r\n"+ flushRequest Nothing `shouldBe` "flush_all\r\n"+ flushRequest (Just 30) `shouldBe` "flush_all 30\r\n"+ versionRequest `shouldBe` "version\r\n"+ statsRequest Nothing `shouldBe` "stats\r\n"+ statsRequest (Just "key") `shouldBe` "stats key\r\n"+ quitRequest `shouldBe` "quit\r\n"+ noOpRequest `shouldBe` "mn\r\n"+ authRequest "user pass" `shouldBe` "set auth 0 0 9\r\nuser pass\r\n"++ it "flags encoded keys on every keyed request" $ do+ let encoded = Base64.encode "a b"+ getRequest (wire "a b") `shouldBe` "mg " <> encoded <> " b v f c\r\n"+ deleteRequest (wire "a b") 0 `shouldBe` "md " <> encoded <> " b\r\n"+ modifyRequest Append (wire "a b") "x" 0 `shouldBe` "ms " <> encoded <> " 1 MA b c\r\nx\r\n"+ getManyRequest (wire "a b") 0 `shouldBe` "mg " <> encoded <> " b v f c q O0\r\n"+ setManyRequest (wire "a b") "x" 0 0 0 `shouldBe` "ms " <> encoded <> " 1 F0 T0 MS b c O0\r\nx\r\n"+ deleteManyRequest (wire "a b") 0 `shouldBe` "md " <> encoded <> " b O0\r\n"++ it "places the binary-key flag after the data length" $+ storeRequest Set (wire "a b") "value" 3 20 0+ `shouldBe` "ms " <> Base64.encode "a b" <> " 5 F3 T20 MS b c\r\nvalue\r\n"++ it "encodes store modes and CAS" $ do+ storeRequest Add (wire "key") "v" 1 2 0 `shouldBe` "ms key 1 F1 T2 ME c\r\nv\r\n"+ storeRequest Replace (wire "key") "v" 1 2 5 `shouldBe` "ms key 1 F1 T2 C5 MR c\r\nv\r\n"+ modifyRequest Append (wire "key") "xyz" 5 `shouldBe` "ms key 3 MA C5 c\r\nxyz\r\n"+ modifyRequest Prepend (wire "key") "xyz" 0 `shouldBe` "ms key 3 MP c\r\nxyz\r\n"+ setManyRequest (wire "key") "v" 1 2 5 `shouldBe` "ms key 1 F1 T2 MS c O5\r\nv\r\n"++ it "encodes arithmetic with auto-creation" $+ arithmeticRequest Decr (wire "n") 10 2 60 7 `shouldBe` "ma n D2 C7 MD v c N60 J10\r\n"++ it "omits arithmetic defaults at the maximum expiration" $+ arithmeticRequest Incr (wire "key") 0 1 maxBound 0+ `shouldBe` "ma key D1 MI v c\r\n"++ it "omits zero CAS values" $+ storeRequest Set (wire "key") "v" 0 0 0 `shouldBe` "ms key 1 F0 T0 MS c\r\nv\r\n"++ -- Most encoders write into a single buffer for speed; check them all+ -- against the plainest possible spelling of each request.+ prop "encodes every request as the concatenation of its pieces" $+ forAll wireKeyGen $ \key ->+ forAll requestValueGen $ \value ->+ forAll ((,,) <$> numberGen <*> numberGen <*> numberGen) $ \(flags, expiry, version) ->+ forAll ((,,) <$> numberGen <*> numberGen <*> numberGen) $ \(initial, delta, opaque) ->+ forAll ((,) <$> elements [Set, Add, Replace, Append, Prepend] <*> elements [Incr, Decr]) $ \(mode, arith) ->+ let keyArgs = wireKey key <> binaryFlag key+ binaryFlag k = if wireB64 k then " b" else ""+ casArg n = if n == 0 then "" else " C" <> decimal n+ decimal :: Show a => a -> B.ByteString+ decimal = C.pack . show+ modeLetter m = case m of+ Set -> "S"+ Add -> "E"+ Replace -> "R"+ Append -> "A"+ Prepend -> "P"+ arithLetter m = case m of+ Incr -> "I"+ Decr -> "D"+ in conjoin+ [ getRequest key === B.concat ["mg ", keyArgs, " v f c\r\n"]+ , getManyRequest key opaque === B.concat ["mg ", keyArgs, " v f c q O", decimal opaque, "\r\n"]+ , gatRequest key expiry === B.concat ["mg ", keyArgs, " v f c T", decimal expiry, "\r\n"]+ , touchRequest key expiry === B.concat ["mg ", keyArgs, " T", decimal expiry, " c\r\n"]+ , storeRequest mode key value flags expiry version === B.concat+ [ "ms ", wireKey key, " ", decimal (B.length value), " F", decimal flags+ , " T", decimal expiry, casArg version, " M", modeLetter mode, binaryFlag key+ , " c\r\n", value, "\r\n" ]+ , setManyRequest key value flags expiry opaque === B.concat+ [ "ms ", wireKey key, " ", decimal (B.length value), " F", decimal flags+ , " T", decimal expiry, " MS", binaryFlag key, " c O", decimal opaque+ , "\r\n", value, "\r\n" ]+ , modifyRequest mode key value version === B.concat+ [ "ms ", wireKey key, " ", decimal (B.length value), " M", modeLetter mode+ , binaryFlag key, casArg version, " c\r\n", value, "\r\n" ]+ , arithmeticRequest arith key initial delta expiry version === B.concat+ [ "ma ", keyArgs, " D", decimal delta, casArg version, " M", arithLetter arith, " v c"+ , if expiry == maxBound then "" else " N" <> decimal expiry <> " J" <> decimal initial+ , "\r\n" ]+ , deleteRequest key version === B.concat ["md ", keyArgs, casArg version, "\r\n"]+ , deleteManyRequest key opaque === B.concat ["md ", keyArgs, " O", decimal opaque, "\r\n"]+ , flushRequest (Just expiry) === B.concat ["flush_all ", decimal expiry, "\r\n"]+ , flushRequest Nothing === "flush_all\r\n"+ , statsRequest Nothing === "stats\r\n"+ , statsRequest (Just (wireKey key)) === B.concat ["stats ", wireKey key, "\r\n"]+ , versionRequest === "version\r\n"+ , quitRequest === "quit\r\n"+ , noOpRequest === "mn\r\n"+ , authRequest value === B.concat ["set auth 0 0 ", decimal (B.length value), "\r\n", value, "\r\n"]+ ]++ describe "response parsing" $ do+ it "parses values and protocol errors" $ do+ recvResponseFrom "VA 3 f7 c9 O2\r\nabc\r\n" `shouldReturn` Response "VA" ["3", "f7", "c9", "O2"] "abc"+ (recvResponseFrom "SERVER_ERROR broken\r\n" >>= responseStatus) `shouldThrow` isServerError+ (recvResponseFrom "CLIENT_ERROR bad\r\n" >>= responseStatus) `shouldThrow` isBadCommand++ it "maps protocol statuses" $ do+ (recvResponseFrom "VA 0 f1 c2\r\n\r\n" >>= responseStatus) `shouldReturn` NoError+ (recvResponseFrom "EN\r\n" >>= responseStatus) `shouldReturn` ErrKeyNotFound+ (recvResponseFrom "NF\r\n" >>= responseStatus) `shouldReturn` ErrKeyNotFound+ (recvResponseFrom "HD\r\n" >>= responseStatus) `shouldReturn` NoError+ (recvResponseFrom "EX\r\n" >>= responseStatus) `shouldReturn` ErrKeyExists+ (recvResponseFrom "NS\r\n" >>= responseStatus) `shouldReturn` ErrItemNotStored+ (recvResponseFrom "MN\r\n" >>= responseStatus) `shouldReturn` NoError+ (recvResponseFrom "VERSION 1.6.0\r\n" >>= responseStatus) `shouldReturn` NoError+ (recvResponseFrom "STAT curr_items 2\r\n" >>= responseStatus) `shouldReturn` NoError++ it "maps server error text back to operation statuses" $ do+ (recvResponseFrom "SERVER_ERROR object too large for cache\r\n" >>= responseStatus)+ `shouldThrow` (== OpError ErrValueTooLarge)+ (recvResponseFrom "SERVER_ERROR out of memory storing object\r\n" >>= responseStatus)+ `shouldThrow` (== OpError ErrOutOfMemory)+ (recvResponseFrom "CLIENT_ERROR cannot increment or decrement non-numeric value\r\n" >>= responseStatus)+ `shouldThrow` (== OpError ErrValueNonNumeric)++ -- Every out-of-memory error in memcached's text protocol. It capitalises+ -- two: for a large value it runs out of memory while reading, and for a+ -- counter it can't create.+ forM_+ [ "out of memory storing object", "out of memory", "Out of memory allocating new item"+ , "Out of memory during read", "out of memory reading request"+ , "out of memory writing get response", "out of memory writing stats" ] $ \text ->+ it ("maps SERVER_ERROR " <> C.unpack text <> " to ErrOutOfMemory") $+ (recvResponseFrom ("SERVER_ERROR " <> text <> "\r\n") >>= responseStatus)+ `shouldThrow` (== OpError ErrOutOfMemory)++ it "rejects unknown and empty response lines" $ do+ (recvResponseFrom "ERROR\r\n" >>= responseStatus) `shouldThrow` isUnknownOp+ (recvResponseFrom "XX\r\n" >>= responseStatus) `shouldThrow` isUnknownStatus+ recvResponseFrom "\r\n" `shouldThrow` isUnknownPkt++ -- Byte 0xa0 is part of many UTF-8 characters, such as the Cyrillic Er+ -- (D0 A0) here.+ it "splits response lines on spaces only" $ do+ recvResponseFrom "VA 1 f0 c23 k\xd0\xa0\xd0\xbe\r\nr\r\n"+ `shouldReturn` Response "VA" ["1", "f0", "c23", "k\xd0\xa0\xd0\xbe"] "r"+ recvResponseFrom "HD c1 kx\t\v\f\r\xa0y\r\n" `shouldReturn` Response "HD" ["c1", "kx\t\v\f\r\xa0y"] ""+ recvResponseFrom "HD c1 \r\n" `shouldReturn` Response "HD" ["c1"] ""+ recvResponseFrom "HD c1\n" `shouldReturn` Response "HD" ["c1"] ""++ -- memcached echoes a key after a space, raw or as base64 and a b flag.+ prop "keeps intact the key echo of every key encodeKey accepts" $+ forAll separatorKeyGen $ \key -> case encodeKey key of+ Left _ -> property True+ Right w -> ioProperty $ do+ let echo = ("k" <> wireKey w) : ["b" | wireB64 w]+ value <- recvResponseFrom (C.unwords ("VA 1 f0 c1" : echo) <> "\r\nv\r\n")+ stored <- recvResponseFrom (C.unwords ("HD c2" : echo) <> "\r\n")+ return $ responseTokens value === ["1", "f0", "c1"] ++ echo+ .&&. responseTokens stored === "c2" : echo++ it "rejects a value without its trailing line break" $+ recvResponseFrom "VA 3\r\nabcX\r\n" `shouldThrow` isBadLength++ it "reads flags, CAS and numbers from responses" $ do+ responseCas (Response "HD" ["c42"] "") `shouldBe` 42+ responseCas (Response "HD" [] "") `shouldBe` 0+ responseCas (Response "HD" ["cxyz"] "") `shouldBe` 0+ responseFlags (Response "VA" ["5", "f7", "c9"] "") `shouldBe` 7+ responseFlags (Response "VA" ["5"] "") `shouldBe` 0+ responseNumber (Response "VA" ["2"] "42") `shouldBe` 42+ responseNumber (Response "VA" ["3"] "abc") `shouldBe` 0++ prop "reads every CAS, flags and counter value written in decimal" $+ forAll ((,) <$> wordGen <*> wordGen) $ \(cas, flags) ->+ let response = Response "VA" ["1", "f" <> asDecimal flags, "c" <> asDecimal cas] (asDecimal cas)+ in (responseCas response, responseFlags response, responseNumber response)+ === (cas, flags, cas)++ it "reads numbers a stock memcached never sends as invalid" $ do+ -- Most of these would read as a non-zero number if accepted.+ let malformed = ["-1", "+5", "(5)", "0x10", "1e3", "5x", "", "18446744073709551616"]+ mapM_ (\n -> responseCas (Response "HD" ["c" <> n] "") `shouldBe` 0) malformed+ mapM_ (\n -> responseNumber (Response "VA" ["1"] n) `shouldBe` 0) (malformed ++ [" 5", "5 ", "5\r\n"])+ mapM_ (\n -> responseFlags (Response "HD" ["f" <> n] "") `shouldBe` 0) ["-1", "(5)", "4294967296"]++ prop "reads numbers too large for their type as invalid rather than wrapping them" $+ forAll (choose (0, 2 ^ (40 :: Int))) $ \excess ->+ let tooBig bits = asDecimal (2 ^ (bits :: Int) + excess :: Integer)+ in responseCas (Response "HD" ["c" <> tooBig 64] "") === 0+ .&&. responseFlags (Response "HD" ["f" <> tooBig 32] "") === 0+ .&&. responseNumber (Response "VA" ["1"] (tooBig 64)) === 0++ it "rejects a value length memcached never sends" $+ mapM_ (\n -> recvResponseFrom ("VA " <> n <> "\r\nabc\r\n") `shouldThrow` isBadLength)+ ["(3)", "+3", "-3", "0x3", "3x", "18446744073709551619"]++ it "rejects a value reply with no length" $+ mapM_ (\reply -> recvResponseFrom reply `shouldThrow` isBadLength) ["VA\r\n", "VA \r\n", "VA\r\nabc\r\n"]++ -- Reply lines are split on spaces alone, so a token can also hold other+ -- whitespace, which a lenient parser would skip around a number.+ it "reads a number that shares its token with other whitespace as invalid" $ do+ let padded n = [n <> "\t", "\t" <> n, n <> "\v", n <> "\f", n <> "\r", n <> "\xa0"]+ forM_ (padded "7") $ \n -> do+ response <- recvResponseFrom ("HD c" <> n <> " f" <> n <> "\r\n")+ (responseCas response, responseFlags response) `shouldBe` (0, 0)+ forM_ (padded "3") $ \n ->+ recvResponseFrom ("VA " <> n <> "\r\nabc\r\n") `shouldThrow` isBadLength++ describe "connection buffering" $ do+ it "keeps bytes after a parsed response" $+ bracket+ (N.socketPair N.AF_UNIX N.Stream 0)+ (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "VA 3 f1 c2\r\nabc\r\nMN\r\n"+ recvResponse connection `shouldReturn` Response "VA" ["3", "f1", "c2"] "abc"+ recvResponse connection `shouldReturn` Response "MN" [] ""++ it "handles a value split across packets" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "VA 5 f1 c2\r\nhe" >> N.sendAll right "llo\r\n"+ recvResponse connection `shouldReturn` Response "VA" ["5", "f1", "c2"] "hello"++ modifyMaxSuccess (const 50) $+ prop "parses responses however they are split across packets" $+ forAll ((,) <$> valueGen <*> listOf (choose (0, 1.0 :: Double))) $ \(value, cuts) ->+ ioProperty $ do+ let bytes = "VA " <> C.pack (show (B.length value)) <> " f1 c2\r\n"+ <> value <> "\r\nHD c3\r\n"+ pieces = splitAtFractions cuts bytes+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ mapM_ (\piece -> N.sendAll right piece >> threadDelay 200) pieces+ first <- recvResponse connection+ second <- recvResponse connection+ return $ first === Response "VA" [C.pack (show (B.length value)), "f1", "c2"] value+ .&&. second === Response "HD" ["c3"] ""++ it "reads through a delimiter split across packets" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "CONFIG x\r\nEN" >> threadDelay 1000 >> N.sendAll right "D\r\nrest"+ recvRawUntil "END\r\n" connection `shouldReturn` "CONFIG x\r\nEND\r\n"+ recvExact connection 4 `shouldReturn` "rest"++ -- The client reads a line, and with it the first 2,045 bytes of the first+ -- reply, before the rest is sent. So its 4 KiB reads are out of step with+ -- the reply, and the reply's END comes in a read that crosses 1 MiB. The+ -- stream ends after the second reply, so that a read with no limit ends+ -- too.+ it "reads through a delimiter that ends 1 MiB in, but no further without one" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ let reply = C.replicate (1024 * 1024 - 5) 'x' <> "END\r\n"+ lineRead <- newEmptyMVar+ sent <- newEmptyMVar+ _ <- forkIO $ (N.sendAll right ("a\r\n" <> B.take 2045 reply) >> takeMVar lineRead+ >> N.sendAll right (B.drop 2045 reply <> C.replicate (1024 * 1024 + 8192) 'x')+ >> N.shutdown right N.ShutdownSend)+ `finally` putMVar sent ()+ recvLine connection `shouldReturn` "a\r"+ putMVar lineRead ()+ recvRawUntil "END\r\n" connection `shouldReturn` reply+ recvRawUntil "END\r\n" connection `shouldThrow` isBadLength+ takeMVar sent++ it "reports EOF in a value" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "VA 5\r\nhe" >> N.close right+ recvResponse connection `shouldThrow` isUnexpectedEOF++ it "rejects unreasonably large value lengths" $+ recvResponseFrom "VA 134217729\r\n" `shouldThrow` isBadLength++ it "rejects an unterminated response line" $+ recvResponseFrom (C.replicate (16 * 1024) 'x') `shouldThrow` isBadLength++ describe "idle check" $ do+ -- A read can take more than the reply it is for. The rest then waits in+ -- the connection's buffer, where the socket doesn't show it.+ it "rejects bytes waiting in the connection's buffer" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ N.sendAll right "MN\r\nMN\r\n"+ recvResponse connection `shouldReturn` Response "MN" [] ""+ checkIdle connection `shouldThrow` isUnexpectedResponse+ recvResponse connection `shouldReturn` Response "MN" [] ""+ checkIdle connection++ -- The check fails on a reset that nothing has read with the error that+ -- reading would get, not as on a connection the server closed. The mock+ -- answers the mn, then resets the connection.+ it "reports a reset that nothing has read as an IOException" $ do+ reset <- newEmptyMVar+ withServerAndClient [ReplyThenReset "MN\r\n" reset] $ \_ client -> do+ [s] <- toList <$> Cluster.getServers client+ Server.withSocket s $ \c -> do+ sendRaw c "mn\r\n"+ recvResponse c `shouldReturn` Response "MN" [] ""+ timeout 5000000 (takeMVar reset) `shouldReturn` Just ()+ checkIdle c `shouldThrow` anyIOException++ describe "error messages" $ do+ let everyError detail =+ map OpError+ [ NoError, ErrKeyNotFound, ErrKeyExists, ErrValueTooLarge+ , ErrItemNotStored, ErrValueNonNumeric, ErrOutOfMemory, TextAuthFail ] +++ map ClientError+ [ NoServersReady, Timeout, InvalidAuthentication detail, KeyTooLong detail ] +++ map (ProtocolError . ($ detail))+ [ UnknownPkt, UnknownOp, UnknownStatus, BadLength, UnexpectedResponse+ , E.ServerError, E.BadCommand, UnexpectedEOF ]+ -- Bytes as a server may send them, sometimes far more than are quoted.+ detailGen = do+ len <- frequency [(3, choose (0, 20)), (1, choose (190, 3000))]+ vectorOf len (frequency [(4, choose (' ', '~')), (1, choose ('\0', '\255'))])+ displays message err = displayException (err :: MemcacheError) == message++ it "describes errors in plain words" $ do+ displayException (OpError ErrKeyNotFound)+ `shouldBe` "memcached operation failed: key not found"+ displayException (ClientError NoServersReady)+ `shouldBe` "memcache client error: no servers ready"+ displayException (ClientError (KeyTooLong "empty key"))+ `shouldBe` "memcache client error: invalid key or argument: empty key"+ displayException (ProtocolError (UnknownStatus "XX"))+ `shouldBe` "memcached protocol error: unknown response code: XX"++ it "describes an empty response line" $+ recvResponseFrom "\r\n" `shouldThrow` displays "memcached protocol error: empty response line"++ it "escapes control characters the server sent" $+ withClient [MR "CLIENT_ERROR \ESC]0;owned\a bad data chunk\r\n"] $ \client ->+ Client.get client "key" `shouldThrow` displays+ "memcached protocol error: command rejected by server: \\ESC]0;owned\\a bad data chunk"++ it "escapes bytes outside ASCII" $+ displayException (ProtocolError (E.ServerError "caf\233 \195\169"))+ `shouldBe` "memcached protocol error: server error: caf\\233 \\195\\169"++ it "quotes at most 200 characters of a detail" $+ displayException (ProtocolError (E.ServerError (replicate 300 'x')))+ `shouldBe` "memcached protocol error: server error: " <> replicate 200 'x' <> "..."++ prop "puts every error on one short line of printable ASCII" $+ forAll detailGen $ \detail -> conjoin+ [ counterexample message $+ -- The prefix, then at most 200 characters, each at most six once escaped.+ not (null message) && all (\c -> isAscii c && isPrint c) message+ && length message <= 1300+ | message <- map displayException (everyError detail) ]++ prop "gives each constructor its own message" $+ forAll detailGen $ \detail ->+ let messages = map displayException (everyError detail)+ in length (nub messages) === length messages++ it "keeps the derived show" $+ show (ProtocolError (UnknownStatus "XX"))+ `shouldBe` "ProtocolError (UnknownStatus {protocolError = \"XX\"})"++ describe "getMany" $ do+ it "matches replies by opaque and skips misses" $+ withServerAndClient [MR "VA 5 f1 c11 O0\r\nfirst\r\n", Noop, MR "MN\r\n"] $ \server client -> do+ Client.getMany client ["one", "two"] `shouldReturn` [("one", ("first", 1, 11))]+ mockRequests server `shouldReturn`+ ["version\r\n", "mg one v f c q O0\r\n", "mg two v f c q O1\r\n", "mn\r\n"]++ it "maps encoded keys back to the caller's key" $+ withClient [MR "VA 3 f0 c4 O0\r\nabc\r\n", MR "MN\r\n"] $ \client ->+ Client.getMany client ["a b"] `shouldReturn` [("a b", ("abc", 0, 4))]++ it "tells a plain key from another key's base64 form" $+ -- " " is sent as IA== with the b flag.+ withServerAndClient [MR "VA 1 f0 c1 O0\r\na\r\n", MR "VA 1 f0 c2 O1\r\nb\r\n", MR "MN\r\n"] $+ \server client -> do+ Client.getMany client ["IA==", " "] `shouldReturn` [("IA==", ("a", 0, 1)), (" ", ("b", 0, 2))]+ mockRequests server `shouldReturn`+ ["version\r\n", "mg IA== v f c q O0\r\n", "mg IA== b v f c q O1\r\n", "mn\r\n"]++ -- memcached echoes a key raw or as base64 according to how its item was+ -- stored, not how the request sent it: "a b" raw once an increment has+ -- created it, and "foo" as base64 once another client has stored it with+ -- the b flag.+ it "matches replies by opaque, not by the key they echo" $+ withClient [MR "VA 1 f0 c1 ka b O0\r\n5\r\n", MR "VA 1 f0 c2 kZm9v b O1\r\nz\r\n", MR "MN\r\n"] $ \client ->+ Client.getMany client ["a b", "foo"] `shouldReturn` [("a b", ("5", 0, 1)), ("foo", ("z", 0, 2))]++ -- Echoed raw, as it is once an increment has created it, the key's+ -- newline would end the reply line, so the request asks for no echo.+ it "gets a key containing a newline and a space" $+ withServerAndClient [MR "VA 1 f0 c1 O0\r\n5\r\n", MR "VA 4 f7 c2 O1\r\nxval\r\n", MR "MN\r\n"] $+ \server client -> do+ Client.getMany client ["x\ny z", "x"] `shouldReturn` [("x\ny z", ("5", 0, 1)), ("x", ("xval", 7, 2))]+ mockRequests server `shouldReturn`+ [ "version\r\n"+ , "mg " <> Base64.encode "x\ny z" <> " b v f c q O0\r\n", "mg x v f c q O1\r\n", "mn\r\n" ]++ it "returns a repeated key once per hit" $+ withClient [MR "VA 1 f0 c1 O0\r\na\r\n", Noop, MR "VA 1 f0 c1 O2\r\na\r\n", MR "MN\r\n"] $ \client ->+ Client.getMany client ["key", "other", "key"] `shouldReturn` [("key", ("a", 0, 1)), ("key", ("a", 0, 1))]++ -- A batch is written whole before its replies are read, so it must be+ -- small enough that the replies can't fill the socket buffers first,+ -- which would leave both ends blocked until the attempt runs out of+ -- time. A server's requests go in batches of at most 256, each ended by+ -- an mn, and the opaques count on from one batch to the next.+ it "closes a batch once it holds 256 requests" $+ withServerAndClient+ (replicate 256 Noop ++ [MR "MN\r\n"] ++ replicate 43 Noop ++ [MR "VA 1 f0 c1 O299\r\nv\r\n", MR "MN\r\n"]) $+ \server client -> do+ Client.getMany client batchKeys `shouldReturn` [("key299", ("v", 0, 1))]+ (version, requests) <- splitAt 1 <$> mockRequests server+ version `shouldBe` ["version\r\n"]+ batchEnds requests `shouldBe` [256, 301]++ it "reads a malformed CAS or flags as zero rather than wrapping them" $+ withClient [MR "VA 3 f2 c-1 O0\r\nfoo\r\n", MR "VA 3 f4294967298 c7 O1\r\nbar\r\n", MR "MN\r\n"] $ \client ->+ Client.getMany client ["one", "two"] `shouldReturn` [("one", ("foo", 2, 0)), ("two", ("bar", 0, 7))]++ it "rejects a reply with a malformed value length" $+ withServerAndClientOpts oneAttempt [MR "VA (3) f2 c7 O0\r\nfoo\r\n", MR "MN\r\n"] $ \_ client ->+ Client.getMany client ["one"] `shouldThrow` isBadLength++ -- memcached sends a bare EN in place of a hit's VA line when reading the+ -- value from extstore misses, and before 1.6.18 for any miss that is not+ -- quiet.+ it "skips a miss reply with no opaque" $+ withServerAndClientOpts oneAttempt+ [ MR "VA 1 f0 c1 O0\r\na\r\n", MR "EN\r\n", MR "VA 1 f0 c3 O2\r\nc\r\n", MR "MN\r\n"+ , MR "VA 3 f2 c7\r\nfoo\r\n" ] $ \server client -> do+ Client.getMany client ["one", "two", "three"]+ `shouldReturn` [("one", ("a", 0, 1)), ("three", ("c", 0, 3))]+ Client.get client "other" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ [ "version\r\n"+ , "mg one v f c q O0\r\n", "mg two v f c q O1\r\n", "mg three v f c q O2\r\n", "mn\r\n"+ , "mg other v f c\r\n" ]++ -- Each bare EN answers a request, so here the second is a reply too many.+ it "rejects bare EN replies without end, then uses a new connection" $+ withServerAndClientOpts oneAttempt+ [Repeat "EN\r\n", MR "VA 1 f0 c1 O0\r\ny\r\nMN\r\n"] $ \_ client -> do+ Client.getMany client ["one"] `shouldThrow` isUnexpectedResponse+ Client.getMany client ["one"] `shouldReturn` [("one", ("y", 0, 1))]++ it "rejects a bare EN after the reply to the last request" $+ withServerAndClientOpts oneAttempt [Noop, MR "VA 1 f0 c2 O1\r\nb\r\nEN\r\nMN\r\n"] $ \_ client ->+ Client.getMany client ["one", "two"] `shouldThrow` isUnexpectedResponse++ forM_+ [ ("no opaque", "VA 1 f0 c1\r\na\r\n")+ , ("an unknown opaque", "VA 1 f0 c1 O2\r\na\r\n")+ , ("an opaque out of order", "VA 1 f0 c2 O1\r\nb\r\nVA 1 f0 c1 O0\r\na\r\n")+ , ("a repeated opaque", "VA 1 f0 c1 O0\r\na\r\nVA 1 f0 c1 O0\r\na\r\n")+ , ("the opaque of the request a bare EN was counted for", "EN\r\nVA 1 f0 c1 O0\r\na\r\n")+ ] $ \(problem, replies) ->+ it ("rejects a reply with " <> problem) $+ withServerAndClientOpts oneAttempt [Noop, Noop, MR (replies <> "MN\r\n")] $ \_ client ->+ Client.getMany client ["one", "two"] `shouldThrow` isUnexpectedResponse++ it "rejects a reply with an unknown code and no opaque as an unknown status" $+ withServerAndClientOpts oneAttempt [Noop, MR "XX\r\nMN\r\n"] $ \_ client ->+ Client.getMany client ["one"] `shouldThrow` (== ProtocolError (UnknownStatus "XX"))++ it "closes the connection after a reply out of step, so the next request gets its own reply" $+ -- Kept, the connection would answer the get with the stale VA.+ withServerAndClientOpts oneAttempt+ [Noop, MR "VA 1 f0 c1 O7\r\nx\r\nVA 3 f9 c9\r\nold\r\nMN\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $+ \server client -> do+ Client.getMany client ["one"] `shouldThrow` isUnexpectedResponse+ Client.get client "two" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ ["version\r\n", "mg one v f c q O0\r\n", "mn\r\n", "version\r\n", "mg two v f c\r\n"]++ it "closes the connection after a reply with an unknown code, so the next request gets its own reply" $+ -- The client can't tell where such a reply ends. Here the reply's two+ -- bytes read as the batch's MN, so, kept, the connection would answer+ -- the get with the real MN.+ withServerAndClientOpts oneAttempt+ [MR "XX 2 O0\r\nMN\r\n", MR "MN\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ Client.getMany client ["one"] `shouldThrow` (== ProtocolError (UnknownStatus "XX"))+ Client.get client "two" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ ["version\r\n", "mg one v f c q O0\r\n", "mn\r\n", "version\r\n", "mg two v f c\r\n"]++ -- A SERVER_ERROR whose text names no status, as memcached's proxy sends+ -- when a backend fails, is a server failure in a batch too: the batch is+ -- sent again on a new connection, and the retry's success leaves the+ -- server alive. The mock answers the first attempt's mn, as the proxy+ -- does, and keeps the connection open, so only the SERVER_ERROR can make+ -- the client send the batch again.+ it "sends the batch again after a server error" $+ withServerAndClientOpts fastFailover+ [MR "SERVER_ERROR backend failure\r\n", MR "MN\r\n", Noop, MR "VA 1 c5 O0\r\nv\r\nMN\r\n"] $+ \server client -> do+ Client.getMany client ["key"] `shouldReturn` [("key", ("v", 0, 5))]+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg key v f c q O0\r\n", "mn\r\n"+ , "version\r\n", "mg key v f c q O0\r\n", "mn\r\n" ]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- The proxy goes on to answer the batch's other keys and its mn. Here+ -- only the first key's backend has failed, so the second key's hit+ -- follows the SERVER_ERROR. The whole batch must still be sent again,+ -- and the result is the retry's.+ it "sends the whole batch again after a server error for one of its keys" $+ withServerAndClientOpts fastFailover+ [ MR "SERVER_ERROR backend failure\r\n", Noop, MR "VA 1 c1 O1\r\nw\r\nMN\r\n"+ , Noop, Noop, MR "VA 1 c5 O0\r\nv\r\nVA 1 c1 O1\r\nw\r\nMN\r\n" ] $+ \server client -> do+ Client.getMany client ["a", "b"] `shouldReturn` [("a", ("v", 0, 5)), ("b", ("w", 0, 1))]+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg a v f c q O0\r\n", "mg b v f c q O1\r\n", "mn\r\n"+ , "version\r\n", "mg a v f c q O0\r\n", "mg b v f c q O1\r\n", "mn\r\n" ]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- As for a server failure, the batch is sent again on a new connection,+ -- and the failure counts towards marking the server dead.+ it "sends the batch again after a reply with an unknown code" $+ withServerAndClientOpts fastFailover+ [MR "XX O0\r\n", MR "MN\r\n", MR "VA 3 f2 c7 O0\r\nfoo\r\n", MR "MN\r\n"] $ \server client -> do+ Client.getMany client ["one"] `shouldReturn` [("one", ("foo", 2, 7))]+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg one v f c q O0\r\n", "mn\r\n"+ , "version\r\n", "mg one v f c q O0\r\n", "mn\r\n" ]++ -- The unknown reply answers the batch's mn, so the mock has recorded both+ -- requests of each attempt before the client sees it.+ it "marks the server dead once every attempt at a batch gets an unknown code" $+ withServerAndClientOpts fastFailover+ [Noop, MR "XX O0\r\nMN\r\n", Noop, MR "XX O0\r\nMN\r\n"] $ \server client -> do+ Client.getMany client ["one"] `shouldThrow` (== ProtocolError (UnknownStatus "XX"))+ Client.get client "two" `shouldThrow` (== ClientError NoServersReady)+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg one v f c q O0\r\n", "mn\r\n"+ , "version\r\n", "mg one v f c q O0\r\n", "mn\r\n" ]++ -- A SERVER_ERROR whose text names no status counts against the server+ -- in a batch too. Each answers the batch's mn, so the mock has recorded+ -- both requests of each attempt before the client sees it.+ it "marks the server dead once every attempt at a batch gets a server error" $+ withServerAndClientOpts fastFailover+ [Noop, MR "SERVER_ERROR backend failure\r\nMN\r\n", Noop, MR "SERVER_ERROR backend failure\r\nMN\r\n"] $+ \server client -> do+ Client.getMany client ["key"] `shouldThrow` (== ProtocolError (E.ServerError "backend failure"))+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg key v f c q O0\r\n", "mn\r\n"+ , "version\r\n", "mg key v f c q O0\r\n", "mn\r\n" ]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ describe "setMany" $ do+ it "stores each pair and returns its version" $+ withServerAndClient [MR "HD c1 O0\r\n", MR "HD c2 O1\r\n", MR "MN\r\n"] $ \server client -> do+ Client.setMany client [("one", "first", 1, 30), ("two", "second", 2, 0)]+ `shouldReturn` [("one", 1), ("two", 2)]+ mockRequests server `shouldReturn`+ [ "version\r\n"+ , "ms one 5 F1 T30 MS c O0\r\nfirst\r\n"+ , "ms two 6 F2 T0 MS c O1\r\nsecond\r\n"+ , "mn\r\n"+ ]++ it "maps encoded keys back to the caller's key" $+ withClient [MR "HD c4 O0\r\n", MR "MN\r\n"] $ \client ->+ Client.setMany client [("a b", "v", 0, 0)] `shouldReturn` [("a b", 4)]++ it "tells a plain key from another key's base64 form" $+ withServerAndClient [MR "HD c1 O0\r\n", MR "HD c2 O1\r\n", MR "MN\r\n"] $ \server client -> do+ Client.setMany client [("IA==", "a", 0, 0), (" ", "b", 0, 0)] `shouldReturn` [("IA==", 1), (" ", 2)]+ mockRequests server `shouldReturn`+ [ "version\r\n"+ , "ms IA== 1 F0 T0 MS c O0\r\na\r\n", "ms IA== 1 F0 T0 MS b c O1\r\nb\r\n", "mn\r\n" ]++ -- Error replies carry no opaque, so they must be read as errors before+ -- the reply is matched by its opaque.+ it "reports a server error in a batch as its status" $+ withServerAndClientOpts oneAttempt [MR "SERVER_ERROR out of memory storing object\r\n"] $ \_ client ->+ Client.setMany client [("key", "v", 0, 0)] `shouldThrow` (== OpError ErrOutOfMemory)++ it "stores a repeated key once per occurrence, in order" $+ withServerAndClient [MR "HD c1 O0\r\n", MR "HD c2 O1\r\n", MR "MN\r\n"] $ \server client -> do+ Client.setMany client [("key", "old", 0, 0), ("key", "new", 0, 0)]+ `shouldReturn` [("key", 1), ("key", 2)]+ mockRequests server `shouldReturn`+ [ "version\r\n"+ , "ms key 3 F0 T0 MS c O0\r\nold\r\n", "ms key 3 F0 T0 MS c O1\r\nnew\r\n", "mn\r\n" ]++ it "groups results by server, keeping each server's keys in order" $+ withTwoServerClient+ [MR "HD c1 O0\r\n", MR "HD c2 O2\r\n", MR "MN\r\n"]+ [MR "HD c3 O1\r\n", MR "MN\r\n"] $ \a b client -> do+ Client.setMany client [("a1", "x", 0, 0), ("b1", "y", 0, 0), ("a2", "z", 0, 0)]+ `shouldReturn` [("a1", 1), ("a2", 2), ("b1", 3)]+ mockRequests a `shouldReturn`+ [ "version\r\n"+ , "ms a1 1 F0 T0 MS c O0\r\nx\r\n", "ms a2 1 F0 T0 MS c O2\r\nz\r\n", "mn\r\n" ]+ mockRequests b `shouldReturn` ["version\r\n", "ms b1 1 F0 T0 MS c O1\r\ny\r\n", "mn\r\n"]++ it "checks every key before sending anything" $+ withServerAndClient [MR "HD c1 O0\r\n", MR "MN\r\n"] $ \server client -> do+ Client.setMany client [("key", "v", 0, 0), (C.replicate 251 'a', "v", 0, 0)]+ `shouldThrow` isKeyTooLong+ Client.setMany client [("key", "v", 0, 0), ("", "v", 0, 0)] `shouldThrow` isKeyTooLong+ mockRequests server `shouldReturn` []++ it "sends nothing for an empty list" $+ withServerAndClient [] $ \server client -> do+ Client.setMany client [] `shouldReturn` []+ mockRequests server `shouldReturn` []++ it "closes a batch once its requests reach 1 MiB" $ do+ let value n = C.replicate n 'x'+ withServerAndClient+ [ MR "HD c1 O0\r\n", MR "HD c2 O1\r\n", MR "MN\r\n"+ , MR "HD c3 O2\r\n", MR "MN\r\n", MR "HD c4 O3\r\n", MR "MN\r\n" ] $ \server client -> do+ -- one and two fit in 1 MiB; three is over 1 MiB, so it goes alone,+ -- and four would take its batch past 1 MiB.+ Client.setMany client+ [ ("one", value 400000, 0, 0), ("two", value 400000, 0, 0)+ , ("three", value 1100000, 0, 0), ("four", "v", 0, 0) ]+ `shouldReturn` [("one", 1), ("two", 2), ("three", 3), ("four", 4)]+ (version, requests) <- splitAt 1 <$> mockRequests server+ version `shouldBe` ["version\r\n"]+ map (== "mn\r\n") requests `shouldBe` [False, False, True, False, True, False, True]++ -- As for getMany, a batch holds at most 256 requests, and the opaques+ -- count on from one batch to the next.+ it "closes a batch once it holds 256 requests" $ do+ let stored i = MR ("HD c" <> asDecimal (i + 1) <> " O" <> asDecimal i <> "\r\n")+ withServerAndClient+ (map stored [0 .. 255 :: Int] ++ [MR "MN\r\n"] ++ map stored [256 .. 299 :: Int] ++ [MR "MN\r\n"]) $+ \server client -> do+ Client.setMany client [(key, "v", 0, 0) | key <- batchKeys] `shouldReturn` zip batchKeys [1 ..]+ (version, requests) <- splitAt 1 <$> mockRequests server+ version `shouldBe` ["version\r\n"]+ batchEnds requests `shouldBe` [256, 301]++ it "throws a reply other than stored, after reading the whole batch" $+ withMockServer+ (authExchange ++ [supported, MR "HD c1 O0\r\n", MR "NS O1\r\n", MR "MN\r\n", MR "VA 1 f0 c1\r\nv\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ Client.setMany client [("one", "v", 0, 0), ("two", "v", 0, 0)]+ `shouldThrow` (== OpError ErrItemNotStored)+ -- The same connection serves the next request; a new one would+ -- authenticate and ask for the version again.+ Client.get client "one" `shouldReturn` Just ("v", 0, 1)++ it "returns the versions from a retried batch" $+ withServerAndClientOpts fastFailover+ [MR "HD c1 O0\r\n", CloseConnection, MR "HD c3 O0\r\n", MR "HD c4 O1\r\n", MR "MN\r\n"] $ \_ client ->+ Client.setMany client [("one", "v", 0, 0), ("two", "v", 0, 0)]+ `shouldReturn` [("one", 3), ("two", 4)]++ it "reads a malformed CAS as zero rather than wrapping it" $+ withClient [MR "HD c-1 O0\r\n", MR "HD c18446744073709551621 O1\r\n", MR "MN\r\n"] $ \client ->+ Client.setMany client [("one", "v", 0, 0), ("two", "v", 0, 0)] `shouldReturn` [("one", 0), ("two", 0)]++ describe "deleteMany" $ do+ it "deletes present, missing and repeated keys alike" $+ withServerAndClient [MR "HD O0\r\n", MR "NF O1\r\n", MR "NF O2\r\n", MR "MN\r\n"] $ \server client -> do+ Client.deleteMany client ["one", "two", "one"] `shouldReturn` ()+ mockRequests server `shouldReturn`+ ["version\r\n", "md one O0\r\n", "md two O1\r\n", "md one O2\r\n", "mn\r\n"]++ it "deletes keys on every server" $+ withTwoServerClient [MR "HD O0\r\n", MR "MN\r\n"] [MR "NF O1\r\n", MR "MN\r\n"] $ \a b client -> do+ Client.deleteMany client ["a1", "b1"] `shouldReturn` ()+ mockRequests a `shouldReturn` ["version\r\n", "md a1 O0\r\n", "mn\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "md b1 O1\r\n", "mn\r\n"]++ -- As for getMany, a batch holds at most 256 requests.+ it "closes a batch once it holds 256 requests" $ do+ let deleted i = MR ("HD O" <> asDecimal i <> "\r\n")+ withServerAndClient+ (map deleted [0 .. 255 :: Int] ++ [MR "MN\r\n"] ++ map deleted [256 .. 299 :: Int] ++ [MR "MN\r\n"]) $+ \server client -> do+ Client.deleteMany client batchKeys `shouldReturn` ()+ (version, requests) <- splitAt 1 <$> mockRequests server+ version `shouldBe` ["version\r\n"]+ batchEnds requests `shouldBe` [256, 301]++ it "checks every key before sending anything" $+ withServerAndClient [MR "HD O0\r\n", MR "MN\r\n"] $ \server client -> do+ Client.deleteMany client ["key", C.replicate 251 'a'] `shouldThrow` isKeyTooLong+ Client.deleteMany client ["key", ""] `shouldThrow` isKeyTooLong+ mockRequests server `shouldReturn` []++ it "sends nothing for an empty list" $+ withServerAndClient [] $ \server client -> do+ Client.deleteMany client [] `shouldReturn` ()+ mockRequests server `shouldReturn` []++ it "throws a reply other than deleted or not found" $+ withClient [MR "EX O0\r\n", MR "MN\r\n"] $ \client ->+ Client.deleteMany client ["key"] `shouldThrow` (== OpError ErrKeyExists)++ it "succeeds when a retried batch finds keys already deleted" $+ withServerAndClientOpts fastFailover+ [MR "HD O0\r\n", CloseConnection, MR "NF O0\r\n", MR "HD O1\r\n", MR "MN\r\n"] $ \server client -> do+ Client.deleteMany client ["one", "two"] `shouldReturn` ()+ mockRequests server `shouldReturn`+ [ "version\r\n", "md one O0\r\n", "md two O1\r\n"+ , "version\r\n", "md one O0\r\n", "md two O1\r\n", "mn\r\n" ]++ describe "client operations" $ do+ it "gets a hit" $+ withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n"]++ it "returns Nothing for a miss" $+ withClient [MR "EN\r\n"] $ \client ->+ Client.get client "key" `shouldReturn` Nothing++ it "rejects a reply meant for a different command" $+ withClient [MR "HD\r\n"] $ \client ->+ Client.get client "key" `shouldThrow` isUnexpectedResponse++ it "rejects an unknown reply" $+ withClient [MR "XX\r\n"] $ \client ->+ Client.get client "key" `shouldThrow` isUnknownStatus++ it "reads a malformed CAS or flags as zero rather than wrapping them" $+ withClient [MR "VA 3 f2 c-1\r\nfoo\r\n", MR "VA 3 f4294967298 c7\r\nfoo\r\n"] $ \client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 0)+ Client.get client "key" `shouldReturn` Just ("foo", 0, 7)++ it "rejects a reply with a malformed value length" $+ -- Sent twice, as a protocol error is retried once on a new connection.+ withClient [MR "VA (3) f2 c7\r\nfoo\r\n", MR "VA (3) f2 c7\r\nfoo\r\n"] $ \client ->+ Client.get client "key" `shouldThrow` isBadLength++ it "gets and touches a value" $+ withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n", MR "EN\r\n"] $ \server client -> do+ Client.gat client "key" 60 `shouldReturn` Just ("foo", 2, 7)+ Client.gat client "key" 60 `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c T60\r\n", "mg key v f c T60\r\n"]++ it "touches a value" $+ withClient [MR "HD c3\r\n", MR "EN\r\n"] $ \client -> do+ Client.touch client "key" 60 `shouldReturn` Just 3+ Client.touch client "key" 60 `shouldReturn` Nothing++ it "stores and returns the server CAS" $+ withServerAndClient [MR "HD c9\r\n"] $ \server client -> do+ Client.set client "key" "foo" 2 30 `shouldReturn` 9+ mockRequests server `shouldReturn` ["version\r\n", "ms key 3 F2 T30 MS c\r\nfoo\r\n"]++ it "reports a set the server did not store" $+ withClient [MR "NS\r\n"] $ \client ->+ Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrItemNotStored)++ it "adds only when the key is new" $+ withClient [MR "HD c5\r\n", MR "NS\r\n"] $ \client -> do+ Client.add client "key" "v" 0 0 `shouldReturn` Just 5+ Client.add client "key" "v" 0 0 `shouldReturn` Nothing++ it "replaces only when the key exists and the CAS matches" $+ withClient [MR "HD c8\r\n", MR "NS\r\n", MR "NF\r\n", MR "EX\r\n"] $ \client -> do+ Client.replace client "key" "v" 0 0 0 `shouldReturn` Just 8+ Client.replace client "key" "v" 0 0 0 `shouldReturn` Nothing+ Client.replace client "key" "v" 0 0 4 `shouldReturn` Nothing+ Client.replace client "key" "v" 0 0 4 `shouldReturn` Nothing++ it "compares and swaps" $+ withServerAndClient [MR "HD c3\r\n", MR "EX\r\n", MR "NF\r\n"] $ \server client -> do+ Client.cas client "key" "v" 0 0 2 `shouldReturn` Just 3+ Client.cas client "key" "v" 0 0 2 `shouldReturn` Nothing+ Client.cas client "key" "v" 0 0 2 `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ "version\r\n" : replicate 3 "ms key 1 F0 T0 C2 MS c\r\nv\r\n"++ it "deletes a present key" $+ withClient [MR "HD\r\n"] $ \client ->+ Client.delete client "key" 0 `shouldReturn` True++ it "reports a delete of a missing key or with a stale CAS" $+ withClient [MR "NF\r\n", MR "EX\r\n"] $ \client -> do+ Client.delete client "key" 0 `shouldReturn` False+ Client.delete client "key" 5 `shouldReturn` False++ it "does not retry a command the server rejected" $+ withClient [MR "CLIENT_ERROR bad\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \client ->+ Client.get client "key" `shouldThrow` isBadCommand++ it "reads version" $+ withClient [supported] $ \client ->+ Client.version client `shouldReturn` supportedVersion++ it "collects stats until END, keeping values that contain spaces" $+ withClient [MR "STAT curr_items 2\r\nSTAT rusage_user 0.1 0.2\r\nEND\r\n"] $ \client -> do+ result <- Client.stats client Nothing+ result `shouldSatisfy` allStats++ it "allows stats arguments containing spaces" $+ withServerAndClient [MR "END\r\n"] $ \server client -> do+ result <- Client.stats client (Just "cachedump 1 100")+ map snd result `shouldBe` [[]]+ mockRequests server `shouldReturn` ["version\r\n", "stats cachedump 1 100\r\n"]++ it "keeps byte 0xa0 in stats lines" $+ withClient [MR "ITEM voil\xc3\xa0 [2 b; 0 s]\r\nEND\r\n"] $ \client -> do+ result <- Client.stats client (Just "cachedump 1 1")+ map snd result `shouldBe` [[("ITEM", "voil\xc3\xa0 [2 b; 0 s]")]]++ it "keeps non-STAT stats lines" $+ withClient [MR "ITEM key [3 b; 0 s]\r\nEND\r\n"] $ \client -> do+ result <- Client.stats client (Just "cachedump 1 1")+ map snd result `shouldBe` [[("ITEM", "key [3 b; 0 s]")]]++ -- memcached lists some keys in cachedump as they are stored, and a key+ -- can hold a line break, here followed by END and a forged reply. The+ -- reply as stats reads it ends at that END, so stats must close the+ -- connection rather than leave the rest to be read as the next reply.+ it "closes its connection, so a dump cut short by a key can't answer a later request" $+ withServerAndClient+ [MR "ITEM k\nEND\r\nVA 4 f0 c0\r\nevil\r\n [1 b; 0 s]\r\nEND\r\n", MR "EN\r\n"] $ \server client -> do+ map snd <$> Client.stats client (Just "cachedump 1 0") `shouldReturn` [[("ITEM", "k")]]+ Client.get client "victim" `shouldReturn` Nothing+ -- The get went over a new connection, which checked the version.+ mockRequests server `shouldReturn`+ ["version\r\n", "stats cachedump 1 0\r\n", "version\r\n", "mg victim v f c\r\n"]+ mockClosed server `shouldReturn` 1++ -- A cachedump key's bytes read as a line of their own after the key's line+ -- break, so a key holding "k\nITEM forged [9 b; 0 s]" splits into two+ -- entries, the second one the server never sent as an entry.+ it "reads the bytes after a key's line break as further entries" $+ withServerAndClient [MR "ITEM k\nITEM forged [9 b; 0 s] [1 b; 0 s]\r\nEND\r\n"] $ \_ client ->+ map snd <$> Client.stats client (Just "cachedump 1 0")+ `shouldReturn` [[("ITEM", "k"), ("ITEM", "forged [9 b; 0 s] [1 b; 0 s]")]]++ -- A cachedump key whose bytes hold an empty line (here "k\n\ny") makes+ -- stats read an empty line after the first entry. That can only be a key's+ -- bytes, not a reply, so stats throws its error without sending the+ -- request again or marking the server dead, and a later request still+ -- reaches the server. The connection is closed all the same.+ it "throws a cachedump key's empty line without a retry or a dead mark" $+ withServerAndClient [MR "ITEM k\n\ny [1 b; 0 s]\r\nEND\r\n", MR "EN\r\n"] $ \server client -> do+ Client.stats client (Just "cachedump 1 0") `shouldThrow` isUnknownPkt+ Client.get client "victim" `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ ["version\r\n", "stats cachedump 1 0\r\n", "version\r\n", "mg victim v f c\r\n"]+ mockClosed server `shouldReturn` 1++ -- The same for a key whose bytes hold a SERVER_ERROR line: memcached never+ -- sends one in the middle of a stats reply, so after the first entry it+ -- can only be a key's bytes, thrown without failing the server over.+ it "throws a cachedump key's SERVER_ERROR line without a retry or a dead mark" $+ withServerAndClient [MR "ITEM k\nSERVER_ERROR x [1 b; 0 s]\r\nEND\r\n", MR "EN\r\n"] $ \server client -> do+ Client.stats client (Just "cachedump 1 0") `shouldThrow` isServerError+ Client.get client "victim" `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ ["version\r\n", "stats cachedump 1 0\r\n", "version\r\n", "mg victim v f c\r\n"]+ mockClosed server `shouldReturn` 1++ -- stats throws a key's error only once every server has answered. Here+ -- both servers list a key holding an empty line, and would list it again+ -- for a resend, so whichever server stats asks first, the other must+ -- still be asked. Neither is sent the request twice, and neither is+ -- marked dead.+ it "asks every server before throwing a cachedump key's error" $ do+ let dump = MR "ITEM k\n\ny [1 b; 0 s]\r\nEND\r\n"+ withTwoServerClient [dump, dump] [dump, dump] $ \a b client -> do+ Client.stats client (Just "cachedump 1 0") `shouldThrow` isUnknownPkt+ mockRequests a `shouldReturn` ["version\r\n", "stats cachedump 1 0\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "stats cachedump 1 0\r\n"]+ Cluster.getServers client >>= mapM_ (\s -> readIORef (Server.failed s) `shouldReturn` 0)++ -- A SERVER_ERROR on the reply's first line is the server's own reply, and+ -- one that names no status, as here, is a server failure: out of retries+ -- it marks the server dead, and a later request to it finds no server+ -- ready.+ it "fails the server over a SERVER_ERROR on the stats reply's first line" $+ withServerAndClientOpts+ Client.def { Client.optsServerRetries = 0, Client.optsDeadRetryDelay = 60000 }+ [MR "SERVER_ERROR x\r\n"] $ \_ client -> do+ Client.stats client Nothing `shouldThrow` isServerError+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)++ -- The entries that keys' line breaks add count towards the limit of+ -- 262,144 entries like any others, so a dump they take over it fails the+ -- server over, as any reply that is too long does. Here 2,900 keys of 181+ -- bytes each hold 90 line breaks, so 2,900 items give 263,900 entries.+ it "fails the server over a dump that keys' line breaks take over the entry limit" $ do+ let item = "ITEM k" <> B.concat (replicate 90 "\nx") <> " [1 b; 0 s]\r\n"+ withServerAndClientOpts+ (slowStats Client.def { Client.optsServerRetries = 0, Client.optsDeadRetryDelay = 60000 })+ [MR (B.concat (replicate 2900 item) <> "END\r\n")] $ \_ client -> do+ Client.stats client (Just "cachedump 1 0") `shouldThrow` isBadLength+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)++ -- stats reads its reply a line at a time: a line that starts with VA is+ -- an entry like any other, and each line is split as other replies are.+ it "splits each stats line as other replies are, even one that starts with VA" $+ withServerAndClientOpts oneAttempt+ [MR "STAT a\tb c\v \r\nVA 1\r\nz\r\nEND\r\n", MR "STAT a 1\r\n\r\nEND\r\n"] $ \_ client -> do+ map snd <$> Client.stats client Nothing `shouldReturn` [[("a\tb", "c\v"), ("VA", "1"), ("z", "")]]+ Client.stats client Nothing `shouldThrow` isUnknownPkt++ -- At 15 MiB, the first reply also checks that the byte limit leaves room+ -- for the longest lines stats conns sends.+ it "accepts a stats conns reply of 262,144 entries, but no entry more" $+ withServerAndClientOpts (slowStats oneAttempt)+ [ MR (B.concat (replicate 65536 connStats) <> "END\r\n")+ , MR (B.concat (replicate 65536 connStats) <> "STAT pid 1\r\nEND\r\n") ] $ \_ client -> do+ result <- Client.stats client (Just "conns")+ map (length . snd) result `shouldBe` [262144]+ Client.stats client (Just "conns") `shouldThrow` isBadLength++ it "rejects an endless stats reply, then uses a new connection" $+ withServerAndClientOpts (slowStats oneAttempt)+ [Repeat (B.concat (replicate 100 "STAT pid 1\r\n")), MR "STAT pid 2\r\nEND\r\n"] $ \_ client -> do+ Client.stats client Nothing `shouldThrow` isBadLength+ map snd <$> Client.stats client Nothing `shouldReturn` [[("pid", "2")]]++ -- Each line counts as sent, wherever its bytes are. 2,100 of these lines+ -- come to just over 16 MiB.+ forM_ [ ("long names", "STAT " <> C.replicate 8000 'x' <> " 1\r\n")+ , ("long values", "STAT big " <> C.replicate 8000 'x' <> "\r\n")+ , ("padding between tokens", "STAT a" <> C.replicate 8100 ' ' <> "b\r\n")+ , ("long keys in ITEM lines", "ITEM " <> C.replicate 8000 'x' <> " [1 b; 0 s]\r\n")+ ] $ \(shape, line) ->+ it ("rejects a stats reply of more than 16 MiB of lines with " <> shape) $+ withServerAndClientOpts (slowStats oneAttempt)+ [MR (B.concat (replicate 2100 line) <> "END\r\n")] $ \_ client ->+ Client.stats client Nothing `shouldThrow` isBadLength++ it "returns Nothing when arithmetic cannot create the item" $+ withClient [MR "NS\r\n"] $ \client ->+ Client.increment client "counter" 0 1 30 0 `shouldReturn` Nothing++ it "returns Nothing for arithmetic on a missing key it may not create, or with a stale CAS" $+ withClient [MR "NF\r\n", MR "EX\r\n"] $ \client -> do+ Client.increment client "counter" 0 1 maxBound 0 `shouldReturn` Nothing+ Client.decrement client "counter" 0 1 0 5 `shouldReturn` Nothing++ it "returns the new counter value and CAS" $+ withServerAndClient [MR "VA 1 c2\r\n0\r\n"] $ \server client -> do+ Client.decrement client "counter" 10 1 60 0 `shouldReturn` Just (0, 2)+ mockRequests server `shouldReturn` ["version\r\n", "ma counter D1 MD v c N60 J10\r\n"]++ it "reads a malformed counter value as zero" $+ withClient [MR "VA 4 c3\r\n(42)\r\n"] $ \client ->+ Client.increment client "counter" 0 1 0 0 `shouldReturn` Just (0, 3)++ it "reports arithmetic on a non-numeric value" $+ withClient [MR "CLIENT_ERROR cannot increment or decrement non-numeric value\r\n"] $ \client ->+ Client.increment client "key" 0 1 0 0 `shouldThrow` (== OpError ErrValueNonNumeric)++ it "returns Nothing appending to a missing key" $+ withClient [MR "NS\r\n"] $ \client ->+ Client.append client "key" "more" 0 `shouldReturn` Nothing++ it "prepends to an existing key" $+ withServerAndClient [MR "HD c4\r\n"] $ \server client -> do+ Client.prepend client "key" "pre" 0 `shouldReturn` Just 4+ mockRequests server `shouldReturn` ["version\r\n", "ms key 3 MP c\r\npre\r\n"]++ it "appends only when the key exists and the CAS matches" $+ withServerAndClient [MR "HD c6\r\n", MR "EX c0\r\n", MR "NS c0\r\n"] $ \server client -> do+ Client.append client "key" "more" 5 `shouldReturn` Just 6+ Client.append client "key" "more" 5 `shouldReturn` Nothing+ Client.append client "key" "more" 5 `shouldReturn` Nothing+ mockRequests server `shouldReturn` "version\r\n" : replicate 3 "ms key 4 MA C5 c\r\nmore\r\n"++ it "prepends only when the key exists and the CAS matches" $+ withServerAndClient [MR "HD c6\r\n", MR "EX c0\r\n", MR "NS c0\r\n"] $ \server client -> do+ Client.prepend client "key" "pre" 5 `shouldReturn` Just 6+ Client.prepend client "key" "pre" 5 `shouldReturn` Nothing+ Client.prepend client "key" "pre" 5 `shouldReturn` Nothing+ mockRequests server `shouldReturn` "version\r\n" : replicate 3 "ms key 3 MP C5 c\r\npre\r\n"++ it "flushes" $+ withServerAndClient [MR "OK\r\n"] $ \server client -> do+ Client.flush client (Just 10) `shouldReturn` ()+ mockRequests server `shouldReturn` ["version\r\n", "flush_all 10\r\n"]++ it "reports an oversized value as a status, not a protocol failure" $+ withClient [MR "SERVER_ERROR object too large for cache\r\n"] $ \client ->+ Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrValueTooLarge)++ it "reports a server out of memory as a status" $+ withClient [MR "SERVER_ERROR out of memory storing object\r\n"] $ \client ->+ Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrOutOfMemory)++ -- memcached capitalises its out-of-memory errors for a large value it+ -- can't get memory for while reading it, and for a counter it can't+ -- create. They are statuses like the lowercase ones. Retried, a set+ -- would upload its value a second time, and the failures would mark a+ -- full but healthy server dead.+ it "reports a large value the server ran out of memory reading as a status, sending it once" $+ withServerAndClient [MR "SERVER_ERROR Out of memory during read\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $+ \server client -> do+ Client.set client "key" "foo" 0 0 `shouldThrow` (== OpError ErrOutOfMemory)+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ -- The error closed the connection, so the get opened a new one.+ mockRequests server `shouldReturn`+ ["version\r\n", "ms key 3 F0 T0 MS c\r\nfoo\r\n", "version\r\n", "mg key v f c\r\n"]++ it "reports a counter the server ran out of memory creating as a status, sending it once" $+ withServerAndClient [MR "SERVER_ERROR Out of memory allocating new item\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $+ \server client -> do+ Client.increment client "counter" 0 1 0 0 `shouldThrow` (== OpError ErrOutOfMemory)+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ ["version\r\n", "ma counter D1 MI v c N0 J0\r\n", "version\r\n", "mg key v f c\r\n"]++ it "returns no stats on a reset reply" $+ withClient [MR "RESET\r\n"] $ \client -> do+ result <- Client.stats client Nothing+ map snd result `shouldBe` [[]]++ -- Each test gets the value of 1 GiB + 1 byte from overLimitValue. hspec's+ -- beforeAll allocates it when the first of these tests runs, so a run that+ -- leaves them all out never allocates it. Every connection attempt to the+ -- client's server is refused, so an operation that tried to send the value+ -- would throw "Connection refused" instead.+ describe "oversized values" $ beforeAll overLimitValue $ do+ forM_+ [ ("set", \c v -> void (Client.set c "key" v 0 0))+ , ("add", \c v -> void (Client.add c "key" v 0 0))+ , ("replace", \c v -> void (Client.replace c "key" v 0 0 0))+ , ("cas", \c v -> void (Client.cas c "key" v 0 0 1))+ , ("append", \c v -> void (Client.append c "key" v 0))+ , ("prepend", \c v -> void (Client.prepend c "key" v 0)) ] $ \(operation, run) ->+ it ("rejects an over-limit value from " <> operation <> " without sending anything") $ \over ->+ withRefusingClient Client.def $ \client ->+ run client over `shouldThrow` (== OpError ErrValueTooLarge)++ it "rejects an over-limit value in setMany without sending anything" $ \over ->+ withRefusingClient Client.def $ \client ->+ Client.setMany client [("key", over, 0, 0)] `shouldThrow` (== OpError ErrValueTooLarge)++ it "sends nothing when only one of setMany's values is over the limit" $ \over ->+ withRefusingClient Client.def $ \client ->+ Client.setMany client [("one", "v", 0, 0), ("two", over, 0, 0), ("three", "v", 0, 0)]+ `shouldThrow` (== OpError ErrValueTooLarge)++ -- No key has a server, so a value the client lets through fails with+ -- NoServersReady, before anything is sent.+ it "does not reject a value of exactly the limit" $ \over -> do+ let noServer = Client.def { Client.optsGetServerForKey = \_ _ -> return Nothing }+ withRefusingClient noServer $ \client -> do+ Client.set client "key" over 0 0 `shouldThrow` (== OpError ErrValueTooLarge)+ Client.set client "key" (B.take valueLimit over) 0 0+ `shouldThrow` (== ClientError NoServersReady)++ -- A reply the request cannot get is likely an earlier request's, so the+ -- connection must be closed. Kept, it would answer the next request with+ -- the stale reply scripted after it, such as "VA 3 f9 c9\r\nold\r\n".+ describe "replies out of step" $ do+ it "closes the connection after an unknown reply, so the next request gets its own reply" $+ withServerAndClient [MR "XX\r\nVA 3 f9 c9\r\nold\r\n", MR "EN\r\n"] $ \server client -> do+ Client.get client "a" `shouldThrow` (== ProtocolError (UnknownStatus "XX"))+ Client.get client "b" `shouldReturn` Nothing+ mockRequests server `shouldReturn` ["version\r\n", "mg a v f c\r\n", "version\r\n", "mg b v f c\r\n"]++ forM_+ [ ("a get", "GET", "HD c1\r\n", \c -> void (Client.get c "key"))+ , ("a get", "GET", "NF\r\n", \c -> void (Client.get c "key"))+ , ("a get-and-touch", "GAT", "HD c1\r\n", \c -> void (Client.gat c "key" 0))+ , ("a touch", "TOUCH", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.touch c "key" 0))+ , ("a touch", "TOUCH", "OK\r\n", \c -> void (Client.touch c "key" 0))+ , ("a set", "SET", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.set c "key" "v" 0 0))+ , ("an add", "ADD", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.add c "key" "v" 0 0))+ , ("a replace", "REPLACE", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.replace c "key" "v" 0 0 0))+ , ("a cas", "SET", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.cas c "key" "v" 0 0 1))+ , ("an append", "APPEND", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.append c "key" "v" 0))+ , ("a prepend", "PREPEND", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.prepend c "key" "v" 0))+ , ("an increment", "INCREMENT", "HD c1\r\n", \c -> void (Client.increment c "key" 0 1 0 0))+ , ("a decrement", "DECREMENT", "HD c1\r\n", \c -> void (Client.decrement c "key" 0 1 0 0))+ , ("a delete", "DELETE", "VA 1 f0 c1\r\nx\r\n", \c -> void (Client.delete c "key" 0))+ , ("a delete", "DELETE", "EN\r\n", \c -> void (Client.delete c "key" 0))+ , ("a flush", "FLUSH", "HD\r\n", (`Client.flush` Nothing))+ , ("a version", "VERSION", "OK\r\n", void . Client.version) ] $+ \(request, operation, reply, run) -> do+ let code = C.takeWhile (\b -> b /= ' ' && b /= '\r') reply+ it ("closes the connection when " <> request <> " is answered with " <> C.unpack code) $+ withServerAndClient [MR (reply <> "VA 3 f9 c9\r\nold\r\n"), MR "VA 3 f2 c7\r\nfoo\r\n"] $+ \_ client -> do+ run client `shouldThrow` (== ProtocolError (UnexpectedResponse+ ("Expected " <> operation <> ", got: " <> show code)))+ Client.get client "next" `shouldReturn` Just ("foo", 2, 7)++ -- The reproduction from the review: the reply to a get from the classic+ -- text protocol, which leaves "foo" and "END" unread.+ it "answers each later request with its own reply" $+ withServerAndClient+ [MR "VALUE key 0 3\r\nfoo\r\nEND\r\n", MR "HD c9\r\n", MR "NF\r\n", MR "EN\r\n"] $ \server client -> do+ Client.get client "a" `shouldThrow` (== ProtocolError (UnknownStatus "VALUE"))+ Client.set client "b" "v" 0 0 `shouldReturn` 9+ Client.delete client "c" 0 `shouldReturn` False+ Client.get client "d" `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ [ "version\r\n", "mg a v f c\r\n"+ , "version\r\n", "ms b 1 F0 T0 MS c\r\nv\r\n", "md c\r\n", "mg d v f c\r\n" ]++ -- memcached 1.6.13 to 1.6.17, run with -o meta_response_old, sends OK+ -- in place of HD.+ it "takes OK as success for stores and deletes" $+ withClient [MR "OK c5\r\n", MR "OK\r\n"] $ \client -> do+ Client.set client "key" "v" 0 0 `shouldReturn` 5+ Client.delete client "key" 0 `shouldReturn` True++ -- These take any command, such as the classic delete, which gets DELETED.+ it "returns any reply but an error from keyedOp, anyOp and allOp" $+ withServerAndClient (replicate 3 (MR "DELETED\r\n")) $ \server client -> do+ keyed <- Cluster.keyedOp client "key" (const "delete key\r\n")+ anyReply <- Cluster.anyOp client "delete key\r\n"+ allReplies <- Cluster.allOp client "delete key\r\n"+ map responseCode (keyed : anyReply : map snd allReplies) `shouldBe` replicate 3 "DELETED"+ mockRequests server `shouldReturn` "version\r\n" : replicate 3 "delete key\r\n"++ describe "retries and failover" $ do+ let fast = Client.def+ { Client.optsServerTimeout = 300+ , Client.optsFailRetryDelay = 10+ , Client.optsDeadRetryDelay = 300+ }+ -- A server marked dead stays dead for the rest of the test.+ fastStaysDead = fast { Client.optsDeadRetryDelay = 60000 }+ -- Options for the attempts helper below. Each attempt fails as soon+ -- as the mock drops its connection; its 5 s are only so that a slow+ -- machine can't turn the drop into a timeout.+ dropping = fastStaysDead { Client.optsServerTimeout = 5000 }+ withRetries n = dropping { Client.optsServerRetries = n }+ -- Up to three connections to the server, rather than one.+ threeConnections opts = opts+ { Client.optsServerOptions = Client.defaultServerOptions { Client.soNumResources = 3 } }+ -- Leave three connections idle in the pool, each closed by the mock,+ -- as a restart of the server or memcached's idle_timeout leaves them.+ -- The mock reads a request on each, then closes it without replying.+ -- If a second request was sent, the mock closes the connection with+ -- it unread, so Linux resets the connection, as a firewall may.+ -- Reading each connection must then fail as the given selector+ -- expects. The mock answers each connection's version check at once,+ -- but serves its scripted responses one connection at a time, so+ -- every request is sent before any connection is read. The server from getServers+ -- has no time limit, so this gives up after 5 s, which fails the test+ -- rather than hanging it.+ leaveThreeClosed s requests failure = do+ finished <- timeout 5000000 $+ Server.withSocket s $ \a -> Server.withSocket s $ \b -> Server.withSocket s $ \c -> do+ mapM_ (`sendRaw` requests) [a, b, c]+ mapM_ ((`shouldThrow` failure) . recvResponse) [a, b, c]+ finished `shouldBe` Just ()+ -- An operation for allOp' that sends mn, reads the reply, then runs+ -- the given action. Against a mock that drops the first connection+ -- and answers the second with MN, its first attempt fails as a+ -- server failure, and only the retry runs the action.+ onRetry action s = Server.withSocket s $ \c -> sendRaw c "mn\r\n" >> recvResponse c >> action+ -- Work that never blocks, as reading a reply doesn't while a server+ -- sends it faster than it is read. While asynchronous exceptions are+ -- masked, nothing can interrupt it: yield lets other threads run,+ -- but isn't interruptible. It ends by itself after 5 s, so work that+ -- nothing interrupts fails the test rather than hanging it.+ neverBlocks = do+ start <- getMonotonicTime+ let loop = do+ yield+ now <- getMonotonicTime+ when (now - start < 5) loop+ loop++ it "retries after the server drops the connection" $+ withServerAndClientOpts fast [CloseConnection, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ -- The retry's new connection checks the version again.+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]++ it "retries after a server error" $+ withServerAndClientOpts fast [MR "SERVER_ERROR busy\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]++ -- A SERVER_ERROR whose text names no status counts against the server+ -- for a request that can be repeated too: once every attempt has got+ -- one, it is thrown and the server is marked dead. Behind memcached's+ -- proxy, a get for a key on a failed backend marks the whole proxy dead+ -- this way.+ it "marks the server dead once every attempt gets a server error" $+ withServerAndClientOpts fastFailover+ [MR "SERVER_ERROR backend failure\r\n", MR "SERVER_ERROR backend failure\r\n"] $ \server client -> do+ Client.get client "key" `shouldThrow` (== ProtocolError (E.ServerError "backend failure"))+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ it "retries after an operation times out" $+ withServerAndClientOpts fast [DelayMS 350 Noop, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \_ client ->+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- Counting the retries down from minBound must not wrap round to+ -- maxBound.+ it "tries an operation once when retries are zero or negative" $+ mapM (\n -> (n,) <$> attempts (withRetries n)) [0, -1, minBound]+ `shouldReturn` [(0, 1), (-1, 1), (minBound, 1)]++ it "tries an operation twice with one retry" $+ attempts (withRetries 1) `shouldReturn` 2++ it "tries an operation three times with two retries" $+ attempts (withRetries 2) `shouldReturn` 3++ it "retries an operation once by default, so tries it twice" $ do+ Client.optsServerRetries Client.def `shouldBe` 1+ attempts dropping `shouldReturn` 2++ it "marks a server dead once its retries run out, then revives it" $+ withServerAndClientOpts fast [CloseConnection, CloseConnection, MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ -- The default of one retry means two attempts in total.+ Client.get client "key" `shouldThrow` isUnexpectedEOF+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)+ threadDelay 400000+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- A retry must be as interruptible as the first attempt: its timeout+ -- must interrupt it wherever it is, not only once it blocks. Here the+ -- retry gets its reply, then works without blocking. Its 300 ms must+ -- still run out, as a server failure that marks the server dead.+ it "times out a retry that never blocks" $+ withServerAndClientOpts fastStaysDead [CloseConnection, MR "MN\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ Cluster.allOp' client (onRetry neverBlocks) `shouldThrow` (== ClientError Timeout)+ mockRequests server `shouldReturn` ["version\r\n", "mn\r\n", "version\r\n", "mn\r\n"]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- So can the caller's own asynchronous exceptions, such as a cancel.+ -- The retry has 5 s, so only the caller can interrupt it in time. An+ -- interrupted operation doesn't count against the server.+ it "lets the caller interrupt a retry that never blocks" $+ withServerAndClientOpts dropping [CloseConnection, MR "MN\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ retrying <- newEmptyMVar+ running <- async $ Cluster.allOp' client (onRetry (putMVar retrying () >> neverBlocks))+ timeout 5000000 (takeMVar retrying) `shouldReturn` Just ()+ start <- getMonotonicTime+ cancel running+ end <- getMonotonicTime+ (end - start) `shouldSatisfy` (< 2)+ mockRequests server `shouldReturn` ["version\r\n", "mn\r\n", "version\r\n", "mn\r\n"]+ readIORef (Server.failed s) `shouldReturn` 0++ -- Each attempt runs with asynchronous exceptions as the caller has them:+ -- a caller that masks them gets every attempt masked, and one that+ -- doesn't, none. The first attempt fails as the mock drops its+ -- connection, and the retry gets MN.+ it "runs every attempt in the caller's masking state" $ do+ let statesOfAttempts withState =+ withServerAndClientOpts dropping [CloseConnection, MR "MN\r\n"] $ \_ client -> do+ states <- newIORef []+ let record = do+ state <- getMaskingState+ atomicModifyIORef' states (\earlier -> (earlier ++ [state], ()))+ _ <- withState $ Cluster.allOp' client $ \s -> Server.withSocket s $ \c ->+ record >> sendRaw c "mn\r\n" >> recvResponse c+ readIORef states+ mapM (\(state, withState) -> (state,) <$> statesOfAttempts withState)+ [(Unmasked, id), (MaskedInterruptible, mask_)]+ `shouldReturn`+ [(Unmasked, [Unmasked, Unmasked]), (MaskedInterruptible, [MaskedInterruptible, MaskedInterruptible])]++ -- The first attempt fails on one of the closed connections: it reads+ -- the end of a closed one, and can't send on a reset one. The retry must+ -- open a new connection rather than take another of them.+ forM_+ [ ("closed", \s -> leaveThreeClosed s "mn\r\n" isUnexpectedEOF)+ , ("reset", \s -> leaveThreeClosed s "mn\r\nmn\r\n" anyIOException)+ ] $ \(ended, leaveThree) ->+ it ("retries on a new connection when the server has " <> ended <> " the idle ones") $+ withServerAndClientOpts (threeConnections fast)+ [CloseConnection, CloseConnection, CloseConnection, MR "EN\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ leaveThree s+ Client.get client "key" `shouldReturn` Nothing+ readIORef (Server.failed s) `shouldReturn` 0+ mockRequests server `shouldReturn`+ replicate 3 "version\r\n" ++ replicate 3 "mn\r\n" ++ ["version\r\n", "mg key v f c\r\n"]++ -- An attempt that times out holding a connection closes the idle ones+ -- too. The mock reads the first connection's requests but never answers+ -- them, and has yet to read the others', so a retry on one of those+ -- would read the reply to its old request, MN. Whichever connection the+ -- first attempt takes, the mock answers a new one with EN.+ it "retries on a new connection when a request times out while others are idle" $+ withServerAndClientOpts (threeConnections fast)+ [Noop, Noop, MR "MN\r\n", MR "MN\r\n", MR "EN\r\n"] $ \_ client -> do+ [s] <- toList <$> Cluster.getServers client+ Server.withSocket s $ \a -> Server.withSocket s $ \b -> Server.withSocket s $ \c ->+ mapM_ (`sendRaw` "mn\r\n") [a, b, c]+ Client.get client "key" `shouldReturn` Nothing+ readIORef (Server.failed s) `shouldReturn` 0++ -- The only attempt fails on one of the closed connections and marks the+ -- server dead, which oneAttempt uses again at once. On Linux, it reads+ -- the end of that connection even once its request has drawn a reset.+ -- The next get must open a new connection rather than take another of+ -- them.+ it "opens a new connection once a server marked dead by a closed connection is back" $+ withServerAndClientOpts (threeConnections oneAttempt)+ [CloseConnection, CloseConnection, CloseConnection, MR "EN\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ leaveThreeClosed s "mn\r\n" isUnexpectedEOF+ Client.get client "key" `shouldThrow` isUnexpectedEOF+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)+ Client.get client "key" `shouldReturn` Nothing+ mockRequests server `shouldReturn`+ replicate 3 "version\r\n" ++ replicate 3 "mn\r\n" ++ ["version\r\n", "mg key v f c\r\n"]++ -- A failed request closes only the idle connections. Here the mock drops+ -- the get's connection 300 ms after the request, while another thread+ -- holds the pool's second connection, which must stay open.+ it "leaves open a connection another thread is using when a request fails" $+ withServerAndClientOpts (threeConnections oneAttempt) [DelayMS 300 CloseConnection, Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ Client.get client "key"+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key v f c\r\n"]+ held <- newEmptyMVar+ release <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \c -> putMVar held c >> takeMVar release+ inUse <- takeMVar held+ wait failing `shouldThrow` isUnexpectedEOF+ putMVar release ()+ wait holder+ sendRaw inUse "mn\r\n" `shouldReturn` ()++ -- The reply to a request that timed out still comes, on its connection,+ -- so the connection must be closed rather than handed to the thread+ -- waiting for the pool, which would read that reply. The waiting thread+ -- uses Server.withSocket, with no time limit, and returns the reply it+ -- reads. The mock serves one connection at a time: it answers that+ -- thread once it has sent the late reply, 250 ms after the get gives up.+ it "doesn't hand the connection of a request that timed out to a waiting thread" $+ withServerAndClientOpts fastStaysDead { Client.optsServerRetries = 0, Client.optsServerTimeout = 500 }+ [DelayMS 750 (MR "VA 3 f9 c9\r\nold\r\n"), MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ timingOut <- async $ Client.get client "key"+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key v f c\r\n"]+ waiting <- async $ Server.withSocket s $ \c -> sendRaw c "mg key v f c\r\n" >> recvResponse c+ waitUntilBlocked waiting+ wait timingOut `shouldThrow` (== ClientError Timeout)+ fmap responseValue <$> timeout 5000000 (wait waiting) `shouldReturn` Just "foo"++ -- The same goes for a request that fails against the server in another+ -- way: a batch's connection may still hold replies it hasn't read, and+ -- one the server dropped can only fail again. retryOp's close s, which+ -- closes the idle connections, can't protect the waiting thread: from+ -- resource-pool 0.3 on, it has the slot as soon as withSocket gives it+ -- back. The mock fails the request 300 ms after it has recorded it.+ forM_+ [ ("a batch whose replies are out of step", \c -> void (Client.getMany c ["one"]),+ [Noop, DelayMS 300 (MR "VA 1 f0 c1 O7\r\nx\r\nVA 3 f9 c9\r\nold\r\nMN\r\n")],+ ["version\r\n", "mg one v f c q O0\r\n", "mn\r\n"], isUnexpectedResponse)+ , ("a batch whose reply has an unknown code", \c -> void (Client.getMany c ["one"]),+ [Noop, DelayMS 300 (MR "XX 2 O0\r\nMN\r\n")],+ ["version\r\n", "mg one v f c q O0\r\n", "mn\r\n"], isUnknownStatus)+ , ("a request the server dropped", \c -> void (Client.get c "key"),+ [DelayMS 300 CloseConnection], ["version\r\n", "mg key v f c\r\n"], isUnexpectedEOF)+ ] $ \(request, run, replies, requests, failure) ->+ it ("doesn't hand the connection of " <> request <> " to a waiting thread") $+ withServerAndClientOpts oneAttempt (replies ++ [MR "VA 3 f2 c7\r\nfoo\r\n"]) $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ run client+ waitForRequests server (length requests) `shouldReturn` requests+ waiting <- async $ Server.withSocket s $ \c -> sendRaw c "mg two v f c\r\n" >> recvResponse c+ waitUntilBlocked waiting+ wait failing `shouldThrow` failure+ fmap responseValue <$> timeout 5000000 (wait waiting) `shouldReturn` Just "foo"++ -- Another thread holds the only connection, through Server.withSocket,+ -- until the operation has given up. Both of its attempts wait their whole+ -- timeout for the connection, so it never reaches the server. Single+ -- requests, batches and stats each reach withSocket by their own path.+ -- The holder lets go after 3 s at the latest, so a wait with no limit+ -- fails the test rather than hanging it.+ forM_+ [ ("a request", \c -> void (Client.delete c "key" 0), [MR "HD\r\n"], ["md key\r\n"])+ , ("a batch", \c -> void (Client.getMany c ["key"]), [Noop, MR "MN\r\n"], ["mg key v f c q O0\r\n", "mn\r\n"])+ , ("stats", \c -> void (Client.stats c Nothing), [MR "END\r\n"], ["stats\r\n"])+ ] $ \(operation, run, replies, requests) ->+ it ("doesn't mark a server dead when " <> operation <> " times out waiting for its only connection") $+ withServerAndClientOpts fastStaysDead replies $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ done <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> timeout 3000000 (takeMVar done)+ takeMVar held+ start <- getMonotonicTime+ run client `shouldThrow` (== ClientError Timeout)+ end <- getMonotonicTime+ putMVar done ()+ _ <- wait holder+ -- It was retried: each attempt waited the whole timeout.+ (end - start) `shouldSatisfy` (>= 0.6)+ readIORef (Server.failed s) `shouldReturn` 0+ run client+ -- The holder opened the connection, and checked the version.+ mockRequests server `shouldReturn` "version\r\n" : requests++ -- An attempt that runs out of time before it holds a slot never reached+ -- the server, so unlike a server failure it closes no idle connection.+ -- With one stripe, an attempt waits only while every slot is taken, so+ -- none is idle then. But with optsServerTimeout = 0, every attempt runs+ -- out of time before it starts. Here the client with that timeout shares+ -- the server with one that has left a connection idle in the pool. If+ -- optsServerTimeout = 0 is ever given another meaning, this test needs+ -- another way to reach that path.+ it "doesn't close the idle connections when a request runs out of time before it gets one" $+ withServerAndClientOpts fastStaysDead [Noop] $ \_ client -> do+ servers <- Cluster.getServers client+ [s] <- return (toList servers)+ idle <- Server.withSocket s return+ impatient <- Client.newClient [] fastStaysDead { Client.optsServerTimeout = 0 }+ Client.get (Cluster.setServers impatient (Right servers)) "key"+ `shouldThrow` (== ClientError Timeout)+ readIORef (Server.failed s) `shouldReturn` 0+ sendRaw idle "mn\r\n" `shouldReturn` ()++ -- A CLIENT_ERROR reply is an answer, not a server failure, so it closes+ -- no idle connection either; a SERVER_ERROR whose text names no status is+ -- a server failure, as a later test checks. The get opens the pool's first+ -- connection, and the mock holds back its reply for 300 ms. Meanwhile a+ -- second connection is opened and given back, so it is idle when the+ -- reply comes.+ it "doesn't close the idle connections when a request gets a CLIENT_ERROR reply" $+ withServerAndClientOpts (threeConnections oneAttempt)+ [DelayMS 300 (MR "CLIENT_ERROR bad\r\n"), Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ Client.get client "key"+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key v f c\r\n"]+ idle <- Server.withSocket s return+ wait failing `shouldThrow` isBadCommand+ readIORef (Server.failed s) `shouldReturn` 0+ sendRaw idle "mn\r\n" `shouldReturn` ()++ -- Nor does the error that stats throws for a listed key's bytes, here an+ -- empty line after a cachedump reply's first entry: stats closes only the+ -- connection the reply came on. As above, the mock holds back the reply+ -- for 300 ms while a second connection is opened and given back.+ it "doesn't close the idle connections when a stats key's bytes hold an empty line" $+ withServerAndClientOpts (threeConnections oneAttempt)+ [DelayMS 300 (MR "ITEM k\n\ny [1 b; 0 s]\r\nEND\r\n"), Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ Client.stats client (Just "cachedump 1 0")+ waitForRequests server 2 `shouldReturn` ["version\r\n", "stats cachedump 1 0\r\n"]+ idle <- Server.withSocket s return+ wait failing `shouldThrow` isUnknownPkt+ readIORef (Server.failed s) `shouldReturn` 0+ sendRaw idle "mn\r\n" `shouldReturn` ()++ -- A request opens its connection, authenticates and checks the version+ -- only once it holds a slot in the pool, so the server is to blame when+ -- any of these never ends, as it is when a request never gets its reply.+ -- The extra Noop keeps the mock reading, so the last attempt times out+ -- rather than reading the end of the connection. The cap turns an+ -- attempt that has lost its time limit into a failure rather than a+ -- hang. An attempt must also open its connection and send within its+ -- 300 ms to reach that stage, and one that runs out of time first counts+ -- against the server all the same, so the requests of one of the two+ -- attempts may be missing. The mocks of the later stages report a+ -- version, which answers each connection's version check without a+ -- scripted response.+ forM_+ [ ("authentication", Auth.Auth "user" "pass", withMockServer, \c -> void (Client.get c "key"), [], ["version\r\n"])+ , ("the version check", Auth.NoAuth, withMockServer, \c -> void (Client.get c "key"), [], ["version\r\n"])+ , ("a request", Auth.NoAuth, withMockMemcached supportedVersion, \c -> void (Client.get c "key"), ["version\r\n"], ["mg key v f c\r\n"])+ , ("a batch", Auth.NoAuth, withMockMemcached supportedVersion, \c -> void (Client.getMany c ["key"]), ["version\r\n"], ["mg key v f c q O0\r\n", "mn\r\n"])+ , ("stats", Auth.NoAuth, withMockMemcached supportedVersion, \c -> void (Client.stats c Nothing), ["version\r\n"], ["stats\r\n"])+ ] $ \(stage, credentials, withMock, run, greeted, requests) ->+ it ("marks a server dead once every attempt times out in " <> stage) $+ withMock (replicate (2 * length requests + 1) Noop) $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = credentials }+ attempt = greeted ++ requests+ client <- Client.newClient [spec] fastStaysDead+ timeout 5000000 (run client) `shouldThrow` (== ClientError Timeout)+ waitForRequests server (2 * length attempt)+ >>= (`shouldSatisfy` (`elem` [attempt, attempt ++ attempt]))+ run client `shouldThrow` (== ClientError NoServersReady)++ -- The delete waits for the only connection, which another thread holds,+ -- and gets it once it is waiting. Its time runs out while the mock holds+ -- back the reply, and as it holds the connection then, that counts+ -- against the server although it waited first: its only attempt marks+ -- the server dead.+ it "marks a server dead when a request that waited for its connection times out using it" $+ withServerAndClientOpts fastStaysDead { Client.optsServerRetries = 0 } [Noop, Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ release <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> takeMVar release+ takeMVar held+ deleting <- async $ Client.delete client "key" 0+ waitUntilBlocked deleting+ putMVar release ()+ wait holder+ wait deleting `shouldThrow` (== ClientError Timeout)+ waitForRequests server 2 `shouldReturn` ["version\r\n", "md key\r\n"]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- An attempt has one timeout, for the wait and the request together.+ -- Here the delete waits 500 ms for the only connection, and the mock+ -- takes 800 ms over the reply, so the delete runs out of its 1 s using+ -- the connection, and its only attempt marks the server dead. With a new+ -- 1 s once it had the connection, it would get the reply.+ it "counts the wait for the connection towards the attempt's timeout" $+ withServerAndClientOpts fastStaysDead { Client.optsServerRetries = 0, Client.optsServerTimeout = 1000 }+ [DelayMS 800 (MR "HD\r\n")] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ release <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> takeMVar release+ takeMVar held+ deleting <- async $ Client.delete client "key" 0+ waitUntilBlocked deleting+ threadDelay 500000+ putMVar release ()+ wait holder+ wait deleting `shouldThrow` (== ClientError Timeout)+ waitForRequests server 2 `shouldReturn` ["version\r\n", "md key\r\n"]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- The last attempt decides whether the server is marked dead. Here the+ -- first attempt times out waiting for the connection, which another+ -- thread holds for 800 ms. The second attempt starts 1 s later, gets the+ -- connection, and the mock drops it.+ it "marks a server dead when a wait timeout is followed by a server failure" $+ withServerAndClientOpts fastStaysDead { Client.optsFailRetryDelay = 1000 }+ [CloseConnection] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> threadDelay 800000+ takeMVar held+ Client.delete client "key" 0 `shouldThrow` isUnexpectedEOF+ wait holder+ mockRequests server `shouldReturn` ["version\r\n", "md key\r\n"]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)+ Client.delete client "key" 0 `shouldThrow` (== ClientError NoServersReady)++ -- The first attempt fails after 300 ms, as the mock drops its connection+ -- or its time runs out: a server failure either way. Another thread,+ -- queued for the connection by then, gets it and holds it until the+ -- touch has given up. The second attempt starts 300 ms later and times+ -- out waiting. A touch can be sent again; a delete would not be.+ it "doesn't mark a server dead when a server failure is followed by a wait timeout" $+ withServerAndClientOpts fastStaysDead { Client.optsFailRetryDelay = 300 }+ [DelayMS 300 CloseConnection, MR "HD\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ touching <- async $ Client.touch client "key" 0+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key T0 c\r\n"]+ held <- newEmptyMVar+ done <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> timeout 3000000 (takeMVar done)+ takeMVar held+ wait touching `shouldThrow` (== ClientError Timeout)+ putMVar done ()+ _ <- wait holder+ readIORef (Server.failed s) `shouldReturn` 0+ Client.touch client "key" 0 `shouldReturn` Just 0+ -- The holder opened the second connection.+ mockRequests server `shouldReturn` ["version\r\n", "mg key T0 c\r\n", "version\r\n", "mg key T0 c\r\n"]++ -- The pool holds one connection and the mock takes 90 ms over each+ -- reply, so only about six of 8 touches sent at once get theirs within+ -- the 600 ms timeout. The rest run out of time, the first of them using+ -- the connection and any others waiting for it, and are retried. Their+ -- retries need about 300 ms of their 600, so every touch succeeds and+ -- the server isn't marked dead. A touch can be sent again; a delete+ -- would not be.+ it "doesn't mark a server dead when retries clear a burst queued for its only connection" $+ withServerAndClientOpts fastStaysDead { Client.optsServerTimeout = 600 }+ (repeat (DelayMS 90 (MR "HD\r\n"))) $ \_ client -> do+ results <- mapConcurrently+ (\i -> try @MemcacheError (Client.touch client (C.pack (show i)) 0)) [1 .. 8 :: Int]+ results `shouldBe` replicate 8 (Right (Just 0))+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0+ Client.touch client "next" 0 `shouldReturn` Just 0++ -- With resource-pool 0.3.0.0 to 0.5.0.0, a connection that failed while+ -- the pool opened it left a request queued for the pool blocked until+ -- its own time ran out. Here the mock drops the first connection 300 ms+ -- into authentication, and the get queued behind it must then open its+ -- own connection.+ it "doesn't leave a request queued for the only connection blocked when opening it fails" $+ withMockServer+ ([DelayMS 300 CloseConnection] ++ authExchange ++ [supported, MR "VA 3 f0 c1\r\nfoo\r\n"]) $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec]+ Client.def { Client.optsServerRetries = 0, Client.optsServerTimeout = 3000 }+ first <- async $ Client.get client "key"+ waitForRequests server 1 `shouldReturn` ["version\r\n"]+ second <- async $ Client.get client "key"+ waitUntilBlocked second+ wait first `shouldThrow` isUnexpectedEOF+ wait second `shouldReturn` Just ("foo", 0, 1)++ -- The same goes for opening a connection that is interrupted, as it is+ -- when the request's time runs out. Here the mock never answers+ -- authentication on the first connection, and the get opening it is+ -- cancelled once the second get is queued behind it. The cap turns a+ -- connect that can't be interrupted into a failure rather than a hang.+ it "doesn't leave a request queued for the only connection blocked when opening it is interrupted" $+ withMockServer+ ([Noop] ++ authExchange ++ [supported, MR "VA 3 f0 c1\r\nfoo\r\n"]) $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec]+ Client.def { Client.optsServerRetries = 0, Client.optsServerTimeout = 3000 }+ first <- async $ Client.get client "key"+ waitForRequests server 1 `shouldReturn` ["version\r\n"]+ second <- async $ Client.get client "key"+ waitUntilBlocked second+ timeout 5000000 (cancel first) `shouldReturn` Just ()+ wait second `shouldReturn` Just ("foo", 0, 1)++ -- With two connections, a request queued behind one that failed to open+ -- stayed blocked even once the other was back in the pool, idle. Here+ -- another thread holds the first connection while the mock drops the+ -- second 300 ms into authentication, and the thread then gives the first+ -- connection back. The get queued behind them must get a connection+ -- rather than run out of time: a new one, or with resource-pool 0.2.3.2+ -- possibly the one given back. It must queue within those 300 ms, as the+ -- drop fails the only attempt of the get opening the second connection+ -- and so marks the server dead. This mock serves both connections at the+ -- same time, as the first stays open throughout.+ it "doesn't leave a request queued for two connections blocked when opening one fails" $+ withConcurrentMockServer+ [ authExchange ++ [supported, MR "VA 3 f0 c1\r\nfoo\r\n"]+ , [DelayMS 300 CloseConnection]+ , authExchange ++ [supported, MR "VA 3 f0 c1\r\nfoo\r\n"] ] $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec] Client.def+ { Client.optsServerRetries = 0, Client.optsServerTimeout = 3000+ , Client.optsServerOptions = Client.defaultServerOptions { Client.soNumResources = 2 } }+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ release <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> takeMVar release+ takeMVar held+ first <- async $ Client.get client "key"+ waitForRequests server 4 `shouldReturn` authRequests ++ ["version\r\n", "version\r\n"]+ second <- async $ Client.get client "key"+ waitUntilBlocked second+ wait first `shouldThrow` isUnexpectedEOF+ putMVar release ()+ wait holder+ wait second `shouldReturn` Just ("foo", 0, 1)++ -- Rejected credentials are an answer, not a server failure. Here the mock+ -- rejects them on the first connection 300 ms into authentication, and+ -- the get queued behind it must get the same answer on a connection of+ -- its own rather than run out of time. Neither answer marks the server+ -- dead.+ it "reports rejected credentials to a request queued for the only connection" $+ withMockServer+ [ DelayMS 300 (MR "CLIENT_ERROR unauthenticated\r\n"), MR "CLIENT_ERROR authentication failure\r\n"+ , MR "CLIENT_ERROR unauthenticated\r\n", MR "CLIENT_ERROR authentication failure\r\n" ] $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec]+ Client.def { Client.optsServerRetries = 0, Client.optsServerTimeout = 3000 }+ first <- async $ Client.get client "key"+ waitForRequests server 1 `shouldReturn` ["version\r\n"]+ second <- async $ Client.get client "key"+ waitUntilBlocked second+ wait first `shouldThrow` (== OpError TextAuthFail)+ wait second `shouldThrow` (== OpError TextAuthFail)+ mockRequests server `shouldReturn`+ [ "version\r\n", "set auth 0 0 9\r\nuser pass\r\n"+ , "version\r\n", "set auth 0 0 9\r\nuser pass\r\n" ]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- A request that mustn't be sent twice isn't sent again once the server+ -- may have it: neither after its reply runs out of time, nor after the+ -- server drops its connection, nor after a SERVER_ERROR reply whose text+ -- names no status, which is a server failure too. The mock would answer a+ -- retry on a new connection. The operation throws the failure, with its+ -- retry unused, and the server is marked dead, as when the last attempt+ -- fails. A retry would have been sent before the operation returned, after+ -- its new connection's version check.+ forM_ sentOnce $ \(name, reply, run) -> do+ it ("doesn't send " <> name <> " again after its reply times out") $+ withServerAndClientOpts fastStaysDead [Noop, MR reply] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ run client `shouldThrow` (== ClientError Timeout)+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ it ("doesn't send " <> name <> " again after the server drops its connection") $+ withServerAndClientOpts fastStaysDead [CloseConnection, MR reply] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ run client `shouldThrow` isUnexpectedEOF+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ it ("doesn't send " <> name <> " again after a server error") $+ withServerAndClientOpts fastStaysDead [MR "SERVER_ERROR backend failure\r\n", MR reply] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ run client `shouldThrow` (== ProtocolError (E.ServerError "backend failure"))+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- Sent again, an append with a version that the first one carried out+ -- would find the version changed, and return Nothing as if it hadn't+ -- matched. Sent once, it throws the failure instead.+ it "doesn't report an append with a version that may have been carried out as a mismatch" $+ withServerAndClientOpts fastStaysDead [Noop, MR "EX c0\r\n"] $ \server client -> do+ Client.append client "key" "v" 9 `shouldThrow` (== ClientError Timeout)+ mockRequests server `shouldReturn` ["version\r\n", "ms key 1 MA C9 c\r\nv\r\n"]++ -- A reply that can't be read, such as one whose value length is+ -- malformed, is a failure after sending too, so the request isn't sent+ -- again either. A get with such a reply is.+ forM_ [("an increment", Client.increment), ("a decrement", Client.decrement)] $ \(name, arith) ->+ it ("doesn't send " <> name <> " again after a reply with a malformed value length") $+ withServerAndClientOpts fastStaysDead [MR "VA (1) c5\r\n6\r\n", MR "VA 1 c5\r\n6\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ arith client "key" 0 1 0 0 `shouldThrow` isBadLength+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- A request counts as sent from when its writing starts, as the server+ -- may have all of it before the write returns. Here a timeout lands while+ -- the request is being written: the listener takes connections and+ -- answers their version check, but never reads from them again, so a+ -- request far larger than the socket buffers can't all be written. A+ -- retry would open a second connection.+ it "doesn't send an append again that times out while it is being written" $+ bracket (N.socket N.AF_INET N.Stream N.defaultProtocol) N.close $ \listener -> do+ N.bind listener loopbackAddress+ N.listen listener 10+ port <- N.socketPort listener+ accepted <- newIORef []+ let takeAll = forever $ do+ (connection, _) <- N.accept listener+ atomicModifyIORef' accepted (\earlier -> (connection : earlier, ()))+ answerVersion connection+ -- Read the version request, one byte at a time so as to read+ -- nothing past it, and report a version the client accepts.+ answerVersion connection = do+ let readLine = do+ byte <- N.recv connection 1+ unless (byte == "\n" || B.null byte) readLine+ readLine+ N.sendAll connection ("VERSION " <> supportedVersion <> "\r\n")+ bracket (forkIO takeAll) killThread $ \_ -> flip finally (readIORef accepted >>= mapM_ N.close) $ do+ client <- Client.newClient [ServerSpec "127.0.0.1" (show port) Auth.NoAuth] fastStaysDead+ Client.append client "key" (C.replicate (32 * 1024 * 1024) 'x') 0+ `shouldThrow` (== ClientError Timeout)+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)+ length <$> readIORef accepted `shouldReturn` 1++ -- Every other request is sent again on a new connection, and the server+ -- isn't marked dead. The mock gets each connection's version check and+ -- the request on it.+ forM_ sentAgain $ \(name, reply, run) ->+ it ("sends " <> name <> " again after the server drops its connection") $+ withServerAndClientOpts fastStaysDead [CloseConnection, MR reply] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ run client+ length <$> mockRequests server `shouldReturn` 4+ readIORef (Server.failed s) `shouldReturn` 0++ -- A failure before any of the request is sent is retried for every+ -- request, and the retry's success leaves the server alive.+ forM_ [("a get", getOnce), ("an increment", incrementOnce)] $ \(name, (run, reply)) -> do+ -- Nothing listens at first, so the first attempt's connection is+ -- refused at once. The mock listens long before the retry, 500 ms+ -- later. Started later, the first attempt would reach the mock and+ -- succeed, so a slow run can only weaken the test, not fail it.+ it ("retries " <> name <> " whose connection was refused") $+ withBoundSocketReuseAddr loopbackAddress $ \port -> do+ let spec = ServerSpec "127.0.0.1" (show port) Auth.NoAuth+ client <- Client.newClient [spec] fastStaysDead { Client.optsFailRetryDelay = 500 }+ running <- async $ run client+ threadDelay 100000+ withMockMemcachedAt supportedVersion (atPort port loopbackAddress) [MR reply] $ \server -> do+ wait running+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- Another thread holds the only connection for 450 ms, so the first+ -- attempt runs out of its 300 ms waiting for it, and the retry, 10 ms+ -- later, gets it.+ it ("retries " <> name <> " that ran out of time waiting for its connection") $+ withServerAndClientOpts fastStaysDead [MR reply] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ held <- newEmptyMVar+ holder <- async $ Server.withSocket s $ \_ -> putMVar held () >> threadDelay 450000+ takeMVar held+ run client+ wait holder+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, False]+ readIORef (Server.failed s) `shouldReturn` 0++ -- The mock drops the first connection during authentication, before+ -- the request is sent. The second authenticates, and then checks the+ -- version.+ it ("retries " <> name <> " whose connection failed while authenticating") $+ withMockServer+ [ CloseConnection, MR "CLIENT_ERROR unauthenticated\r\n", MR "STORED\r\n", supported, MR reply ] $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec] fastStaysDead+ run client+ requests <- mockRequests server+ take 4 requests `shouldBe`+ ["version\r\n", "version\r\n", "set auth 0 0 9\r\nuser pass\r\n", "version\r\n"]+ length requests `shouldBe` 5+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- The mock drops the first connection during its version check, which+ -- is also before the request is sent.+ it ("retries " <> name <> " whose connection failed while checking the version") $+ withMockServer [CloseConnection, supported, MR reply] $ \server -> do+ client <- Client.newClient [mockServerSpec server] fastStaysDead+ run client+ map (== "version\r\n") <$> mockRequests server `shouldReturn` [True, True, False]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0++ -- Before sending a request that mustn't be sent twice, the connection is+ -- checked. One that the server has closed or reset, as a restart or+ -- memcached's idle_timeout leaves them, is closed instead, and that+ -- failure, before sending, is retried: the request is sent once, on a new+ -- connection, and the server isn't marked dead. Sent on the old one, it+ -- would fail after it may have reached the server. leaveThreeClosed reads+ -- each connection, and a read that reports a reset leaves the connection+ -- reading as closed, so here the check finds all three closed. The test+ -- after this one leaves a reset that nothing has read.+ forM_+ [ ("closed the idle ones", \s -> leaveThreeClosed s "mn\r\n" isUnexpectedEOF)+ , ("reset the idle ones and the client has read the resets"+ , \s -> leaveThreeClosed s "mn\r\nmn\r\n" anyIOException)+ ] $ \(ended, leaveThree) ->+ it ("sends an increment once, on a new connection, when the server has " <> ended) $+ withServerAndClientOpts (threeConnections fast)+ [CloseConnection, CloseConnection, CloseConnection, MR "VA 1 c5\r\n6\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ leaveThree s+ Client.increment client "key" 0 1 0 0 `shouldReturn` Just (6, 5)+ readIORef (Server.failed s) `shouldReturn` 0+ mockRequests server `shouldReturn`+ replicate 3 "version\r\n" ++ replicate 3 "mn\r\n" ++ ["version\r\n", "ma key D1 MI v c N0 J0\r\n"]++ -- Here the mock answers an mn on the connection and then resets it, as a+ -- firewall or a load balancer may reset an idle connection, and the+ -- connection goes back to the pool with the reset unread. Sent on it, the+ -- increment would fail once it counts as sent, so it would be thrown and+ -- the server marked dead. The increment must start only once the mock has+ -- reset the connection, or it could be sent before the reset.+ it "sends an increment once, on a new connection, when an unread reset is waiting on the idle one" $ do+ reset <- newEmptyMVar+ withServerAndClientOpts fast [ReplyThenReset "MN\r\n" reset, MR "VA 1 c5\r\n6\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ Server.withSocket s $ \c -> sendRaw c "mn\r\n" >> void (recvResponse c)+ timeout 5000000 (takeMVar reset) `shouldReturn` Just ()+ Client.increment client "key" 0 1 0 0 `shouldReturn` Just (6, 5)+ readIORef (Server.failed s) `shouldReturn` 0+ mockRequests server `shouldReturn`+ ["version\r\n", "mn\r\n", "version\r\n", "ma key D1 MI v c N0 J0\r\n"]++ -- Bytes waiting on an idle connection would be read as the reply to the+ -- next request, so that connection is closed too, before the request is+ -- sent. Here a request is sent on the connection and given back without+ -- its reply being read. The mock answers the increment on a new one,+ -- whose version check leaves nothing waiting.+ it "sends an increment once, on a new connection, when a reply is waiting on the idle one" $+ withServerAndClientOpts fast [MR "MN\r\n", MR "VA 1 c5\r\n6\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ Server.withSocket s (`sendRaw` "mn\r\n")+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mn\r\n"]+ -- The mock sends MN once it has recorded the request.+ threadDelay 100000+ Client.increment client "key" 0 1 0 0 `shouldReturn` Just (6, 5)+ readIORef (Server.failed s) `shouldReturn` 0+ mockRequests server `shouldReturn`+ ["version\r\n", "mn\r\n", "version\r\n", "ma key D1 MI v c N0 J0\r\n"]++ -- Bytes can also wait in the connection's own buffer, where the socket+ -- doesn't show them: an earlier read took them with the reply before+ -- them. Here the mock answers one mn with two MNs, which the client reads+ -- at once, and takes one as the reply. The mock sees the connection+ -- closed before the increment reaches it, on a new one.+ it "sends an increment once, on a new connection, when a reply is waiting in the idle one's buffer" $+ withServerAndClientOpts fast [MR "MN\r\nMN\r\n", MR "VA 1 c5\r\n6\r\n"] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ Server.withSocket s $ \c -> sendRaw c "mn\r\n" >> void (recvResponse c)+ Client.increment client "key" 0 1 0 0 `shouldReturn` Just (6, 5)+ readIORef (Server.failed s) `shouldReturn` 0+ mockClosed server `shouldReturn` 1+ mockRequests server `shouldReturn`+ ["version\r\n", "mn\r\n", "version\r\n", "ma key D1 MI v c N0 J0\r\n"]++ -- A failure after sending that ends the operation still closes the+ -- server's idle connections, as any server failure does. The increment+ -- opens the pool's first connection, and the mock drops it 300 ms after+ -- its request. Meanwhile a second connection is opened and given back.+ it "closes the idle connections when a request sent once fails" $+ withServerAndClientOpts (threeConnections dropping)+ [DelayMS 300 CloseConnection, Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ Client.increment client "key" 0 1 0 0+ waitForRequests server 2 `shouldReturn` ["version\r\n", "ma key D1 MI v c N0 J0\r\n"]+ idle <- Server.withSocket s return+ wait failing `shouldThrow` isUnexpectedEOF+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)+ sendRaw idle "mn\r\n" `shouldThrow` anyIOException++ -- So does a SERVER_ERROR reply whose text names no status, as memcached's+ -- proxy sends when a backend fails, although the server answered. Here+ -- the mock sends it 300 ms after the increment's request, while a second+ -- connection is idle. The increment has a retry left, and doesn't use it.+ it "closes the idle connections and marks the server dead when a request sent once gets a server error" $+ withServerAndClientOpts (threeConnections dropping)+ [DelayMS 300 (MR "SERVER_ERROR backend failure\r\n"), Noop] $ \server client -> do+ [s] <- toList <$> Cluster.getServers client+ failing <- async $ Client.increment client "key" 0 1 0 0+ waitForRequests server 2 `shouldReturn` ["version\r\n", "ma key D1 MI v c N0 J0\r\n"]+ idle <- Server.withSocket s return+ wait failing `shouldThrow` (== ProtocolError (E.ServerError "backend failure"))+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)+ sendRaw idle "mn\r\n" `shouldThrow` anyIOException+ -- The second version request is the idle connection's.+ mockRequests server `shouldReturn`+ ["version\r\n", "ma key D1 MI v c N0 J0\r\n", "version\r\n"]++ -- The mock answers each connection's version check at once, but serves+ -- the rest of its responses one connection at a time, so it only answers a+ -- second client's request once the first client's connection has closed.+ describe "closing connections" $ do+ it "quit closes the client's connections without sending anything" $+ withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n", MR "VA 3 f2 c7\r\nbar\r\n"] $ \server client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ Client.quit client+ getFromNewClient server `shouldReturn` Just ("bar", 2, 7)+ mockRequests server `shouldReturn`+ ["version\r\n", "mg key v f c\r\n", "version\r\n", "mg key v f c\r\n"]++ it "quit closes connections to a server marked dead" $+ withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n", MR "VA 3 f2 c7\r\nbar\r\n"] $ \server client -> do+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ -- A failure time in the future keeps the server marked dead.+ Cluster.getServers client >>= mapM_ (\s -> writeIORef (Server.failed s) 1e10)+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)+ Client.quit client+ getFromNewClient server `shouldReturn` Just ("bar", 2, 7)++ it "quit sends nothing to a server it has no connection to" $+ withServerAndClient [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ Client.quit client+ getFromNewClient server `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n"]++ it "quit closes a connection in use once its request finishes" $+ withServerAndClient [DelayMS 300 (MR "VA 3 f2 c7\r\nfoo\r\n"), MR "VA 3 f2 c7\r\nbar\r\n"] $ \server client -> do+ inFlight <- async $ Client.get client "key"+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key v f c\r\n"]+ Client.quit client+ -- The operation isn't interrupted: it gets its reply.+ wait inFlight `shouldReturn` Just ("foo", 2, 7)+ -- The connection was closed rather than returned to the pool, so the+ -- mock, which serves one connection at a time, can answer a new client.+ getFromNewClient server `shouldReturn` Just ("bar", 2, 7)++ it "closeAll closes a connection in use instead of handing it to a waiting thread" $+ withMockMemcached supportedVersion [Noop, Noop] $ \server -> do+ s <- Server.newServerDefault Server.defaultServerOptions (mockServerSpec server)+ held <- newEmptyMVar+ release <- newEmptyMVar+ first <- async $ Server.withSocket s $ \c -> putMVar held c >> takeMVar release+ c1 <- takeMVar held+ Server.closeAll s+ -- The pool holds one connection, so this waits for the first one.+ second <- async $ Server.withSocket s return+ waitUntilBlocked second+ putMVar release ()+ wait first+ c2 <- wait second+ -- The connection in use was closed, not handed to the waiting thread.+ sendRaw c1 "x\r\n" `shouldThrow` anyException+ sendRaw c2 "x\r\n" `shouldReturn` ()++ -- The mock answers the first request of authentication after 500 ms, so+ -- closeAll runs while the connection is being opened. The function+ -- returns the requests the mock had received when it ran: each+ -- connection authenticated and then checked the version.+ it "closeAll closes a connection being opened instead of handing it to the function" $+ withMockServer+ [ DelayMS 500 (MR "CLIENT_ERROR unauthenticated\r\n"), MR "STORED\r\n", supported+ , MR "CLIENT_ERROR unauthenticated\r\n", MR "STORED\r\n", supported ] $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ s <- Server.newServerDefault Server.defaultServerOptions spec+ opening <- async $ Server.withSocket s $ \_ -> mockRequests server+ waitForRequests server 1 `shouldReturn` ["version\r\n"]+ Server.closeAll s+ -- It ran on a second connection, opened after closeAll. The mock+ -- serves one connection at a time, so it answers the second only+ -- once the first is closed.+ timeout 2000000 (wait opening) `shouldReturn` Just+ (authRequests ++ ["version\r\n"] ++ authRequests ++ ["version\r\n"])++ -- The caller cancels the get while it waits for its reply, which the+ -- mock sends later on the same connection. So that connection must be+ -- closed, or the next get would read that reply.+ it "closes a connection whose request the caller interrupts" $+ withServerAndClient [DelayMS 300 (MR "VA 3 f9 c9\r\nold\r\n"), MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server client -> do+ interrupted <- async $ Client.get client "key"+ waitForRequests server 2 `shouldReturn` ["version\r\n", "mg key v f c\r\n"]+ cancel interrupted+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- A failed attempt closes the server's idle connections. A cancel that+ -- arrives meanwhile must end the operation, not be lost so that it+ -- retries, and must not leave any of them open. Here a listener takes+ -- connections and answers their version check, but nothing else, 500 of+ -- them are idle in the pool, and the get is cancelled as soon as the+ -- first idle one is closed. Without the mask in 'Server.close', this fails+ -- with resource-pool 0.2, which closes them unmasked, and with 0.3, which+ -- has a stripe for each capability and lets an exception in between two of+ -- them. It fails reliably only on more than one capability, which the+ -- suite's +RTS -N gives on a multi-core machine. On one, the get's thread+ -- nearly always keeps the capability until it has closed them all, so the+ -- cancel lands only then, whatever the version. On several, it still lands+ -- that late now and then, so the test runs three times.+ it "lets the caller interrupt a failed request while it closes the idle connections" $+ withOpenFiles 1100 $ replicateM_ 3 $ withSilentListener $ \port idleClosed firstClosed -> do+ let idle = 500+ opts = Client.def+ { Client.optsServerTimeout = 200+ , Client.optsFailRetryDelay = 3000+ , Client.optsServerOptions = Client.defaultServerOptions { Client.soNumResources = idle + 1 } }+ client <- Client.newClient [ServerSpec "127.0.0.1" (show port) Auth.NoAuth] opts+ [s] <- toList <$> Cluster.getServers client+ fillPool s (idle + 1)+ running <- async $ Client.get client "key"+ timeout 5000000 (takeMVar firstClosed) `shouldReturn` Just ()+ start <- getMonotonicTime+ cancel running+ end <- getMonotonicTime+ -- A lost cancel would let the get retry, 3 s later.+ (end - start) `shouldSatisfy` (< 1)+ wait running `shouldThrow` (== AsyncCancelled)+ -- Each close reaches the listener within moments, whereas a+ -- connection left open would be closed only by a later garbage+ -- collection.+ timeout 1000000 (waitFor idle idleClosed) `shouldReturn` Just idle+ keepReachable client++ -- closeAll closes the idle connections the same way, and a cancel that+ -- arrives meanwhile mustn't be lost or leave any of them open either.+ -- The thread waits after closeAll, so that a cancel that lands once+ -- closeAll has returned still ends it at once, as one that closeAll lost+ -- wouldn't. Without the mask in 'Server.closeAll', this fails with+ -- resource-pool 0.2 and 0.3, but, like the get's test, reliably only on+ -- more than one capability.+ it "lets the caller interrupt closeAll while it closes the idle connections" $+ withOpenFiles 1100 $ replicateM_ 3 $ withSilentListener $ \port idleClosed firstClosed -> do+ let idle = 500+ opts = Client.def+ { Client.optsServerOptions = Client.defaultServerOptions { Client.soNumResources = idle } }+ client <- Client.newClient [ServerSpec "127.0.0.1" (show port) Auth.NoAuth] opts+ [s] <- toList <$> Cluster.getServers client+ fillPool s idle+ running <- async $ Server.closeAll s >> threadDelay 3000000+ timeout 5000000 (takeMVar firstClosed) `shouldReturn` Just ()+ start <- getMonotonicTime+ cancel running+ end <- getMonotonicTime+ (end - start) `shouldSatisfy` (< 1)+ wait running `shouldThrow` (== AsyncCancelled)+ timeout 1000000 (waitFor idle idleClosed) `shouldReturn` Just idle+ keepReachable client++ -- quit closes each server's idle connections in turn, and a cancel that+ -- arrives meanwhile must wait until every server's are closed. Here 450+ -- connections to one listener and 50 to another are idle, and quit is+ -- cancelled as soon as the first one to the first server is closed: most+ -- are to that one, so that the cancel lands while quit closes those. As+ -- in closeAll's test, the thread waits after quit. Without the mask in+ -- 'Client.quit', the cancel ended quit before it got to the second+ -- server, with any resource-pool version. Like the get's test, this one+ -- catches that reliably only on more than one capability.+ it "lets the caller interrupt quit while it closes every server's idle connections" $+ withOpenFiles 1100 $ replicateM_ 3 $+ withSilentListener $ \port1 idleClosed1 firstClosed ->+ withSilentListener $ \port2 idleClosed2 _ -> do+ let spec port = ServerSpec "127.0.0.1" (show port) Auth.NoAuth+ opts = Client.def+ { Client.optsServerOptions = Client.defaultServerOptions { Client.soNumResources = 450 } }+ client <- Client.newClient [spec port1, spec port2] opts+ [s1, s2] <- toList <$> Cluster.getServers client+ fillPool s1 450+ fillPool s2 50+ running <- async $ Client.quit client >> threadDelay 3000000+ timeout 5000000 (takeMVar firstClosed) `shouldReturn` Just ()+ start <- getMonotonicTime+ cancel running+ end <- getMonotonicTime+ (end - start) `shouldSatisfy` (< 1)+ wait running `shouldThrow` (== AsyncCancelled)+ timeout 1000000 (waitFor (450, 50) ((,) <$> idleClosed1 <*> idleClosed2))+ `shouldReturn` Just (450, 50)+ keepReachable client++ it "withClient returns the action's result and closes the connection" $+ withMockMemcached supportedVersion [MR "VA 3 f2 c7\r\nfoo\r\n", MR "VA 3 f2 c7\r\nbar\r\n"] $ \server -> do+ (client, result) <- Client.withClient [mockServerSpec server] Client.def $ \client ->+ (,) client <$> Client.get client "key"+ result `shouldBe` Just ("foo", 2, 7)+ getFromNewClient server `shouldReturn` Just ("bar", 2, 7)+ keepReachable client++ it "withClient rethrows the action's exception and closes the connection" $+ withMockMemcached supportedVersion [MR "VA 3 f2 c7\r\nfoo\r\n", MR "VA 3 f2 c7\r\nbar\r\n"] $ \server -> do+ saved <- newEmptyMVar+ Client.withClient [mockServerSpec server] Client.def (\client -> do+ putMVar saved client+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ ioError (userError "boom"))+ `shouldThrow` (== userError "boom")+ getFromNewClient server `shouldReturn` Just ("bar", 2, 7)+ takeMVar saved >>= keepReachable++ it "withClient returns normally when every server has been marked dead" $+ withMockMemcached supportedVersion [CloseConnection, CloseConnection] $ \server ->+ Client.withClient [mockServerSpec server] fastFailover (\client ->+ Client.get client "key" `shouldThrow` isUnexpectedEOF)+ `shouldReturn` ()++ describe "authenticate" $ do+ it "accepts credentials once the server confirms auth-file mode" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "CLIENT_ERROR unauthenticated\r\nSTORED\r\n"+ Auth.authenticate connection (Auth.Auth "user" "pass") `shouldReturn` ()+ recvExactly right (B.length expectedAuthRequests) `shouldReturn` expectedAuthRequests++ it "reports rejected credentials" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "CLIENT_ERROR unauthenticated\r\nCLIENT_ERROR authentication failure\r\n"+ Auth.authenticate connection (Auth.Auth "user" "pass") `shouldThrow` (== OpError TextAuthFail)++ it "refuses to send credentials when the server is not auth-gated" $+ bracket+ (N.socketPair N.AF_UNIX N.Stream 0)+ (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right "VERSION 1.6\r\n"+ Auth.authenticate connection (Auth.Auth "user" "pass") `shouldThrow` isInvalidAuthentication++ it "rejects credentials containing delimiters before contacting the server" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, _) -> do+ connection <- newConnection left+ Auth.authenticate connection (Auth.Auth "us er" "pass") `shouldThrow` isInvalidAuthentication++ it "does nothing without credentials" $+ bracket (N.socketPair N.AF_UNIX N.Stream 0) closePair $ \(left, _) -> do+ connection <- newConnection left+ Auth.authenticate connection Auth.NoAuth `shouldReturn` ()++ it "validates credential delimiters" $ do+ Auth.validateCredentials "user" "pass" `shouldBe` Right ()+ -- memcached's auth file splits each line at its first ':'.+ Auth.validateCredentials "user" "pa:ss" `shouldBe` Right ()+ mapM_ (\c -> Auth.validateCredentials ("us" <> c <> "er") "pass" `shouldSatisfy` isLeft)+ [" ", "\t", "\r", "\n", "\0", ":"]+ mapM_ (\c -> Auth.validateCredentials "user" ("pa" <> c <> "ss") `shouldSatisfy` isLeft)+ [" ", "\t", "\r", "\n", "\0"]++ it "authenticates each new client connection" $+ withMockServer (authExchange ++ [supported, MR "VA 3 f2 c7\r\nfoo\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n", "mg key v f c\r\n"]++ -- memcached reports success with STORED alone. Kept, the connection+ -- would answer the get with the stale reply scripted after HD.+ it "does not use a connection whose authentication got another reply" $+ withMockServer+ [ MR "CLIENT_ERROR unauthenticated\r\n", MR "HD\r\nVA 3 f9 c9\r\nold\r\n"+ , MR "CLIENT_ERROR unauthenticated\r\n", MR "STORED\r\n", supported, MR "VA 3 f2 c7\r\nfoo\r\n" ] $ \server -> do+ let spec = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ client <- Client.newClient [spec] oneAttempt+ Client.get client "key"+ `shouldThrow` (== ProtocolError (UnexpectedResponse "Expected AUTH, got: \"HD\""))+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn`+ authRequests ++ authRequests ++ ["version\r\n", "mg key v f c\r\n"]++ -- Older servers don't read the data block of the meta set this client+ -- sends, so they run the value's bytes as commands. The client must not+ -- send them anything but its version check (and, with credentials, the+ -- authentication exchange before it). Each mock has a reply left over for+ -- a request that should never come, so that one would be recorded.+ describe "server version check" $ do+ forM_ oldVersions $ \v ->+ it ("refuses memcached " <> C.unpack v <> ", sending it nothing but version") $+ withMockServer [MR ("VERSION " <> v <> "\r\n"), MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` versionRefused ("VERSION " <> v)+ mockRequests server `shouldReturn` ["version\r\n"]++ -- Every operation that sends a value, not only set, must be refused+ -- before the value reaches the server.+ forM_ valueOps $ \(name, op) ->+ it ("refuses memcached 1.6.9 for " <> name <> ", sending it nothing but version") $+ withMockServer [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ op client `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]++ -- The batch operations open their connections as the others do. Each of+ -- their requests is a line of its own, so the mock has a reply left over+ -- for each line of a batch.+ forM_ batchOps $ \(name, op) ->+ it ("refuses memcached 1.6.9 for " <> name <> ", sending it nothing but version") $+ withMockServer [MR "VERSION 1.6.9\r\n", Noop, Noop, Noop] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ op client `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]++ forM_ newVersions $ \v ->+ it ("accepts memcached " <> C.unpack v) $+ withMockServer [MR ("VERSION " <> v <> "\r\n"), MR "HD c9\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldReturn` 9+ mockRequests server `shouldReturn` ["version\r\n", "ms key 5 F0 T0 MS c\r\nvalue\r\n"]++ forM_ unreadableReplies $ \reply ->+ it ("refuses a server that answers version with " <> show reply) $+ withMockServer [MR (reply <> "\r\n"), MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` versionRefused reply+ mockRequests server `shouldReturn` ["version\r\n"]++ -- Such as a server in auth-file mode, when the client has no credentials.+ it "throws a CLIENT_ERROR reply to version as it would for any request" $+ withMockServer [MR "CLIENT_ERROR unauthenticated\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0+ `shouldThrow` (== ProtocolError (E.BadCommand "unauthenticated"))+ mockRequests server `shouldReturn` ["version\r\n"]++ it "throws an ERROR reply to version as it would for any request" $+ withMockServer [MR "ERROR\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` (== ProtocolError (UnknownOp "ERROR"))+ mockRequests server `shouldReturn` ["version\r\n"]++ -- A SERVER_ERROR whose text names no status is a server failure, as for+ -- any request: the operation is retried, on a new connection that asks+ -- for the version again, and once every attempt has failed, the server+ -- is marked dead.+ it "counts a SERVER_ERROR reply to version against the server, as it would for any request" $+ withMockServer+ [MR "SERVER_ERROR backend failure\r\n", MR "SERVER_ERROR backend failure\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] fastFailover+ Client.set client "key" "value" 0 0+ `shouldThrow` (== ProtocolError (E.ServerError "backend failure"))+ mockRequests server `shouldReturn` ["version\r\n", "version\r\n"]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) >>= (`shouldNotBe` 0)++ -- stats closes the connection it reads a reply from, so each stats opens+ -- a new connection, which checks the version like any other. Here the+ -- server reports 1.6.9 to the second.+ it "checks the version on the new connection each stats opens" $+ withMockServer+ [supported, MR "STAT pid 1\r\nEND\r\n", MR "VERSION 1.6.9\r\n", MR "STAT pid 2\r\nEND\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ map snd <$> Client.stats client Nothing `shouldReturn` [[("pid", "1")]]+ Client.stats client Nothing `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n", "stats\r\n", "version\r\n"]++ it "describes a refused version in plain words" $+ withMockServer [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` \err -> displayException (err :: MemcacheError) ==+ "memcached protocol error: command not recognized by server: memcached 1.6.10 or later \+ \is required, but the server answered version with \"VERSION 1.6.9\""++ -- The server's version won't change on a retry: a refusal is a client+ -- fault, so the operation fails at once and the server stays alive.+ it "neither retries a refused server nor marks it dead" $+ withMockServer [MR "VERSION 1.6.9\r\n", MR "VERSION 1.6.9\r\n", MR "HD c1\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]+ [s] <- toList <$> Cluster.getServers client+ readIORef (Server.failed s) `shouldReturn` 0+ -- A server marked dead would fail with NoServersReady without being+ -- asked again.+ Client.get client "key" `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n", "version\r\n"]++ it "checks the version again on the connection a retry opens" $+ withMockServer [supported, CloseConnection, MR "VERSION 1.6.9\r\n", MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server -> do+ client <- Client.newClient [mockServerSpec server] fastFailover+ Client.get client "key" `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n", "version\r\n"]++ -- quit never connects, so a refused server gets nothing more from it.+ it "withClient rethrows a refused version, and its quit sends nothing" $+ withMockServer [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do+ Client.withClient [mockServerSpec server] Client.def (\client ->+ Client.set client "key" injection 0 0) `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]++ describe "with credentials" $ do+ forM_ oldVersions $ \v ->+ it ("asks again after authenticating, and refuses memcached " <> C.unpack v) $+ withMockServer (authExchange ++ [MR ("VERSION " <> v <> "\r\n"), MR "HD c1\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` versionRefused ("VERSION " <> v)+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]++ forM_ valueOps $ \(name, op) ->+ it ("refuses memcached 1.6.9 for " <> name <> " after authenticating") $+ withMockServer (authExchange ++ [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ op client `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]++ forM_ batchOps $ \(name, op) ->+ it ("refuses memcached 1.6.9 for " <> name <> " after authenticating") $+ withMockServer (authExchange ++ [MR "VERSION 1.6.9\r\n", Noop, Noop, Noop]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ op client `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]++ forM_ newVersions $ \v ->+ it ("asks again after authenticating, and accepts memcached " <> C.unpack v) $+ withMockServer (authExchange ++ [MR ("VERSION " <> v <> "\r\n"), MR "HD c9\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ Client.set client "key" "value" 0 0 `shouldReturn` 9+ mockRequests server `shouldReturn`+ authRequests ++ ["version\r\n", "ms key 5 F0 T0 MS c\r\nvalue\r\n"]++ forM_ unreadableReplies $ \reply ->+ it ("refuses a server that answers version with " <> show reply <> " after authenticating") $+ withMockServer (authExchange ++ [MR (reply <> "\r\n"), MR "HD c1\r\n"]) $ \server -> do+ client <- Client.newClient [withCredentials server] Client.def+ Client.set client "key" "value" 0 0 `shouldThrow` versionRefused reply+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]++ describe "credentials in output" $ do+ it "shows credentials without the password" $ do+ let auth = Auth.Auth "user" "secret"+ show auth `shouldBe` "Auth {username = \"user\", password = <redacted>}"+ showsPrec 11 auth "" `shouldBe` "(Auth {username = \"user\", password = <redacted>})"+ showsPrec 11 Auth.NoAuth "" `shouldBe` "NoAuth"+ show (Just (ServerSpec "cache.example" "1234" auth)) `shouldBe`+ "Just (ServerSpec {ssHost = \"cache.example\", ssPort = \"1234\", \+ \ssAuth = Auth {username = \"user\", password = <redacted>}})"++ prop "never shows the password" $+ forAll ((,) <$> keyGen <*> keyGen) $ \(user, pass) ->+ let render p =+ let auth = Auth.Auth user p+ spec = ServerSpec "cache.example" "1234" auth+ in map C.pack [show auth, showsPrec 11 auth "", show spec, show (Just spec)]+ -- The password as raw bytes, and as escaped inside a string literal.+ leaks = any (\out -> any (`B.isInfixOf` out) [pass, C.pack (init (drop 1 (show pass)))])+ in not (leaks (render B.empty)) ==>+ counterexample (unlines (map C.unpack (render pass))) (not (leaks (render pass)))++ it "does not quote an invalid URI in the error" $ do+ parseServerSpec "memcached://user:pa ss@cache.example:1234" `shouldBe` Left "Not a valid URI"+ parseServerSpec "memcached://user:secret" `shouldBe` Left "Not a valid URI"+ case EC.parseConfigurationEndpoint "memcached://user:pa ss@c.cfg.example" of+ Left message -> message `shouldBe` "Not a valid URI"+ Right _ -> expectationFailure "parsed an invalid URI"++ -- ElastiCache node addresses come from the server and hold no+ -- credentials, so errors still quote them.+ it "names a malformed ElastiCache node in the error" $ do+ let body = "1\nbad{host|127.0.0.1|11211\n"+ config = "CONFIG cluster 0 " <> C.pack (show (B.length body)) <> "\r\n"+ <> body <> "\r\nEND\r\n"+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config, Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Not a valid URI: memcached://bad{host:11211"++ prop "never echoes the password of an invalid URI" $+ forAll keyGen $ \pass ->+ -- The space makes the URI invalid whatever the password is.+ let message p = either C.pack (const B.empty)+ (parseServerSpec ("memcached://user:" <> C.unpack p <> " @cache.example"))+ in not (pass `B.isInfixOf` message B.empty) ==>+ counterexample (C.unpack (message pass)) (not (pass `B.isInfixOf` message pass))++ describe "server URIs" $ do+ let parses (uri, host, port, auth) =+ it ("parses " <> uri) $+ fmap specFields (parseServerSpec uri) `shouldBe` Right (specFields (ServerSpec host port auth))+ rejects (uri, message) =+ it ("rejects " <> uri) $ fmap specFields (parseServerSpec uri) `shouldBe` Left message+ rawAtMessage =+ "Unix socket path contains @; credentials go right after memcached://, as in "+ <> "memcached://user:password@host, and an @ in a socket path is written %40"++ mapM_ parses+ [ ("memcached://cache.example:1234", "cache.example", "1234", Auth.NoAuth)+ , ("memcached://cache.example", "cache.example", "11211", Auth.NoAuth)+ , ("memcached://", "127.0.0.1", "11211", Auth.NoAuth)+ , ("memcached://user:secret@cache.example:1234", "cache.example", "1234", Auth.Auth "user" "secret")+ -- Credentials need both a username and a password.+ , ("memcached://user@cache.example", "cache.example", "11211", Auth.NoAuth)+ , ("memcached://user:@cache.example", "cache.example", "11211", Auth.NoAuth)+ , ("memcached://:secret@cache.example", "cache.example", "11211", Auth.NoAuth)+ -- Credentials are percent-decoded to bytes, which needn't be UTF-8.+ , ("memcached://us%65r:p%40ss@cache.example", "cache.example", "11211", Auth.Auth "user" "p@ss")+ , ("memcached://user:%e2%82%ac%FF@cache.example", "cache.example", "11211", Auth.Auth "user" "\xe2\x82\xac\xff")+ -- A password may contain ':', escaped or not.+ , ("memcached://user:pa:ss@cache.example", "cache.example", "11211", Auth.Auth "user" "pa:ss")+ , ("memcached://user:pa%3Ass@cache.example", "cache.example", "11211", Auth.Auth "user" "pa:ss")+ -- IPv6 addresses lose their brackets.+ , ("memcached://[::1]:1234", "::1", "1234", Auth.NoAuth)+ , ("memcached://[2001:db8::1]", "2001:db8::1", "11211", Auth.NoAuth)+ , ("memcached://user:secret@[::1]:1234", "::1", "1234", Auth.Auth "user" "secret")+ , ("memcached://[fe80::1%25eth0]:1234", "fe80::1%eth0", "1234", Auth.NoAuth)+ -- A path after an empty host is a Unix socket.+ , ("memcached:///run/memcached.sock", "/run/memcached.sock", "11211", Auth.NoAuth)+ , ("memcached://user:secret@/run/mc%20d.sock", "/run/mc d.sock", "11211", Auth.Auth "user" "secret")+ , ("memcached://cache.example/ignored", "cache.example", "11211", Auth.NoAuth)+ -- Its path is percent-decoded to bytes, one Char each, as SockAddrUnix+ -- writes them.+ , ("memcached:///run/caf%C3%A9.sock", "/run/caf\xc3\xa9.sock", "11211", Auth.NoAuth)+ , ("memcached:///run/a%40b.sock", "/run/a@b.sock", "11211", Auth.NoAuth)+ -- A path of only slashes is no socket: the root directory isn't one.+ , ("memcached:///", "127.0.0.1", "11211", Auth.NoAuth)+ , ("memcached://:11211/", "127.0.0.1", "11211", Auth.NoAuth)+ , ("memcached:////", "127.0.0.1", "11211", Auth.NoAuth)+ , ("memcached:///%2F", "127.0.0.1", "11211", Auth.NoAuth)+ , ("memcached://:11211//", "127.0.0.1", "11211", Auth.NoAuth)+ ]++ mapM_ rejects+ [ ("http://cache.example", "Must begin memcached://")+ , ("not a uri", "Not a valid URI")+ -- The URI parser rejects malformed escapes.+ , ("memcached://user:p%zzss@cache.example", "Not a valid URI")+ -- Decoded credentials must pass validateCredentials, whose message+ -- doesn't quote the URI.+ , ("memcached://us%3Aer:secret@cache.example", "username contains a forbidden delimiter")+ , ("memcached://user:pa%20ss@cache.example", "password contains a forbidden delimiter")+ , ("memcached://user:pa%0Ass@cache.example", "password contains a forbidden delimiter")+ , ("memcached://user:pa%0Dss@cache.example", "password contains a forbidden delimiter")+ , ("memcached://us%00er:secret@cache.example", "username contains a forbidden delimiter")+ , ("memcached://user:pa%00ss@cache.example", "password contains a forbidden delimiter")+ -- A Unix socket takes no port, and its path no NUL, which would end it.+ , ("memcached://:1234/run/memcached.sock", "A Unix socket path takes no port")+ , ("memcached:///run/mc%00d.sock", "Unix socket path contains NUL")+ -- Nor a raw @, as a URI with one slash too many has: as a socket path,+ -- its password would be in ssHost, which show prints. The message says+ -- where credentials go before it says that an @ in a socket path, as in+ -- the last URI, is written %40: in the first three, a %40 would put the+ -- password in ssHost after all.+ , ("memcached:///user:secret@cache.example", rawAtMessage)+ , ("memcached:///user:secret@/run/memcached.sock", rawAtMessage)+ , ("memcached:////user:secret@cache.example", rawAtMessage)+ , ("memcached:///run/a@b.sock", rawAtMessage)+ ]++ -- Without the "//", the URI has no host, not even an empty one, so its+ -- path is no socket. Were it one, the password of the second URI, which+ -- is missing a slash, would be in ssHost, which show prints.+ it "takes no Unix socket path from a URI without //" $+ forM_ ["memcached:/run/memcached.sock", "memcached:/user:secret@cache.example"] $ \uri ->+ fmap ssHost (parseServerSpec uri) `shouldSatisfy` either (const True) (not . isPrefixOf "/")++ it "gives getAddrInfo an IPv6 address it accepts" $ do+ spec <- either fail return $ parseServerSpec "memcached://[::1]:11211"+ let hints = N.defaultHints { N.addrFlags = [N.AI_NUMERICHOST] }+ addrs <- N.getAddrInfo (Just hints) (Just (ssHost spec)) (Just (ssPort spec))+ map N.addrFamily addrs `shouldContain` [N.AF_INET6]++ prop "percent-decodes any credentials validateCredentials accepts" $+ forAll ((,) <$> credentialGen ":" <*> credentialGen "") $ \(user, pass) ->+ let uri = "memcached://" <> percentEncode user <> ":" <> percentEncode pass <> "@cache.example"+ in fmap specFields (parseServerSpec uri) === Right ("cache.example", "11211", Just (user, pass))++ describe "Database.Memcache.Client alone" $+ -- Every name is qualified by Client, so this stops compiling if the module+ -- stops exporting a type its own signatures use.+ it "names the types its signatures use" $+ withMockMemcached supportedVersion [MR "HD c9\r\n", MR "VA 1 c10\r\n3\r\n", MR "STAT pid 1\r\nEND\r\n"] $ \server -> do+ spec <- either fail return $+ Client.parseServerSpec ("memcached://127.0.0.1:" <> show (mockPort server))+ let serverOpts = Client.def { Client.soNumResources = 2 } :: Client.ServerOptions+ client <- Client.newClient [spec] Client.def+ { Client.optsServerRetries = 1 :: Client.Retries+ , Client.optsServerOptions = serverOpts+ }+ cas <- Client.set client ("count" :: Client.Key) ("2" :: Client.Value)+ (0 :: Client.Flags) (60 :: Client.Expiration)+ cas `shouldBe` (9 :: Client.Version)+ Client.increment client "count" (0 :: Client.Initial) (1 :: Client.Delta) 60 cas+ `shouldReturn` Just (3, 10)+ results <- Client.stats client Nothing+ map snd (results :: [(Client.Server, Client.StatResults)]) `shouldBe` [[("pid", "1")]]++ describe "named defaults" $ do+ -- Options and ServerOptions have no Eq instance (Options holds functions),+ -- so these compare the fields that can be compared.+ let serverFields so = (Client.soNumResources so, Client.soNumStripes so, Client.soKeepAlive so)+ optionFields o =+ ( Client.optsServerRetries o, Client.optsFailRetryDelay o+ , Client.optsDeadRetryDelay o, Client.optsServerTimeout o+ , serverFields (Client.optsServerOptions o) )++ it "have the documented values" $ do+ Client.defaultServerSpec `shouldBe` Client.ServerSpec "127.0.0.1" "11211" Client.NoAuth+ serverFields Client.defaultServerOptions `shouldBe` (1, 1, 300)+ optionFields Client.defaultOptions `shouldBe` (1, 200, 1500, 750, (1, 1, 300))++ it "are the values def gives" $ do+ Client.defaultServerSpec `shouldBe` Client.def+ serverFields Client.defaultServerOptions `shouldBe` serverFields Client.def+ optionFields Client.defaultOptions `shouldBe` optionFields Client.def++ describe "clusters" $ do+ it "fails over to another server once one is marked dead" $+ withMockMemcached supportedVersion [MR "VA 1 f0 c1\r\nA\r\n", CloseConnection, CloseConnection] $ \a ->+ withMockMemcached supportedVersion [MR "VA 1 f0 c1\r\nB\r\n", CloseConnection, CloseConnection] $ \b -> do+ client <- Client.newClient [mockServerSpec a, mockServerSpec b] fastFailover+ first <- Client.get client "key"+ -- The server that answered now drops two connections in a row, which+ -- uses up its retries and marks it dead...+ Client.get client "key" `shouldThrow` isUnexpectedEOF+ -- ...so the same key is now served by the other server.+ second <- Client.get client "key"+ map (fmap fst3) [first, second] `shouldSatisfy` (`elem` [[Just "A", Just "B"], [Just "B", Just "A"]])++ it "throws NoServersReady once every server is dead" $+ withMockMemcached supportedVersion [CloseConnection, CloseConnection] $ \a -> do+ client <- Client.newClient [mockServerSpec a] fastFailover+ Client.get client "key" `shouldThrow` isUnexpectedEOF+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)+ Client.flush client Nothing `shouldThrow` (== ClientError NoServersReady)+ Client.version client `shouldThrow` (== ClientError NoServersReady)++ it "flushes every server" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b -> do+ client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]++ it "stops flushing at a server whose reply is rejected" $+ withMockMemcached supportedVersion [MR "HD\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "HD\r\n"] $ \b -> do+ client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def+ Client.flush client Nothing+ `shouldThrow` (== ProtocolError (UnexpectedResponse "Expected FLUSH, got: \"HD\""))+ requests <- (++) <$> mockRequests a <*> mockRequests b+ requests `shouldBe` ["version\r\n", "flush_all\r\n"]++ it "collects stats from every server" $+ withMockMemcached supportedVersion [MR "STAT pid 1\r\nEND\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "STAT pid 2\r\nEND\r\n"] $ \b -> do+ client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def+ results <- Client.stats client Nothing+ concatMap snd results `shouldMatchList` [("pid", "1"), ("pid", "2")]++ it "asks a single server for the version" $+ withMockMemcached supportedVersion [supported] $ \a ->+ withMockMemcached supportedVersion [supported] $ \b -> do+ client <- Client.newClient [mockServerSpec a, mockServerSpec b] Client.def+ Client.version client `shouldReturn` supportedVersion+ -- The connection's own version check, then the operation.+ requests <- sequence [mockRequests a, mockRequests b]+ requests `shouldMatchList` [["version\r\n", "version\r\n"], []]++ -- These are pending unless localhost resolves to more than one address;+ -- see withLocalhostAddresses. Each takes a free port at the first address,+ -- and listens at the same port at the later one.+ describe "connecting" $ do+ -- The socket bound at the first address refuses the connection, and+ -- keeps other processes from listening there.+ it "tries the next address when one refuses the connection" $+ withLocalhostAddresses $ \first later ->+ withBoundSocket first $ \port ->+ withMockMemcachedAt supportedVersion (atPort port later) [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server -> do+ client <- Client.newClient [localhostSpec server Auth.NoAuth] patient+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- The address the client falls back to is checked like any other: the+ -- mock there reports 1.6.9, and must get nothing but version.+ it "checks the version at the next address once one refuses the connection" $+ withLocalhostAddresses $ \first later ->+ withBoundSocket first $ \port ->+ withMockServerAt (atPort port later) [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [localhostSpec server Auth.NoAuth] patient+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]++ -- Sockets bound at both addresses refuse the connection, so the get fails+ -- with a connection error, as it does where a host has one address. The+ -- socket at the later address sets SO_REUSEADDR, as connections to an+ -- earlier mock at that port can still be in TIME_WAIT there. Once a mock+ -- listens at the later address, a new client's get reaches it.+ it "fails with the connection error when every address refuses the connection" $+ withLocalhostAddresses $ \first later ->+ withBoundSocket first $ \port -> do+ let spec = ServerSpec "localhost" (show port) Auth.NoAuth+ refused <- Client.newClient [spec] oneAttempt { Client.optsServerTimeout = 10000 }+ withBoundSocketReuseAddr (atPort port later) $ \_ ->+ Client.get refused "key" `shouldThrow` isDoesNotExistError+ withMockMemcachedAt supportedVersion (atPort port later) [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \_ -> do+ client <- Client.newClient [spec] patient+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- A second mock, at the same port on the later address, records whether+ -- the client tries there too once the credentials are rejected. Once the+ -- first mock's responses have run out, it stops listening, and the next+ -- get moves on to the second mock, which shows that the client can reach+ -- it. A get that connects before then has its connection reset, and its+ -- retry moves on.+ it "does not try the next address once authentication fails" $+ withLocalhostAddresses $ \first later ->+ withMockServerAt first+ [MR "CLIENT_ERROR unauthenticated\r\n", MR "CLIENT_ERROR authentication failure\r\n"] $ \server ->+ withMockServerAt (atPort (mockPort server) later)+ (authExchange ++ [supported, MR "VA 3 f2 c7\r\nfoo\r\n"]) $ \other -> do+ client <- Client.newClient [localhostSpec server (Auth.Auth "user" "pass")] patient+ Client.get client "key" `shouldThrow` (== OpError TextAuthFail)+ mockRequests server `shouldReturn` authRequests+ mockRequests other `shouldReturn` []+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ -- Nor once the version is refused: the server answered, and another+ -- address would only hide that. The first mock refuses two connections,+ -- each with one reply left over for a request that should never come.+ -- The second answers its version check with "HD c1", which is no version+ -- either. Once its replies have run out, the first mock stops listening,+ -- and the next set moves on to the second mock, which shows that the+ -- client can reach it.+ it "does not try the next address once the version is refused" $+ withLocalhostAddresses $ \first later ->+ withMockServerAt first [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server ->+ withMockMemcachedAt supportedVersion (atPort (mockPort server) later) [MR "HD c9\r\n"] $ \other -> do+ client <- Client.newClient [localhostSpec server Auth.NoAuth] patient+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests other `shouldReturn` []+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "HD c1"+ mockRequests server `shouldReturn` ["version\r\n", "version\r\n"]+ mockRequests other `shouldReturn` []+ Client.set client "key" "value" 0 0 `shouldReturn` 9+ mockRequests other `shouldReturn` ["version\r\n", "ms key 5 F0 T0 MS c\r\nvalue\r\n"]++ -- Connecting to the first address gets no answer, as its listener's+ -- queue is full, so the only attempt runs out of time there. The mock at+ -- the later address must get nothing. Once the listener has closed, the+ -- first address refuses, and a new client gets the mock's reply, which+ -- shows that the mock could be reached all along.+ it "tries no later address once one that doesn't answer has used up the attempt's time" $+ withLocalhostAddresses $ \first later ->+ withFullListener first $ \port stopListening ->+ withMockMemcachedAt supportedVersion (atPort port later) [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server -> do+ impatient <- Client.newClient [localhostSpec server Auth.NoAuth]+ oneAttempt { Client.optsServerTimeout = 300 }+ Client.get impatient "key" `shouldThrow` (== ClientError Timeout)+ mockRequests server `shouldReturn` []+ stopListening+ client <- Client.newClient [localhostSpec server Auth.NoAuth] patient+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ describe "Unix sockets" $ do+ -- A host that begins with / is the path of a Unix socket, which the+ -- client connects to without looking anything up: a lookup with this+ -- port, which names no service, would fail. Nor may it set TCP's+ -- options, as a Unix socket rejects NoDelay.+ it "connects to a host that begins with / through the socket at that path" $+ withUnixSocketPath $ \path ->+ withMockMemcachedAt supportedVersion (N.SockAddrUnix path) [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \server -> do+ client <- Client.newClient [ServerSpec path "no-such-service" Auth.NoAuth] oneAttempt+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn` ["version\r\n", "mg key v f c\r\n"]++ -- A connection through a Unix socket checks the version like any other.+ it "refuses memcached 1.6.9 over a Unix socket, sending it nothing but version" $+ withUnixSocketPath $ \path ->+ withMockServerAt (N.SockAddrUnix path) [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \server -> do+ client <- Client.newClient [ServerSpec path "11211" Auth.NoAuth] oneAttempt+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` ["version\r\n"]++ it "refuses memcached 1.6.9 over a Unix socket after authenticating" $+ withUnixSocketPath $ \path ->+ withMockServerAt (N.SockAddrUnix path) (authExchange ++ [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"]) $ \server -> do+ client <- Client.newClient [ServerSpec path "11211" (Auth.Auth "user" "pass")] oneAttempt+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n"]++ it "authenticates over a Unix socket" $+ withUnixSocketPath $ \path ->+ withMockServerAt (N.SockAddrUnix path) (authExchange ++ [supported, MR "VA 3 f2 c7\r\nfoo\r\n"]) $ \server -> do+ client <- Client.newClient [ServerSpec path "11211" (Auth.Auth "user" "pass")] oneAttempt+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)+ mockRequests server `shouldReturn` authRequests ++ ["version\r\n", "mg key v f c\r\n"]++ -- The path is percent-encoded, but for its slashes, in case the+ -- temporary directory's name has characters a URI can't.+ it "connects through the socket a memcached:/// URI names" $+ withUnixSocketPath $ \path ->+ withMockMemcachedAt supportedVersion (N.SockAddrUnix path) [MR "VA 3 f2 c7\r\nfoo\r\n"] $ \_ -> do+ let escape c = if c == '/' then "/" else percentEncode (C.singleton c)+ spec <- either fail return $ parseServerSpec ("memcached://" <> concatMap escape path)+ ssHost spec `shouldBe` path+ client <- Client.newClient [spec] oneAttempt+ Client.get client "key" `shouldReturn` Just ("foo", 2, 7)++ describe "ElastiCache discovery" $ do+ it "rejects an endpoint that is not a configuration endpoint" $+ isLeft (EC.parseConfigurationEndpoint "memcached://cache.example:11211") `shouldBe` True++ -- With this time limit, the reply reaches the length limit long before+ -- discovery could run out of time.+ forM_ [ ("whose first line never ends", C.replicate 4096 'x')+ , ("whose END line never comes", "CONFIG cluster 0 9\r\n" <> C.replicate 4096 'x' <> "\r\n")+ ] $ \(problem, bytes) ->+ it ("gives up on a config reply " <> problem) $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", Repeat bytes] $ \cfg ->+ EC.newClient Client.def { Client.optsServerTimeout = 10000 } hour (endpoint cfg)+ `shouldThrow` isBadLength++ it "discovers nodes with the config command" $+ withMockMemcached supportedVersion [MR "VA 3 f0 c1\r\nfoo\r\n"] $ \node -> do+ let config = "CONFIG cluster 0 " <> nodeList [node] <> "END\r\n"+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config] $ \cfg -> do+ client <- EC.newClient Client.def hour (endpoint cfg)+ Client.get client "key" `shouldReturn` Just ("foo", 0, 1)+ -- The connection's own version check, then discovery's.+ waitForRequests cfg 3 `shouldReturn`+ ["version\r\n", "version\r\n", "config get cluster\r\n"]+ -- So was the node's connection.+ mockRequests node `shouldReturn` ["version\r\n", "mg key v f c\r\n"]++ -- Engines before 1.4.14 keep the node list in a key, which discovery+ -- reads with a plain get. Like any server older than memcached 1.6.10,+ -- they are now refused before discovery sends anything. The mock has a+ -- reply left over for a request that should never come.+ it "refuses a configuration endpoint older than memcached 1.6.10" $+ withMockServer [MR "VA 3 f0 c1\r\nfoo\r\n"] $ \node -> do+ let value = "VALUE AmazonElastiCache:cluster 0 " <> nodeList [node] <> "END\r\n"+ withMockServer [MR "VERSION 1.4.5\r\n", MR value, Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg))+ `shouldThrow` versionRefused "VERSION 1.4.5"+ mockRequests cfg `shouldReturn` ["version\r\n"]+ mockRequests node `shouldReturn` []++ -- The client's connections to the nodes it discovers are checked too.+ it "refuses a discovered node older than memcached 1.6.10" $+ withMockServer [MR "VERSION 1.6.9\r\n", MR "HD c1\r\n"] $ \node -> do+ let config = "CONFIG cluster 0 " <> nodeList [node] <> "END\r\n"+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config] $ \cfg -> do+ client <- EC.newClient Client.def hour (endpoint cfg)+ Client.set client "key" injection 0 0 `shouldThrow` versionRefused "VERSION 1.6.9"+ mockRequests node `shouldReturn` ["version\r\n"]++ -- Discovery still reads the cluster key when the endpoint answers its own+ -- version request with a version before 1.4.14. An endpoint that does+ -- has passed the connection's version check, so it reported 1.6.10 or+ -- later then, as these mocks do.+ it "discovers nodes from the cluster key when discovery's version request gets one before 1.4.14" $+ withMockMemcached supportedVersion [MR "VA 3 f0 c1\r\nfoo\r\n"] $ \node -> do+ let value = "VALUE AmazonElastiCache:cluster 0 " <> nodeList [node] <> "END\r\n"+ withMockMemcached supportedVersion [MR "VERSION 1.4.5\r\n", MR value] $ \cfg -> do+ client <- EC.newClient Client.def hour (endpoint cfg)+ Client.get client "key" `shouldReturn` Just ("foo", 0, 1)+ waitForRequests cfg 3 `shouldReturn`+ ["version\r\n", "version\r\n", "get AmazonElastiCache:cluster\r\n"]++ -- The version only chooses the request for the configuration. Tags, as+ -- in a version that git describe makes, don't change the choice. Here+ -- and in the tests of discovery's version reply below, the mock answers+ -- the connection's own version check, so the reply scripted first+ -- answers discovery's version request.+ forM_+ [ ("1.4.13", "get AmazonElastiCache:cluster")+ , ("1.4.14", "config get cluster")+ , ("1.6.24-15-g1a2b3c4", "config get cluster")+ ] $ \(version, request) ->+ it ("asks an endpoint at version " <> version <> " with " <> request) $+ withMockMemcached supportedVersion [MR ("VERSION " <> C.pack version <> "\r\n"), MR "ERROR\r\n", Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: ERROR"+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n", C.pack request <> "\r\n"]++ -- The version is read leniently: discovery skips the line's first 8+ -- bytes without checking them, and Data.Version.parseVersion takes a+ -- number of any length.+ forM_+ [ ("version 1.4.13", "get AmazonElastiCache:cluster")+ , ("VERSION 1.6.1234567890", "config get cluster")+ ] $ \(line, request) ->+ it ("reads the version reply " <> show line) $+ withMockMemcached supportedVersion [MR (C.pack line <> "\r\n"), MR "ERROR\r\n", Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: ERROR"+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n", C.pack request <> "\r\n"]++ -- A version reply discovery can't read is an error, as another reply it+ -- can't parse is, and discovery asks for nothing more.+ forM_+ ["ERROR", "VERSION 1.6.", "VERSION 1.6.12 extra", "VERSION 1.6.12-"] $ \line ->+ it ("rejects the version reply " <> show line) $+ withMockMemcached supportedVersion [MR (C.pack line <> "\r\n"), Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Unrecognized version reply: " <> line+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n"]++ it "reads a version of up to 64 bytes" $ do+ let versionOf n = "VERSION 1.6.12-" <> C.replicate (n - 7) 'a'+ withMockMemcached supportedVersion [MR (versionOf 64 <> "\r\n"), MR "ERROR\r\n", Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: ERROR"+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n", "config get cluster\r\n"]+ withMockMemcached supportedVersion [MR (versionOf 65 <> "\r\n"), Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Unrecognized version reply: " <> C.unpack (versionOf 65)+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n"]++ -- Reading a version of 32,000 numbers took seconds, which the time limit+ -- can't interrupt where asynchronous exceptions are masked, as in a+ -- bracket's acquire: there newClient overran optsServerTimeout, and with+ -- the configuration sent along with the version, it returned a client.+ -- Now discovery reads no version longer than 64 bytes, so it fails at once.+ forM_+ [ ("", id, MR longVersion)+ , (" inside mask_", mask_, MR longVersion)+ , (" as a bracket's acquire", \make -> bracket make Client.quit pure, MR longVersion)+ , ( " inside mask_ with the configuration sent along", mask_+ , MR (longVersion <> configListing ["127.0.0.1|127.0.0.1|11211"]) )+ ] $ \(how, run, reply) ->+ it ("fails at once on a long version reply" <> how) $+ withMockMemcached supportedVersion [reply, Noop, Noop] $ \cfg -> do+ begin <- getMonotonicTime+ run (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> "Unrecognized version reply: VERSION 1.1.1." `isPrefixOf` message+ elapsed <- subtract begin <$> getMonotonicTime+ elapsed `shouldSatisfy` (< 2)+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n"]++ -- Discovery makes memcached://<host>:<port> of each node line, so hosts+ -- such as these, whose "?" puts the port in the query, would make the+ -- node a Unix socket on this machine, of the endpoint's choosing. The+ -- third gives its URI an empty port, which a socket can't have. The+ -- fourth puts an "@" in the socket's path, and whatever parseServerSpec+ -- makes of that, the error is that the node isn't a plain host and port.+ forM_+ [ ("/var/run/docker.sock?", "Node is not a plain host and port: memcached:///var/run/docker.sock?:11211")+ , ("u:p@/x.sock?", "Node is not a plain host and port: memcached://u:p@/x.sock?:11211")+ , (":/x.sock?", "A Unix socket path takes no port: memcached://:/x.sock?:11211")+ , ("/a@b?", "Node is not a plain host and port: memcached:///a@b?:11211")+ ] $ \(host, expected) ->+ it ("won't take the node " <> C.unpack host <> " for a Unix socket") $ do+ let body = "1\n" <> host <> "|127.0.0.1|11211\n"+ config = "CONFIG cluster 0 " <> C.pack (show (B.length body)) <> "\r\n"+ <> body <> "\r\nEND\r\n"+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config, Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == expected++ -- Other hosts change the node in other ways. "/" or "?" ends the host,+ -- so the port is lost. The host is what follows "@", and what comes+ -- before it is taken as credentials if it holds a user and a password.+ -- An IPv6 address loses its brackets. An empty host would be 127.0.0.1,+ -- and an empty port would fail only at the first request. A host or port+ -- with an "@" gets this error even where parseServerSpec would turn down+ -- the credentials before the "@", as it does a user or password holding+ -- a space, written %20.+ forM_+ [ ("h/x", "22122"), ("h?x", "22122"), ("u@h", "22122"), ("u:p@h", "22122"), ("[::1]", "22122")+ , ("", "11211"), ("h", ""), ("u%20:p@h", "22122"), ("u", "p%20@h")+ ] $+ \(host, port) -> do+ let node = host <> "|10.0.0.1|" <> port+ it ("won't take the node " <> show node) $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR (configListing [node]), Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow` \(StringException message _) ->+ message == "Node is not a plain host and port: memcached://" <> C.unpack host <> ":" <> C.unpack port++ -- An error reply is one line with no END line after it, and memcached+ -- keeps the connection open, as the Noop does: discovery must not wait+ -- for END. Plain memcached answers config get cluster with ERROR, as it+ -- has no config command. The outer timeout turns a hang into a failure.+ forM_ [("1.6.12", "config get cluster"), ("1.4.5", "get AmazonElastiCache:cluster")] $ \(version, request) ->+ forM_ ["ERROR", "CLIENT_ERROR bad command line format", "SERVER_ERROR out of memory"] $ \reply ->+ it ("throws at once when " <> C.unpack request <> " gets " <> reply) $+ withMockMemcached supportedVersion [MR ("VERSION " <> version <> "\r\n"), MR (C.pack reply <> "\r\n"), Noop] $ \cfg -> do+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: " <> reply+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n", request <> "\r\n"]++ -- With no Noop the mock closes the connection after the error line, as+ -- memcached started with -o idle_timeout does to an idle connection. The+ -- error line is reported, not the closed connection.+ it "reports an error reply even when the endpoint then closes the connection" $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR "ERROR\r\n"] $ \cfg ->+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Configuration endpoint replied: ERROR"++ -- A miss is END alone, so the reply's first line is also its last.+ it "reports a missing cluster key when discovery's version request gets a version before 1.4.14" $+ withMockMemcached supportedVersion [MR "VERSION 1.4.5\r\n", MR "END\r\n", Noop] $ \cfg ->+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == "Unrecognized number of lines: 1"++ -- The endpoint's text goes into these errors escaped, and cut to its+ -- first 200 characters, as displayException does for a MemcacheError, so+ -- that it can't break the lines of a log.+ forM_+ [ ("an error reply", MR "SERVER_ERROR x\ESC[2J\nforged log line\r\n",+ "Configuration endpoint replied: SERVER_ERROR x\\ESC[2J\\nforged log line")+ , ("a long error reply", MR ("SERVER_ERROR " <> C.replicate 1000 'x' <> "\r\n"),+ "Configuration endpoint replied: SERVER_ERROR " <> replicate 187 'x' <> "...")+ , ("a malformed node", MR (configListing ["a\ESCb|127.0.0.1|11211"]),+ "Not a valid URI: memcached://a\\ESCb:11211")+ , ("node info it can't split", MR (configListing ["a|\tb"]),+ "Unrecognized components in node info: a|\\tb")+ ] $ \(what, reply, expected) ->+ it ("escapes and shortens the endpoint's text in the error for " <> what) $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", reply, Noop] $ \cfg ->+ timeout 5000000 (EC.newClient Client.def hour (endpoint cfg)) `shouldThrow`+ \(StringException message _) -> message == expected++ -- A config whose node line is empty, or only spaces, lists no nodes. The+ -- error says so, rather than that the line, taken as one empty entry,+ -- isn't a host, an IP address and a port.+ forM_ [("empty", []), ("only spaces", [" "])] $ \(what, entries) ->+ it ("rejects a config whose node line is " <> what) $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR (configListing entries), Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Configuration lists no nodes"++ -- Two spaces between nodes make an empty entry, as the line is split at+ -- each space. The error says that the config lists an empty node, rather+ -- than that the entry isn't a host, an IP address and a port.+ it "rejects a config that lists an empty node" $ do+ let config = configListing ["a|10.0.0.1|1", "", "b|10.0.0.2|2"]+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config, Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Configuration lists an empty node"++ -- ElastiCache allows far fewer nodes in a cluster. A config reply of+ -- 300 KB can list 10,000, and discovery would make a server for each,+ -- and then the first request would wait seconds for the ring that routes+ -- keys. Discovery counts the nodes before it makes any server.+ it "takes a config listing 1,000 nodes, but not 1,001" $ do+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR (configListing (manyNodes 1000))] $ \cfg ->+ withDiscovery Client.def $ \opts _ made -> do+ client <- EC.newClient opts hour (endpoint cfg)+ length <$> Cluster.getServers client `shouldReturn` 1000+ -- newClient called the server constructor for the endpoint and for+ -- each node. As start is never run, the refresh thread is held at+ -- its first call, for the endpoint. Wait until it gets there, so+ -- that it is killed when the test ends. Otherwise the test could end+ -- first, and the thread, not killed, would be held at that call,+ -- keeping the 1,000 servers, each with a pool whose collector thread+ -- wakes every second, until a major GC ended it, or would fail+ -- withDiscovery's check that it has ended.+ waitFor 1002 (length <$> made) `shouldReturn` 1002+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR (configListing (manyNodes 1001)), Noop] $ \cfg ->+ withDiscovery Client.def $ \opts _ made -> do+ EC.newClient opts hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Configuration lists 1001 nodes, over the limit of 1000"+ filter (/= [mockServerSpec cfg]) <$> made `shouldReturn` []++ -- Discovery counts the nodes before it parses any of them, so that a+ -- reply listing too many costs no parsing: the first entry here can't+ -- be parsed, and the count fails first.+ it "counts the nodes before parsing any of them" $+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR (configListing ("bad" : manyNodes 1000)), Noop] $ \cfg ->+ EC.newClient Client.def hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Configuration lists 1001 nodes, over the limit of 1000"++ -- A node listed twice would get a second server, which its keys would+ -- fall back to when the first is marked failed. Nodes are told apart by+ -- host and port, as a refresh does, whatever their IP addresses.+ it "rejects a config that lists a node twice" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b -> do+ let node m ip = "127.0.0.1|" <> ip <> "|" <> C.pack (show (mockPort m))+ config = configListing [node a "127.0.0.1", node b "127.0.0.1", node a "10.0.0.2"]+ withMockMemcached "1.6.12" [MR "VERSION 1.6.12\r\n", MR config, Noop] $ \cfg ->+ withDiscovery Client.def $ \opts _ made -> do+ EC.newClient opts hour (endpoint cfg) `shouldThrow`+ \(StringException message _) -> message == "Node listed twice: 127.0.0.1:" <> show (mockPort a)+ filter (/= [mockServerSpec cfg]) <$> made `shouldReturn` []++ -- When a request is never answered, discovery gives up once+ -- optsServerTimeout has passed. The limit is above the 750 ms default, so+ -- giving up no sooner than 1 s shows that the default was not used. The+ -- extra Noop keeps the mock reading, as an endpoint that has stopped+ -- answering would, rather than closing the connection. The first request+ -- is the connection's own version check, which the last two mocks answer+ -- without a scripted reply.+ forM_ [ ("the connection's version check", withMockServer, [], ["version\r\n"])+ , ("version", withMockMemcached "1.6.12", [], ["version\r\n", "version\r\n"])+ , ("config get cluster", withMockMemcached "1.6.12", [MR "VERSION 1.6.12\r\n"]+ , ["version\r\n", "version\r\n", "config get cluster\r\n"])+ ] $ \(request, withMock, replies, requests) ->+ it ("gives up after optsServerTimeout when the endpoint never answers " <> request) $+ withMock (replies ++ [Noop, Noop]) $ \cfg -> do+ start <- getMonotonicTime+ timeout 5000000 (EC.newClient Client.def { Client.optsServerTimeout = 1000 } hour (endpoint cfg))+ `shouldThrow` (== ClientError Timeout)+ elapsed <- subtract start <$> getMonotonicTime+ elapsed `shouldSatisfy` (>= 1)+ -- Nothing answers the last request, so wait for the mock to record it.+ waitForRequests cfg (length requests) `shouldReturn` requests++ -- Each refresh opens its own connection to the endpoint, which is+ -- checked like any other. Here the endpoint reports 1.6.9 to the+ -- connections of the first two refreshes, so those refreshes fail,+ -- sending nothing but version, and the servers stay as they are. The+ -- endpoint then stops answering.+ it "keeps the servers when a refresh's connection is refused" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockServer ([supported] ++ refreshTo [a]+ ++ [MR "VERSION 1.6.9\r\n", MR "VERSION 1.6.9\r\n"]+ ++ noReplies) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ waitFor 6 (length <$> mockRequests cfg) `shouldReturn` 6+ mockRequests cfg `shouldReturn` ["version\r\n", "version\r\n", "config get cluster\r\n"]+ ++ replicate 3 "version\r\n"+ length <$> Cluster.getServers client `shouldReturn` 1+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]++ it "keeps discovering after a refresh fails" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a]+ -- The endpoint drops the connection and replies with garbage+ -- before a refresh succeeds. Then it lists no nodes, and then it+ -- stops answering.+ ++ [CloseConnection]+ ++ [MR "VERSION 1.6.12\r\n", MR "garbage\r\nEND\r\n"]+ ++ refreshTo [a, b]+ ++ refreshTo []+ ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ -- The first discovery and each refresh connect to the endpoint+ -- afresh, and each new connection first checks its version.+ waitFor 16 (length <$> mockRequests cfg) `shouldReturn` 16+ mockRequests cfg `shouldReturn`+ ["version\r\n", "version\r\n", "config get cluster\r\n", "version\r\n", "version\r\n"]+ ++ concat (replicate 3 ["version\r\n", "version\r\n", "config get cluster\r\n"])+ ++ ["version\r\n", "version\r\n"]+ -- The refresh after the one listing no nodes has started, so that+ -- one has been handled, and listing no nodes must not drop the+ -- servers.+ length <$> Cluster.getServers client `shouldReturn` 2+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]++ -- A refresh that gets an error reply throws, as the first discovery+ -- does. memcached keeps the connection open after one, as the mock does.+ -- The time limit is longer than waitFor waits, so each error reply must+ -- end its refresh at once, rather than leave it to run out of time.+ it "keeps discovering after a refresh gets an error reply" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a]+ ++ [MR "VERSION 1.6.12\r\n", MR "ERROR\r\n"]+ ++ refreshTo [a, b]+ ++ [MR "VERSION 1.6.12\r\n", MR "SERVER_ERROR out of memory\r\n"]+ ++ noReplies) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ -- The first discovery and each refresh connect to the endpoint+ -- afresh, and each new connection first checks its version.+ waitFor 14 (length <$> mockRequests cfg) `shouldReturn` 14+ mockRequests cfg `shouldReturn`+ concat (replicate 4 ["version\r\n", "version\r\n", "config get cluster\r\n"])+ ++ ["version\r\n", "version\r\n"]+ -- The refresh after the second error reply has started, so that one+ -- has been handled, and it may not drop the servers.+ length <$> Cluster.getServers client `shouldReturn` 2+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]++ -- Each connection to the endpoint gets its own replies. The first refresh+ -- gets its reply, which lists b alone, only after the limit. Had the+ -- client waited for it, it would have dropped a, and the next refresh,+ -- which connects again, would have made a's server again.+ it "abandons a refresh slower than optsServerTimeout and runs the next one" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withConcurrentMockMemcached "1.6.12"+ ( refreshTo [a]+ : ( MR "VERSION 1.6.12\r\n"+ : DelayMS 2000 (MR ("CONFIG cluster 0 " <> nodeList [b] <> "END\r\n"))+ : noReplies )+ : refreshTo [a, b]+ : repeat noReplies ) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 1000 } $ \opts start made -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ begin <- getMonotonicTime+ start+ waitFor 2 (length <$> Cluster.getServers client) `shouldReturn` 2+ -- The limit is above the 750 ms default, so taking at least 1 s shows+ -- that the refresh was given the caller's limit.+ elapsed <- subtract begin <$> getMonotonicTime+ elapsed `shouldSatisfy` (>= 1)+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]+ -- Servers were made for a when the client was created and for b at+ -- the second refresh, and never again: a was never dropped.+ filter (/= [mockServerSpec cfg]) <$> made+ `shouldReturn` [[mockServerSpec a], [mockServerSpec b]]++ it "keeps discovering after a refresh gets no reply" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a] ++ [Noop] ++ refreshTo [a, b] ++ noReplies) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 300 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ waitFor 2 (length <$> Cluster.getServers client) `shouldReturn` 2+ -- The refresh after the one that found b gets no reply either. Once+ -- the refresh after that has sent its first request, its connection's+ -- version check, that one has run out of time, and it may not drop+ -- the servers. Each refresh connects to the endpoint afresh, and each+ -- new connection first checks its version.+ waitFor True ((>= 11) . length <$> mockRequests cfg) `shouldReturn` True+ length <$> Cluster.getServers client `shouldReturn` 2+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]++ -- The time limit is longer than waitFor waits, so the reply's length+ -- must end the refresh.+ it "keeps the servers when a refresh gets a config reply that never ends" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached "1.6.12" (refreshTo [a]+ ++ [MR "VERSION 1.6.12\r\n", Repeat ("CONFIG cluster 0 9\r\n" <> C.replicate 4096 'x' <> "\r\n")]+ ++ noReplies) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ -- The refresh after the one whose reply never ends has sent its+ -- request, so that one has ended, and it may not drop the server.+ -- Each new connection to the endpoint first checks its version.+ waitFor 8 (length <$> mockRequests cfg) `shouldReturn` 8+ mockRequests cfg `shouldReturn`+ concat (replicate 2 ["version\r\n", "version\r\n", "config get cluster\r\n"])+ ++ ["version\r\n", "version\r\n"]+ length <$> Cluster.getServers client `shouldReturn` 1+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]++ -- A refresh whose reply lists no nodes fails, rather than dropping every+ -- server, and one whose reply lists too many nodes, an empty node or a+ -- node twice, fails before making a server for any of them. Each keeps+ -- the servers.+ forM_+ [ ("no nodes", [])+ , ("more than 1,000 nodes", manyNodes 1001)+ , ("an empty node", ["one.example|10.0.0.1|11211", "", "two.example|10.0.0.2|11211"])+ , ("a node twice", ["dup.example|10.0.0.1|11211", "dup.example|10.0.0.2|11211"])+ ] $ \(problem, nodes) ->+ it ("keeps the servers when a refresh lists " <> problem) $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a]+ ++ [MR "VERSION 1.6.12\r\n", MR (configListing nodes)]+ ++ refreshTo [a, b]+ ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start made -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ waitFor 2 (length <$> Cluster.getServers client) `shouldReturn` 2+ -- Servers were made for a when the client was created and for b at+ -- the last refresh, and for no node of the refresh between them.+ filter (/= [mockServerSpec cfg]) <$> made+ `shouldReturn` [[mockServerSpec a], [mockServerSpec b]]++ -- A refresh runs with asynchronous exceptions unmasked even when the+ -- client is made inside mask_, as in a bracket's acquire, so that its+ -- time limit can interrupt it while it computes, not only where it+ -- waits. The refresh thread calls the server constructor when it starts,+ -- for the endpoint, and at the refresh that finds b, and each call+ -- records the masking state it runs in.+ it "refreshes unmasked when the client is made inside mask_" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a] ++ refreshTo [a, b] ++ noReplies) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ me <- myThreadId+ states <- newIORef []+ let recording serverOptions specs = do+ thread <- myThreadId+ when (thread /= me) $ do+ state <- getMaskingState+ atomicModifyIORef' states (\earlier -> (state : earlier, ()))+ Client.optsServerSpecsToServers opts serverOptions specs+ client <- mask_ $+ EC.newClient opts { Client.optsServerSpecsToServers = recording } tenMs (endpoint cfg)+ start+ waitFor 2 (length <$> Cluster.getServers client) `shouldReturn` 2+ readIORef states `shouldReturn` [Unmasked, Unmasked]++ it "keeps the connections of nodes still listed" $+ withMockMemcached supportedVersion [MR "OK\r\n", MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ -- Listing b first checks that servers are matched by host and port.+ withMockMemcached "1.6.12" (refreshTo [a] ++ refreshTo [b, a] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ Client.flush client Nothing+ start+ waitFor 2 (length <$> Cluster.getServers client) `shouldReturn` 2+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]+ -- a's mock serves one connection at a time, so with the first still+ -- open, both flushes went over it.+ mockClosed a `shouldReturn` 0++ -- ElastiCache lists a node it has replaced with the same host and port+ -- and a new IP address. The node's connections may lead to the node it+ -- replaced, so they must be closed, though the node keeps its server.+ it "closes the connections of a node whose IP address changes" $+ withMockMemcached supportedVersion [MR "OK\r\n", MR "OK\r\n"] $ \a ->+ withMockMemcached "1.6.12" (refreshTo [a] ++ refreshToAt "10.0.0.2" [a] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ [original] <- toList <$> Cluster.getServers client+ Client.flush client Nothing+ start+ waitFor 1 (mockClosed a) `shouldReturn` 1+ [kept] <- toList <$> Cluster.getServers client+ -- The same server, so the same failure state.+ (Server.failed kept == Server.failed original) `shouldBe` True+ Client.flush client Nothing+ -- The second flush went over a new connection, which checked the+ -- version again.+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n", "version\r\n", "flush_all\r\n"]++ -- a answers the flush late, so the refresh that lists a with a new IP+ -- address runs while the flush is still using a's connection. With a+ -- reply left, a's mock then waits on that connection until it closes.+ it "closes a connection in use by a node whose IP address changes once its request finishes" $+ withMockMemcached supportedVersion [DelayMS 300 (MR "OK\r\n"), MR "OK\r\n"] $ \a ->+ withMockMemcached "1.6.12" (refreshTo [a] ++ refreshToAt "10.0.0.2" [a] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ flushing <- async $ Client.flush client Nothing+ waitForRequests a 2 `shouldReturn` ["version\r\n", "flush_all\r\n"]+ start+ -- The refresh after the one that lists the new IP address has sent+ -- its request, so that one has been handled. Each refresh connects+ -- to the endpoint afresh, and each new connection first checks its+ -- version.+ waitFor 8 (length <$> mockRequests cfg) `shouldReturn` 8+ -- The flush isn't interrupted: it gets a's reply.+ wait flushing+ waitFor 1 (mockClosed a) `shouldReturn` 1++ -- The endpoint's name leads to one of the nodes, so once ElastiCache+ -- replaces that node, a connection to the endpoint made before then may+ -- get no reply. Each connection to the endpoint gets its own replies: the+ -- first refresh's connection stops answering once that refresh is done,+ -- and the next connection lists a with a new IP address. The time limit+ -- is longer than waitFor waits, so a refresh that took the first+ -- refresh's connection again would close a's connection too late.+ it "connects to the endpoint afresh for each refresh" $+ withMockMemcached supportedVersion [MR "OK\r\n", MR "OK\r\n"] $ \a ->+ withConcurrentMockMemcached "1.6.12"+ ( refreshTo [a]+ : (refreshTo [a] ++ noReplies)+ : (refreshToAt "10.0.0.2" [a] ++ noReplies)+ : repeat noReplies ) $ \cfg ->+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ Client.flush client Nothing+ start+ waitFor 1 (mockClosed a) `shouldReturn` 1+ -- The client closed the connections of both refreshes once they were+ -- done, rather than leave them open.+ waitFor 2 (mockClosed cfg) `shouldReturn` 2++ -- Making b's server fails, so the first refresh fails after it has read+ -- the endpoint's reply, with the connection back in the pool. The+ -- refresh must close it all the same. Each connection to the endpoint+ -- gets its own replies, and that one gets no more, so a refresh that+ -- took it again would wait on it for longer than waitFor waits.+ it "closes its connection to the endpoint when a refresh fails to make a server" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withConcurrentMockMemcached "1.6.12"+ ( refreshTo [a]+ : (refreshTo [a, b] ++ noReplies)+ : repeat noReplies ) $ \cfg -> do+ let failForB serverOptions specs+ | specs == [mockServerSpec b] = ioError (userError "no server for b")+ | otherwise = Client.optsServerSpecsToServers Client.def serverOptions specs+ failing = Client.def+ { Client.optsServerTimeout = 10000, Client.optsServerSpecsToServers = failForB }+ withDiscovery failing $ \opts start made -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ waitFor 1 (mockClosed cfg) `shouldReturn` 1+ -- The refresh failed at b's server, and kept a's.+ filter (/= [mockServerSpec cfg]) <$> made+ `shouldReturn` [[mockServerSpec a], [mockServerSpec b]]+ length <$> Cluster.getServers client `shouldReturn` 1++ -- The order breaks ties on the ketama ring. Discovery used to sort the+ -- servers by sid, so both the first discovery and the refresh list the+ -- node with the higher sid first.+ it "keeps the servers in the order discovery lists them in" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \c -> do+ -- Discovery makes its servers from the same specs, so with these+ -- sids. Each is read from a cluster of its own, which can't sort it.+ let sidOf m = map Server.sid . toList+ <$> (Cluster.getServers =<< Cluster.newCluster [mockServerSpec m] Client.def)+ [sidA] <- sidOf a+ [sidB] <- sidOf b+ let (hi, lo) = if sidA > sidB then (a, b) else (b, a)+ withMockMemcached "1.6.12" (refreshTo [hi, lo] ++ refreshTo [hi, lo, c] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow+ -- machine, which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ let ports = map Server.serverPort . toList <$> Cluster.getServers client+ portsOf = map (show . mockPort)+ ports `shouldReturn` portsOf [hi, lo]+ start+ waitFor 3 (length <$> ports) `shouldReturn` 3+ ports `shouldReturn` portsOf [hi, lo, c]++ -- The first refresh adds b, the second fails, and the third lists a and b+ -- again.+ it "reuses the servers of known nodes after a refresh fails" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a]+ ++ refreshTo [a, b]+ ++ [CloseConnection]+ ++ refreshTo [a, b]+ ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start made -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ start+ -- The refresh after the third has sent its request, so the third has+ -- been handled. Each refresh connects to the endpoint afresh, and+ -- each new connection first checks its version.+ waitFor 13 (length <$> mockRequests cfg) `shouldReturn` 13+ mockRequests cfg `shouldReturn`+ concat (replicate 2 ["version\r\n", "version\r\n", "config get cluster\r\n"])+ ++ ["version\r\n", "version\r\n", "version\r\n", "version\r\n", "config get cluster\r\n"]+ ++ ["version\r\n", "version\r\n"]+ -- Servers were made for a when the client was created and for b at+ -- the first refresh, and never again: the failed refresh kept the+ -- nodes the thread knew.+ filter (/= [mockServerSpec cfg]) <$> made+ `shouldReturn` [[mockServerSpec a], [mockServerSpec b]]+ length <$> Cluster.getServers client `shouldReturn` 2+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n"]+ mockRequests b `shouldReturn` ["version\r\n", "flush_all\r\n"]++ it "closes the connections of nodes no longer listed" $+ withMockMemcached supportedVersion [MR "OK\r\n", MR "OK\r\n"] $ \a ->+ -- With a reply left, b's mock waits on its connection until it closes.+ withMockMemcached supportedVersion [MR "OK\r\n", MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a, b] ++ refreshTo [a] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start made -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ Client.flush client Nothing+ initial <- Cluster.getServers client+ start+ waitFor 1 (length <$> Cluster.getServers client) `shouldReturn` 1+ waitFor 1 (mockClosed b) `shouldReturn` 1+ Client.flush client Nothing+ mockRequests a `shouldReturn` ["version\r\n", "flush_all\r\n", "flush_all\r\n"]+ mockClosed a `shouldReturn` 0+ -- The client made the servers of the two nodes it found one at a+ -- time, and the refresh made none. The other calls are for the+ -- endpoint.+ filter (/= [mockServerSpec cfg]) <$> made+ `shouldReturn` [[mockServerSpec a], [mockServerSpec b]]+ -- Checked last so that b's old server stays reachable until now, and+ -- the garbage collector cannot close its connection in place of the+ -- client.+ length initial `shouldBe` 2++ -- b answers the flush late, so the refresh that drops b runs while the+ -- flush is still using b's connection. With a reply left, b's mock then+ -- waits on that connection until it closes.+ it "closes a connection in use by a node no longer listed once its request finishes" $+ withMockMemcached supportedVersion [MR "OK\r\n"] $ \a ->+ withMockMemcached supportedVersion [DelayMS 300 (MR "OK\r\n"), MR "OK\r\n"] $ \b ->+ withMockMemcached "1.6.12" (refreshTo [a, b] ++ refreshTo [a] ++ noReplies) $ \cfg ->+ -- With this time limit no refresh runs out of time on a slow machine,+ -- which would shift the replies.+ withDiscovery Client.def { Client.optsServerTimeout = 10000 } $ \opts start _ -> do+ client <- EC.newClient opts tenMs (endpoint cfg)+ initial <- Cluster.getServers client+ flushing <- async $ Client.flush client Nothing+ waitForRequests b 2 `shouldReturn` ["version\r\n", "flush_all\r\n"]+ start+ waitFor 1 (length <$> Cluster.getServers client) `shouldReturn` 1+ -- The flush isn't interrupted: it gets b's reply.+ wait flushing+ waitFor 1 (mockClosed b) `shouldReturn` 1+ -- Checked last, as in the test above.+ length initial `shouldBe` 2++ routingSpec++ where+ recvResponseFrom bytes =+ bracket+ (N.socketPair N.AF_UNIX N.Stream 0)+ (\(left, right) -> N.close left >> N.close right) $ \(left, right) -> do+ connection <- newConnection left+ _ <- forkIO $ N.sendAll right bytes+ recvResponse connection+ wire key = either (error . show) id (encodeKey key)+ isKeyTooLong (ClientError (KeyTooLong _)) = True+ isKeyTooLong _ = False+ isInvalidAuthentication (ClientError (InvalidAuthentication _)) = True+ isInvalidAuthentication _ = False+ isServerError (ProtocolError (E.ServerError _)) = True+ isServerError _ = False+ isBadCommand (ProtocolError (E.BadCommand _)) = True+ isBadCommand _ = False+ isUnexpectedEOF (ProtocolError (UnexpectedEOF _)) = True+ isUnexpectedEOF _ = False+ isBadLength (ProtocolError (BadLength _)) = True+ isBadLength _ = False+ isUnknownOp (ProtocolError (UnknownOp _)) = True+ isUnknownOp _ = False+ isUnknownStatus (ProtocolError (UnknownStatus _)) = True+ isUnknownStatus _ = False+ isUnknownPkt (ProtocolError (UnknownPkt _)) = True+ isUnknownPkt _ = False+ isUnexpectedResponse (ProtocolError (UnexpectedResponse _)) = True+ isUnexpectedResponse _ = False+ closePair (left, right) = N.close left >> N.close right+ -- How many times a get is tried against a server that drops every+ -- connection. The server must then be marked dead. A get still retrying+ -- after 5 s fails the test, naming the retries, rather than hanging it.+ -- Each attempt opens a new connection, which checks the version first.+ attempts opts = withServerAndClientOpts opts (replicate 5 CloseConnection) $ \server client -> do+ ended <- timeout 5000000 (Client.get client "key" `shouldThrow` isUnexpectedEOF)+ case ended of+ Just () -> pure ()+ Nothing -> expectationFailure $ "still retrying after 5 s, with optsServerRetries = "+ <> show (Client.optsServerRetries opts)+ Client.get client "key" `shouldThrow` (== ClientError NoServersReady)+ requests <- mockRequests server+ let tries = length requests `div` 2+ requests `shouldBe` concat (replicate tries ["version\r\n", "mg key v f c\r\n"])+ return tries+ allStats [(_, [ ("curr_items", "2"), ("rusage_user", "0.1 0.2") ])] = True+ allStats _ = False+ expectedAuthRequests = "version\r\nset auth 0 0 9\r\nuser pass\r\n"+ -- A server in auth-file mode rejects the client's first version request,+ -- then accepts its credentials.+ authExchange = [MR "CLIENT_ERROR unauthenticated\r\n", MR "STORED\r\n"]+ authRequests = ["version\r\n", "set auth 0 0 9\r\nuser pass\r\n"]+ withCredentials server = (mockServerSpec server) { ssAuth = Auth.Auth "user" "pass" }+ supported = MR ("VERSION " <> supportedVersion <> "\r\n")+ -- Versions as memcached reports them, with Ubuntu's suffix and a+ -- release candidate's; 1.10.0 and 1.6.9 catch comparing them as text.+ oldVersions =+ ["1.6.9", "1.5.22", "1.4.25", "1.6.0", "1.4.25 Ubuntu", "1.6.9-rc1", "0.10.10"]+ newVersions =+ ["1.6.10", "1.6.14", "1.6.24", "1.6.100", "1.7.0", "1.10.0", "2.0.0", "1.6.10-rc1", "1.6.24 Ubuntu"]+ unreadableReplies =+ [ "VERSION", "VERSION 1.6", "VERSION 1.6.x", "VERSION v1.6.24", "VERSION 1..6.24"+ , "VERSION 1.6.+10", "VERSION -1.6.24", "HD", "END" ]+ -- The operations that send a value, each storing one that would run as+ -- a command on a server older than 1.6.10.+ valueOps :: [(String, Client.Client -> IO ())]+ valueOps =+ [ ("set", \c -> void $ Client.set c "key" injection 0 0)+ , ("add", \c -> void $ Client.add c "key" injection 0 0)+ , ("replace", \c -> void $ Client.replace c "key" injection 0 0 0)+ , ("cas", \c -> void $ Client.cas c "key" injection 0 0 1)+ , ("append", \c -> void $ Client.append c "key" injection 0)+ , ("prepend", \c -> void $ Client.prepend c "key" injection 0)+ ]+ -- The batch operations, which pipeline their requests.+ batchOps :: [(String, Client.Client -> IO ())]+ batchOps =+ [ ("setMany", \c -> void $ Client.setMany c [("key", injection, 0, 0)])+ , ("deleteMany", \c -> Client.deleteMany c ["key", "other"])+ , ("getMany", \c -> void $ Client.getMany c ["key", "other"])+ ]+ injection :: B.ByteString+ injection = "set injected 0 0 3\r\nbad"+ fastFailover = Client.def { Client.optsFailRetryDelay = 10, Client.optsDeadRetryDelay = 60000 }+ -- One attempt per operation, and a server it marks dead is used again at once.+ oneAttempt = Client.def { Client.optsServerRetries = 0, Client.optsDeadRetryDelay = 0 }+ -- 300 keys, more than a batch holds: a batch of 256, then one of 44.+ batchKeys = [C.pack ("key" <> show i) | i <- [0 .. 299 :: Int]]+ -- Where each batch ended among the requests the mock received: the+ -- positions of the mn requests.+ batchEnds requests = [i | (i, "mn\r\n") <- zip [0 :: Int ..] requests]+ -- The operations whose request mustn't be sent twice, each with a+ -- successful reply the mock would give a retry.+ sentOnce :: [(String, B.ByteString, Client.Client -> IO ())]+ sentOnce =+ [ ("an increment", "VA 1 c5\r\n6\r\n", \c -> void (Client.increment c "key" 0 1 0 0))+ , ("a decrement", "VA 1 c5\r\n4\r\n", \c -> void (Client.decrement c "key" 0 1 0 0))+ , ("an append", "HD c5\r\n", \c -> void (Client.append c "key" "v" 0))+ , ("a prepend", "HD c5\r\n", \c -> void (Client.prepend c "key" "v" 0))+ , ("an append with a version", "HD c5\r\n", \c -> void (Client.append c "key" "v" 9))+ , ("a prepend with a version", "HD c5\r\n", \c -> void (Client.prepend c "key" "v" 9))+ , ("an add", "HD c5\r\n", \c -> void (Client.add c "key" "v" 0 0))+ , ("a delete", "HD\r\n", \c -> void (Client.delete c "key" 0))+ , ("a cas with a version", "HD c5\r\n", \c -> void (Client.cas c "key" "v" 0 0 9))+ , ("a replace with a version", "HD c5\r\n", \c -> void (Client.replace c "key" "v" 0 0 9))+ ]+ -- The operations whose request can be sent again, each with a reply the+ -- mock gives the retry, and a check of the result.+ sentAgain :: [(String, B.ByteString, Client.Client -> Expectation)]+ sentAgain =+ [ ("a get", "VA 1 c5\r\nv\r\n", \c -> Client.get c "key" `shouldReturn` Just ("v", 0, 5))+ , ("a get-and-touch", "VA 1 c5\r\nv\r\n", \c -> Client.gat c "key" 0 `shouldReturn` Just ("v", 0, 5))+ , ("a touch", "HD c5\r\n", \c -> Client.touch c "key" 0 `shouldReturn` Just 5)+ , ("a set", "HD c5\r\n", \c -> Client.set c "key" "v" 0 0 `shouldReturn` 5)+ , ("a cas with version 0", "HD c5\r\n", \c -> Client.cas c "key" "v" 0 0 0 `shouldReturn` Just 5)+ , ("a replace with version 0", "HD c5\r\n", \c -> Client.replace c "key" "v" 0 0 0 `shouldReturn` Just 5)+ , ("a getMany batch", "VA 1 c5 O0\r\nv\r\nMN\r\n", \c -> Client.getMany c ["key"] `shouldReturn` [("key", ("v", 0, 5))])+ , ("a setMany batch", "HD c5 O0\r\nMN\r\n", \c -> Client.setMany c [("key", "v", 0, 0)] `shouldReturn` [("key", 5)])+ , ("a deleteMany batch", "HD O0\r\nMN\r\n", \c -> Client.deleteMany c ["key"])+ , ("a version request", "VERSION 1.6.24\r\n", \c -> Client.version c `shouldReturn` "1.6.24")+ , ("a flush", "OK\r\n", (`Client.flush` Nothing))+ , ("a stats request", "STAT pid 1\r\nEND\r\n", \c -> map snd <$> Client.stats c Nothing `shouldReturn` [[("pid", "1")]])+ ]+ -- A get and an increment, each with the mock's reply to it.+ getOnce = (\c -> Client.get c "key" `shouldReturn` Just ("v", 0, 5), "VA 1 c5\r\nv\r\n")+ incrementOnce = (\c -> Client.increment c "key" 0 1 0 0 `shouldReturn` Just (6, 5), "VA 1 c5\r\n6\r\n")+ loopbackAddress = N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1))+ -- Run a test with a fresh path for a Unix socket, or mark it pending+ -- where there are no Unix sockets.+ withUnixSocketPath test+ | N.isUnixDomainSocketAvailable = withSocketPath test+ | otherwise = pendingWith "Unix sockets are unavailable"+ -- Long enough to read a stats reply of hundreds of thousands of lines.+ slowStats opts = opts { Client.optsServerTimeout = 10000 }+ -- What @stats conns@ reports for one client connection at its longest,+ -- 240 bytes: a five-digit descriptor, a server listening on a Unix socket+ -- at a 107-byte path (the longest memcached takes), the longest state name+ -- and a ten-digit idle time. Over TCP the longest take 224 bytes.+ connStats = "STAT 65535:addr ?:<AF 0>\r\nSTAT 65535:listen_addr unix:" <> C.replicate 107 'p'+ <> "\r\nSTAT 65535:state conn_io_pending\r\nSTAT 65535:secs_since_last_cmd 2147483647\r\n"+ fst3 (value, _, _) = value+ localhostSpec server = ServerSpec "localhost" (show (mockPort server))+ -- Each attempt has 10 s, so that even on a slow machine one is unlikely+ -- to run out of time after using up a mock's reply.+ patient = Client.def { Client.optsServerTimeout = 10000 }+ hour = 3600 * 1000000+ tenMs = 10 * 1000+ -- The endpoint check only looks for ".cfg" somewhere in the URI, so put it+ -- in the path and keep the host a plain loopback address.+ endpoint cfg = either error id $+ EC.parseConfigurationEndpoint ("memcached://127.0.0.1:" <> show (mockPort cfg) <> "/x.cfg")+ -- A config reply body listing the given nodes, from its length line+ -- onwards.+ nodeList = nodeListAt "127.0.0.1"+ -- The same, giving each node the IP address @ip@.+ nodeListAt ip nodes =+ let node n = "127.0.0.1|" <> ip <> "|" <> C.pack (show (mockPort n))+ body = "1\n" <> C.unwords (map node nodes) <> "\n"+ in C.pack (show (B.length body)) <> "\r\n" <> body <> "\r\n"+ -- A version reply of 32,000 numbers, 64,009 bytes long.+ longVersion = "VERSION " <> C.intercalate "." (replicate 32000 "1") <> "\r\n"+ -- Node entries for n nodes, each with a name of its own.+ manyNodes n = ["node" <> C.pack (show i) <> ".example|10.0.0.1|11211" | i <- [1 .. n :: Int]]+ -- A config reply listing the given node entries, each host|IP|port.+ configListing entries =+ let body = "1\n" <> C.unwords entries <> "\n"+ in "CONFIG cluster 0 " <> C.pack (show (B.length body)) <> "\r\n" <> body <> "\r\nEND\r\n"+ -- The config endpoint's replies to one discovery of the given nodes,+ -- whether it creates a client or refreshes one.+ refreshTo = refreshToAt "127.0.0.1"+ refreshToAt ip nodes =+ [MR "VERSION 1.6.12\r\n", MR ("CONFIG cluster 0 " <> nodeListAt ip nodes <> "END\r\n")]+ -- What the endpoint does once a test's script is done: it reads requests+ -- and never answers. A mock that stopped listening instead would give+ -- its port back while the refresh thread, which runs until the test+ -- ends, still connects to it every interval. Another process running+ -- the tests could then get that port for a mock of its own, and the+ -- thread would take that mock's replies.+ noReplies = repeat Noop++-- | Run an action with a client connected to a fresh mock server, which+-- reports 'supportedVersion' to each connection's version check.+withClient :: [MockResponse] -> (Client.Client -> IO a) -> IO a+withClient responses action = withServerAndClient responses (const action)++withServerAndClient :: [MockResponse] -> (MockServer -> Client.Client -> IO a) -> IO a+withServerAndClient = withServerAndClientOpts Client.def++withServerAndClientOpts+ :: Client.Options -> [MockResponse] -> (MockServer -> Client.Client -> IO a) -> IO a+withServerAndClientOpts opts responses action = withMockMemcached supportedVersion responses $ \server ->+ bracket (Client.newClient [mockServerSpec server] opts) keepReachable (action server)++-- | The version that mock servers report unless a test says otherwise: one+-- the client accepts.+supportedVersion :: B.ByteString+supportedVersion = "1.6.24"++-- | Whether an exception is the client refusing a server that answered its+-- version check with the given reply: 'UnknownOp', with a message that quotes+-- the reply and names the minimum version.+versionRefused :: B.ByteString -> MemcacheError -> Bool+versionRefused reply (ProtocolError (UnknownOp message)) =+ show reply `isInfixOf` message && "1.6.10 or later" `isInfixOf` message+versionRefused _ _ = False++-- | Use the client, so it stays reachable until this point: otherwise the+-- pool's finalizers could close its connections while a test is checking that+-- 'Client.quit' closed them.+keepReachable :: Client.Client -> IO ()+keepReachable = Cluster.getServers >=> mapM_ (readIORef . Server.failed)++-- | Get @"key"@ from the mock with a new client.+getFromNewClient :: MockServer -> IO (Maybe (Client.Value, Client.Flags, Client.Version))+getFromNewClient server = do+ client <- Client.newClient [mockServerSpec server] Client.def+ Client.get client "key"++-- | Run an action with a client for two mock servers that routes keys starting+-- with @a@ to the first and all other keys to the second, so a test knows which+-- server sees which key.+withTwoServerClient+ :: [MockResponse] -> [MockResponse]+ -> (MockServer -> MockServer -> Client.Client -> IO a) -> IO a+withTwoServerClient as bs action =+ withMockMemcached supportedVersion as $ \a ->+ withMockMemcached supportedVersion bs $ \b -> do+ [serverA] <- serversFor [mockServerSpec a]+ [serverB] <- serversFor [mockServerSpec b]+ let route cluster key =+ find (== if "a" `B.isPrefixOf` key then serverA else serverB) <$> Cluster.getServers cluster+ client <- Client.newClient [mockServerSpec a, mockServerSpec b]+ Client.def { Client.optsGetServerForKey = route }+ action a b client+ where+ -- Servers compare equal when they are for the same host and port.+ serversFor specs = toList <$> (Cluster.getServers =<< Client.newClient specs Client.def)++-- | Run an action with a client, made with the given options, whose only+-- server is at a port that is bound but never listened on, so that every+-- attempt to connect to it is refused.+withRefusingClient :: Client.Options -> (Client.Client -> IO a) -> IO a+withRefusingClient opts action =+ withBoundSocket (N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1))) $ \port ->+ Client.withClient [ServerSpec "127.0.0.1" (show port) Auth.NoAuth] opts action++-- | The longest value the client sends: 1 GiB, the largest item memcached can+-- store.+valueLimit :: Int+valueLimit = 1073741824++-- | A value one byte longer than 'valueLimit'. It is malloc'd outside the GHC+-- heap and never written, and the code under test reads only its length, so+-- its pages are never touched and the tests stay small. With glibc's+-- MALLOC_PERTURB_ set, malloc fills it, and the tests then take over 1 GB.+-- It is freed once it is garbage.+overLimitValue :: IO B.ByteString+overLimitValue = do+ let len = valueLimit + 1+ bytes <- mallocBytes len+ unsafePackMallocCStringLen (bytes, len)++-- | Wait (up to a second) for the mock to have received at least @n@ requests.+waitForRequests :: MockServer -> Int -> IO [B.ByteString]+waitForRequests server n = do+ result <- timeout 1000000 go+ maybe (mockRequests server) return result+ where+ go = do+ requests <- mockRequests server+ if length requests >= n then return requests else threadDelay 1000 >> go++-- | Poll an action until it returns the expected value, for up to five+-- seconds, returning its last result.+waitFor :: Eq a => a -> IO a -> IO a+waitFor expected action = do+ result <- timeout 5000000 go+ maybe action return result+ where+ go = do+ value <- action+ if value == expected then return value else threadDelay 1000 >> go++-- | Run an ElastiCache test with options whose server constructor holds the+-- discovery thread back, when that thread starts and whenever it adds a node,+-- until the test runs the action it is passed. The test is also passed an+-- action that lists the specs given to each call of the constructor, oldest+-- first. The thread is killed when the test ends, as nothing else stops it,+-- if it has called the constructor by then, and a test that passed then+-- checks that the thread has ended.+withDiscovery+ :: Client.Options -> (Client.Options -> IO () -> IO [[ServerSpec]] -> IO a) -> IO a+withDiscovery opts test = do+ me <- myThreadId+ started <- newEmptyMVar+ threads <- newIORef []+ calls <- newIORef []+ let makeServers serverOptions specs = do+ thread <- myThreadId+ let discovery = thread /= me+ -- Record the thread before its call, so that once a test has seen+ -- the call, the thread is killed when the test ends.+ when discovery $ atomicModifyIORef' threads (\ts -> (thread : ts, ()))+ atomicModifyIORef' calls (\earlier -> (specs : earlier, ()))+ when discovery $ readMVar started+ Client.optsServerSpecsToServers opts serverOptions specs+ -- The thread is recorded at each call it makes, but killed only once.+ discoveryThreads = nub <$> readIORef threads+ result <- test opts { Client.optsServerSpecsToServers = makeServers }+ (void (tryPutMVar started ())) (reverse <$> readIORef calls)+ `finally` (discoveryThreads >>= mapM_ killThread)+ -- A refresh lets asynchronous exceptions through, so the thread ends and+ -- stops connecting to the endpoint's port.+ killed <- discoveryThreads+ forM_ killed $ \thread -> do+ _ <- waitFor True (ended <$> threadStatus thread)+ threadStatus thread >>= (`shouldSatisfy` ended)+ return result+ where+ ended status = status == ThreadFinished || status == ThreadDied++-- | Run an action with a listener on 127.0.0.1 that takes every connection+-- and answers its version check, but nothing else. The action is passed the+-- listener's port, an action that counts the connections closed after their+-- version check without sending anything more, as the client closes an idle+-- one, and an 'MVar' that is filled once the first of them is. The+-- connections are closed when the action ends.+withSilentListener :: (N.PortNumber -> IO Int -> MVar () -> IO a) -> IO a+withSilentListener action =+ bracket (N.socket N.AF_INET N.Stream N.defaultProtocol) N.close $ \listener -> do+ N.bind listener (N.SockAddrInet 0 (N.tupleToHostAddress (127, 0, 0, 1)))+ -- Room in the queue for all the connections a test opens at once:+ -- 'N.maxListenQueue' is only 128 on some systems.+ N.listen listener 1024+ port <- N.socketPort listener+ accepted <- newIORef []+ idleClosed <- newIORef (0 :: Int)+ firstClosed <- newEmptyMVar+ let serve connection = do+ checked <- answerVersion connection+ -- After its version check, the client closes an idle connection+ -- without sending anything.+ request <- N.recv connection 4096+ when (checked && B.null request) $ do+ atomicModifyIORef' idleClosed (\n -> (n + 1, ()))+ void $ tryPutMVar firstClosed ()+ -- Read the version request, one byte at a time so as to read nothing+ -- past it, and report a version the client accepts. False if the+ -- client closed the connection first.+ answerVersion connection = do+ let readLine = do+ byte <- N.recv connection 1+ case byte of+ "" -> return False+ "\n" -> return True+ _ -> readLine+ asked <- readLine+ when asked $ N.sendAll connection ("VERSION " <> supportedVersion <> "\r\n")+ return asked+ takeAll = forever $ do+ (connection, _) <- N.accept listener+ atomicModifyIORef' accepted (\earlier -> (connection : earlier, ()))+ forkIO (serve connection)+ bracket (forkIO takeAll) killThread $ \_ ->+ action port (readIORef idleClosed) firstClosed `finally` (readIORef accepted >>= mapM_ N.close)++-- | Open @n@ connections to the server at once, then give them all back, so+-- that they are idle in its pool. The threads that open them are spread over+-- the capabilities: with resource-pool 0.3, a pool has a stripe for each+-- capability, whatever 'Client.soNumStripes' says, each holding an even share+-- of the connections, and a thread uses its capability's stripe.+fillPool :: Server.Server -> Int -> IO ()+fillPool s n = do+ capabilities <- getNumCapabilities+ arrived <- newIORef (0 :: Int)+ release <- newEmptyMVar+ let hold = Server.withSocket s $ \_ ->+ atomicModifyIORef' arrived (\k -> (k + 1, ())) >> readMVar release+ holders <- mapM (\i -> asyncOn (i `mod` capabilities) hold) [1 .. n]+ waitFor n (readIORef arrived) `shouldReturn` n+ putMVar release ()+ mapM_ wait holders++-- | Wait until the thread is blocked, as when it waits for the pool. A thread+-- that ends instead fails the test rather than hanging it.+waitUntilBlocked :: Async a -> IO ()+waitUntilBlocked thread = do+ status <- threadStatus (asyncThreadId thread)+ case status of+ ThreadRunning -> threadDelay 1000 >> waitUntilBlocked thread+ ThreadBlocked _ -> return ()+ _ -> expectationFailure "the thread ended without blocking"++-- | Receive exactly @n@ bytes from a socket.+recvExactly :: N.Socket -> Int -> IO B.ByteString+recvExactly _ 0 = return B.empty+recvExactly socket n = do+ chunk <- N.recv socket n+ if B.null chunk+ then return B.empty+ else (chunk <>) <$> recvExactly socket (n - B.length chunk)++-- | Run a test with the first address localhost resolves to and a later,+-- different one, or mark it pending where there are no two such addresses that+-- a mock server can listen on. There are where localhost resolves to both ::1+-- and 127.0.0.1 and IPv6 is enabled, but not where it resolves to 127.0.0.1+-- alone or doesn't resolve at all.+withLocalhostAddresses :: (N.SockAddr -> N.SockAddr -> Expectation) -> Expectation+withLocalhostAddresses test = do+ resolved <- tryIO $ N.getAddrInfo (Just hints) (Just "localhost") (Just "0")+ case map N.addrAddress <$> resolved of+ Left err -> pendingWith $ "cannot resolve localhost: " ++ displayException err+ Right addresses@(first : rest) | later : _ <- reverse (filter (/= first) rest) -> do+ listenable <- and <$> mapM canListenAt [first, later]+ if listenable+ then test first later+ else pendingWith $ "cannot listen on the addresses of localhost: " ++ show addresses+ Right addresses -> pendingWith $ "localhost resolves to a single address: " ++ show addresses+ where+ hints = N.defaultHints { N.addrSocketType = N.Stream }++-- | Run a test that has up to @n@ sockets open at once, or mark it pending+-- where the process can't open that many files. The soft limit on open files+-- is raised first, by @n@ or as far as the hard limit allows: the usual ones,+-- 1024 on Linux and 256 on macOS, are too low for these tests.+withOpenFiles :: Int -> Expectation -> Expectation+withOpenFiles n test = do+ raiseFileLimit n+ opened <- newIORef []+ let open = N.socket N.AF_INET N.Stream N.defaultProtocol+ >>= \socket -> atomicModifyIORef' opened (\sockets -> (socket : sockets, ()))+ enough <- isRight <$> tryIO (replicateM_ n open) `finally` (readIORef opened >>= mapM_ N.close)+ if enough then test else pendingWith $ "cannot have " ++ show n ++ " files open at once"++-- | The given address with its port replaced.+atPort :: N.PortNumber -> N.SockAddr -> N.SockAddr+atPort port (N.SockAddrInet _ host) = N.SockAddrInet port host+atPort port (N.SockAddrInet6 _ flow host scope) = N.SockAddrInet6 port flow host scope+atPort _ address = address++-- | Keys of up to 260 arbitrary bytes, biased towards safe ones.+keyGen :: Gen B.ByteString+keyGen = B.pack <$> do+ len <- frequency [(4, choose (1, 64)), (1, choose (180, 260))]+ vectorOf len (frequency [(8, choose (0x21, 0x7e)), (1, arbitrary)])++-- | Non-empty credentials of arbitrary bytes, biased towards ones a URI+-- must escape, without the bytes 'validateCredentials' rejects (space, tab,+-- CR, LF, NUL) or any in @extra@.+credentialGen :: String -> Gen B.ByteString+credentialGen extra = C.pack <$> listOf1 (byte `suchThat` (`notElem` (" \t\r\n\0" <> extra)))+ where+ byte = frequency [(4, choose ('\0', '\xff')), (1, elements ":@%/?#[]")]++-- | Percent-encode every byte except RFC 3986's unreserved characters.+percentEncode :: B.ByteString -> String+percentEncode = concatMap encode . C.unpack+ where+ encode c+ | c `elem` (['A' .. 'Z'] <> ['a' .. 'z'] <> ['0' .. '9'] <> "-._~") = [c]+ | otherwise = ['%', hex (fromEnum c `div` 16), hex (fromEnum c `mod` 16)]+ hex = C.index "0123456789ABCDEF"++-- | A server spec's fields, with the password as bytes: 'show' hides it, so+-- a failed comparison of specs wouldn't show the password that was parsed.+specFields :: ServerSpec -> (String, String, Maybe (B.ByteString, B.ByteString))+specFields (ServerSpec host port auth) = (host, port, credentials auth)+ where+ credentials (Auth.Auth user pass) = Just (user, pass)+ credentials Auth.NoAuth = Nothing++valueGen :: Gen B.ByteString+valueGen = B.pack <$> do+ len <- choose (0, 3000)+ vectorOf len arbitrary++-- | Keys the encoder accepts, plain and base64.+wireKeyGen :: Gen WireKey+wireKeyGen = keyGen `suchThatMap` (either (const Nothing) Just . encodeKey)++-- | Keys of up to 260 arbitrary bytes, often including the bytes 'C.words'+-- splits on: space, tab to CR, and 0xa0, which memcached keeps in a key.+separatorKeyGen :: Gen B.ByteString+separatorKeyGen = B.pack <$> do+ len <- frequency [(4, choose (1, 16)), (1, choose (17, 260))]+ vectorOf len (frequency+ [ (8, choose (0x21, 0x7e)), (2, return 0xa0)+ , (1, elements [0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x20]), (1, arbitrary) ])++-- | Request values, sometimes over the size above which+-- 'Data.ByteString.Builder.byteString' stops copying.+requestValueGen :: Gen B.ByteString+requestValueGen = frequency+ [ (4, valueGen)+ , (1, B.replicate <$> choose (8 * 1024, 20 * 1024) <*> arbitrary)+ ]++-- | Numbers of any size, with the edges of their range picked often.+numberGen :: (Arbitrary a, Bounded a, Num a) => Gen a+numberGen = frequency [(1, elements [0, 1, maxBound]), (4, arbitrary)]++unsafeByte :: (Ord a, Num a) => a -> Bool+unsafeByte c = c <= 0x20 || c == 0x7f++base64Length :: B.ByteString -> Int+base64Length key = 4 * ((B.length key + 2) `div` 3)++-- | Add flags to the end of a request's command line.+addFlags :: B.ByteString -> B.ByteString -> B.ByteString+addFlags flags request = line <> flags <> rest+ where (line, rest) = B.breakSubstring "\r\n" request++-- | A request line: ends in CRLF and contains no other line break.+oneLine :: B.ByteString -> Property+oneLine request =+ counterexample (show request) $+ C.count '\n' request === 1 .&&. C.count '\r' request === 1 .&&.+ property ("\r\n" `B.isSuffixOf` request)++-- | Split bytes at the given fractions of their length.+splitAtFractions :: [Double] -> B.ByteString -> [B.ByteString]+splitAtFractions cuts bytes = go 0 (map toIndex cuts) bytes+ where+ toIndex f = floor (f * fromIntegral (B.length bytes))+ go _ [] rest = [rest]+ go offset (i:is) rest+ | i <= offset = go offset is rest+ | otherwise =+ let (now, later) = B.splitAt (i - offset) rest+ in now : go i is later++-- | Numbers from across a type's whole range, including both ends, rather than+-- only the small ones 'arbitrary' mostly gives.+wordGen :: (Arbitrary a, Bounded a, Integral a) => Gen a+wordGen = oneof [arbitrary, arbitraryBoundedIntegral, elements [minBound, maxBound]]++-- | A number as memcached writes it.+asDecimal :: Show a => a -> B.ByteString+asDecimal = C.pack . show