Misc

ACLs

Buffered HTTP

Constraints

Container Sync Realms

Direct Client

Exceptions

Internal Client

Manager

MemCacheD

Why our own memcache client? By Michael Barton

python-memcached doesn’t use consistent hashing, so adding or removing a memcache server from the pool invalidates a huge percentage of cached items.

If you keep a pool of python-memcached client objects, each client object has its own connection to every memcached server, only one of which is ever in use. So you wind up with n * m open sockets and almost all of them idle. This client effectively has a pool for each server, so the number of backend connections is hopefully greatly reduced.

python-memcache uses pickle to store things, and there was already a huge stink about Swift using pickles in memcache (http://osvdb.org/show/osvdb/86581). That seemed sort of unfair, since nova and keystone and everyone else use pickles for memcache too, but it’s hidden behind a “standard” library. But changing would be a security regression at this point.

Also, pylibmc wouldn’t work for us because it needs to use python sockets in order to play nice with eventlet.

Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:

http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt

class swift.common.memcached.MemcacheConnPool(server, size, connect_timeout)

Bases: eventlet.pools.Pool

Connection pool for Memcache Connections

The server parameter can be a hostname, an IPv4 address, or an IPv6 address with an optional port. If an IPv6 address is specified it must be enclosed in [], like [::1] or [::1]:11211. This follows the accepted prescription for IPv6 host literals.

Examples:

memcache.local:11211
127.0.0.1:11211
[::1]:11211
[::1]
class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.3, io_timeout=2.0, pool_timeout=1.0, tries=3, allow_pickle=False, allow_unpickle=False, max_conns=2)

Bases: object

Simple, consistent-hashed memcache client.

decr(key, delta=1, time=0)

Decrements a key which has a numeric value by delta. Calls incr with -delta.

Parameters:
  • key – key
  • delta – amount to subtract to the value of key (or set the value to 0 if the key is not found) will be cast to an int
  • time – the time to live
Returns:

result of decrementing

Raises MemcacheConnectionError:
 

delete(key)

Deletes a key/value pair from memcache.

Parameters:key – key to be deleted
get(key)

Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.

Parameters:key – key
Returns:value of the key in memcache
get_multi(keys, server_key)

Gets multiple values from memcache for the given keys.

Parameters:
  • keys – keys for values to be retrieved from memcache
  • servery_key – key to use in determining which server in the ring is used
Returns:

list of values

incr(key, delta=1, time=0)

Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.

Parameters:
  • key – key
  • delta – amount to add to the value of key (or set as the value if the key is not found) will be cast to an int
  • time – the time to live
Returns:

result of incrementing

Raises MemcacheConnectionError:
 

set(key, value, serialize=True, time=0, min_compress_len=0)

Set a key/value pair in memcache

Parameters:
  • key – key
  • value – value
  • serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
  • time – the time to live
Min_compress_len:
 

minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it.

set_multi(mapping, server_key, serialize=True, time=0, min_compress_len=0)

Sets multiple key/value pairs in memcache.

Parameters:
  • mapping – dictionary of keys and values to be set in memcache
  • servery_key – key to use in determining which server in the ring is used
  • serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
  • time – the time to live
Min_compress_len:
 

minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it

swift.common.memcached.sanitize_timeout(timeout)

Sanitize a timeout value to use an absolute expiration time if the delta is greater than 30 days (in seconds). Note that the memcached server translates negative values to mean a delta of 30 days in seconds (and 1 additional second), client beware.

Request Helpers

Swob

Utils

WSGI

Storage Policy

Table Of Contents

Previous topic

Object

Next topic

Middleware

This Page