The workloads of social networking sites fall mostly into the ‘read lots, write once’ class (most of the web exists within this paradigm.) Regardless of the database company that’s responsible for the software, the main idea in scaling this read heavy workload is to remove the burden from the database and move it to distributed memory stores.

As an engineer, you want applications to pull from the same cache pool to reduce I/O pressure. To ensure that every machine isn’t replicating data in individual caches, you have to go distributed. That’s the win with memcached.

Putting a distributed cache between the application and the database increases performance and shares data across your application servers, something that the database cannot do on it’s own. The database has on-disk and in memory caching, but eventually you’ll run out of memory on a single host if your working set exceeds the host’s memory.

Memcached also covers up replication lag (MySQL is terrible at replication, Oracle not so much) in large environments by putting data into the distributed cache (Write-through caching) before the slave database has finished it’s writing. Data is available immediately to clients, before the replication has completed.

It will also provide a large amount of savings when you’re constantly executing that O(n x m) query to find out who is friends with whom on your social networking site.

This comes with a cost, though. Relational database functions, like joining across large data sets, and atomic operations, become very difficult to execute. Memcached becomes the central server, and there is always a fear that an important key will drop out of cache because of a random eviction.

It’s not without risk, either. Dependence on the cache can hurt you severely if lots of memcached servers fail (and they do fail), Leaving you in a ‘cold cache’ situation where it can take hours to repopulate your working set back into the cache pool.

Don’t question MySQL’s performance — relational databases are great, but they are not the only solution to storage problems. the two problems that are being solved here are, highly orthogonal.

I’d also like to state that the majority of alternate key-value store databases listed in Richard Jones’ article and in Lenoard Lin’s blog are really not ready for high production loads (with maybe the exception of Tokyo Cabinet, HDFS, and Cassandra). There is still a ton of ‘secret sauce’ the large sites are keeping quiet about in order to make these into effective data stores.

Lin states this in his review as well: “Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.”

Tread lightly.

I turned to CAIDA, the Cooperative Association for Internet Data Analysis, which has long been a provider of excellent network performance tools. Their research focuses on developing tools to measure the Internet in many amazing ways, such as this map of interconnections between the all Autonomous Systems (AS) of the Internet. 

There’s many tools available, but the unofficial standard for bandwidth measurement is iperf. It’s a simple tool to show the maximum possible bandwidth between two points. One machine runs a server, using ‘iperf -s’. The client connects to the server (using iperf -c server) and as much data as can be sent in a single interval is sent. 

Between two of my machines running OS X 10.5.5, I get great results:

retina:/tmp jna$ ./iperf -c hackintosh -i 1
------------------------------------------------------------
Client connecting to hackintosh, TCP port 5001
TCP window size:   129 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.15 port 52150 connected with 10.1.1.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    107 MBytes    894 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec    109 MBytes    912 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec    107 MBytes    901 Mbits/sec

The situation is not so wonderful between my laptop, An Intel Macbook Pro, on 802.11N wireless via a Netgear WNR3500.

dhcp-102:iperf-2.0.4 jna$ src/iperf -i 1 -c hackintosh
------------------------------------------------------------
Client connecting to hackintosh, TCP port 5001
TCP window size:   129 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.102 port 49518 connected with 10.1.1.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  2.45 MBytes  20.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  2.59 MBytes  21.8 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  2.36 MBytes  19.8 Mbits/sec

I don’t believe wireless sales materials anymore (when did I ever?) regarding the maximum speed of these devices. I have five bars on Apple’s wireless icon here, the Macbook Pro supports 802.11N, and I can’t get more than 21.8 Mbits/sec to local machines here. On my Comcast cable, my maximum download speed is around 13 Mbits/sec, so I probably have the best speed possible for downloading from the Internet, but moving files across the local LAN via wireless is a different story. It’s much slower and I’ll go directly to the gigabit, hard wired connection for movies and music.

Interestingly enough, 21.8 Mbits/s is well within 802.11g’s allocation of 25 Mbits/s per client. My 802.11N configuration is no better than 802.11g, even though Apple’s Network Utility reports a link speed of 130Mbits/second and that I have the 802.11 a/b/g/n Network adapter installed.

I’m unsure as to why the network configuration disagrees with the achieved bandwidth, though. It’ll be something I research in the next few days.

I’d also like to take a moment here and condemn Belkin’s entire line of wireless devices. They use a piece of Javascript with breaks the RFC standards in so many ways. If you attempt to configure these devices using CIDR, such as “10.1.1.0/24″, which is what my home network is, their devices force you to a netmask of “255.0.0.0″ because the javascript in the setup form sees 10.0.0.0/8 as a class A network. Classless notation in network allocation is the standard these days, and Class A, B, and C notation is a thing of the past.

This broke my network for hours until I was frustrated enough to bring the device back to Best Buy and purchase the WGR3500. At least I didn’t have to deal with an online return!

For a browser with a completely empty cache, the browser will load the requested library (and block all other requests during the load). For every site the user visits, the library is loaded again and again because the cache does not work across sites. Loading site1.com/jquery.js and site2.com/jquery.js takes the same amount of time.

Google’s come up with a clever solution to this problem. Instead of pointing to your local copy of whatever AJAX library you want, you point to Google’s copy using their AJAX Libraries API.

There’s many advantages to this. If the user has been to Gmail or any Google property recently, the library is already loaded in their browser, ready to execute. If the user goes to other sites which use the API, their cache is already primed and does not need to be loaded again. 

We’ve just put this in place over at Twitter, and it should improve page load times dramatically.

It’s the only piece of software I know of that has ‘–thpppt’ as an option (Install it, and you’ll see.) It’s an optimized replacement for many of your favorite awk | grep | sed combinations, and includes syntax highlighting.

In other news, I’ve resolved a number of bugs and issued new code for running mod_telemetry on 64-bit Linux. Check out the SVN trunk for the latest branch. The data that this module has been providing to me has been invaluable for researching slow points in the back end.