Hello! It’s 2010, a new year and I’ve got some speaking engagements coming up, where I’ll discuss Twitter operations and scaling.
Web 2.0 Expo 2010, San Francisco, CA
May 3rd-6th
In the Belly of the Whale: Operations at Twitter
Chirp (Twitter’s Official Developer Conference)
San Francisco, CA
April 15th, 2010, 4:30PM (Hack Day)
Scaling Twitter: I’ll be speaking on some of the issues we’ve experienced in scaling twitter and you’ll get to meet members of our Ops team.
Velocity 2010
San Francisco, CA
June 22nd-24th, 2010
Waiting for proposal approval, but probably will be similar to my talk at Chirp
Engadget recently featured an article describing YouTube’s blocking of 1080p content from select sites which allowed users to display the content on televisions instead of their computer monitors, or sites which utilized the YouTube API. Like the Hulu block last year, blocking the PS3 from watching shows, it marked another moment when television content producers failed to ‘get it’. Their understanding of content in the face of their own dying industry is poor and misguided.
The blocks on both services are easily removed through the use of a proxy that can replace the browser’s header in the outbound HTTP request.
I have both professional and personal experience in media convergence; Throughout my career I’ve worked for three companies that did streaming video, from adult content (Gamelink), to mainstream media and Independent film (Ifilm/Viacom). On a personal level, the flooding of my loft space has forced me into some temporary housing where I currently cannot not install Internet or Cable service, and I’m forced into using the slow (but not entirely awful) landlord provided WiFi.
Initially the WiFi service was a nightmare, but after the introduction of a pair of Meraki mesh access points, I was able to boost the signal to the point where the PS3 and laptops in the living room could access video. Meraki’s hardware has proven to be excellent under poor signal conditions and simple to use.
On the big Samsung TV that I own, this leaves me with a few options for video at home:
Hulu/Youtube via the PS3
The same, via laptop
Pay-to-play via the Playstation Store
Pay-to-play via iTunes
Basic Cable (no DVR, no channels, no time-shifting)
Most of these are great options (basic cable not withstanding). Laptop based options require me to connect cables, to lose the use of my laptop for the duration of the show, and because of the way the Mac supports full-screen websites, I can’t use fullscreen and the laptop’s screen at the same time. The PS3 is slow to download (although some of the best video I’ve seen on my TV), all Laptop options inconvenient (because of the cables)
It’s not about the technology either; We have the technology! It all works, just not as smoothly as the experience of loafing one’s self in front of the TV and pressing a couple buttons on a remote.
Content creators should be making every attempt to make it easier to consume their content, with advertising. There’s a duality here, where the online video world treats the laptop as a 1st class citizen and the TV as a second class citizen, and vice-versa when it comes to the Big Media world of Television.
All of this is about money — whom is paid and whom is not for the big business of the media world. The blocking needs to stop, and ad revenues shared between the content creators and the new distribution world of digital devices connected to large screens. There is fundamentally no difference between a large monitor, and the large flatscreen in front of my couch.
Here’s two scripts you can run if you have a 3G card, and a few friends stuck in the hotel room with you who want WiFi.
First, turn your laptop into an access point. These scripts work on MacOS X running 10.5.4 or better — good luck!
I call this script ‘make-me-an-access-point.sh’:
#!/bin/sh
#Edit your /etc/rc.conf to set the option firewall_enable to YES
# Edit your /etc/rc.firewall to add lines:
/usr/sbin/natd -dynamic -interface ppp0
sysctl -w net.inet.ip.forwarding=1
/sbin/ipfw -f flush
/sbin/ipfw add 1000 pass all from 127.0.0.1 to 127.0.0.1
/sbin/ipfw add 2000 divert natd ip from any to any via ppp0
/sbin/ipfw add 6500 pass all from any to any
ifconfig en1 192.168.1.1 up netmask 255.255.255.0 broadcast 192.168.0.255
dhcpd
and when you want this to go away, do:
laptop:bin jna$ cat undo-make-me-an-access-point.sh
#!/bin/sh
#
# undo the make-me-an-access-point script
#
sysctl -w net.inet.ip.forwarding=0
sudo /sbin/ipfw -f flush
sudo kill -9 `ps -efl | grep dhcp | egrep -v VMware | awk '{ print $2 }'`
echo "Now turn off the Airport, then wait a moment, and turn it back on."
The contents of my dhcpd.conf are very simple:
# dhcpd.conf
#
# Sample configuration file for ISC dhcpd
#
ddns-update-style ad-hoc;
# option definitions common to all supported networks...
option domain-name "retina.net";
option domain-name-servers 209.183.50.151;
default-lease-time 600;
max-lease-time 7200;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;
# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;
# No service will be given on this subnet, but declaring it helps the
# DHCP server to understand the network topology.
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.200;
option routers 192.168.1.1;
option domain-name-servers 206.13.28.12;
option broadcast-address 192.168.1.255;
}
Read that, perhaps while watching/listening toKevin Kelly’s riveting talk (from the TED EG conference) on the next 5000 days of the web (that’s right, we’re only 5000 days old) and the future of the Semantic web. Kelly says we’re building not a series of small machines on the Internet, but one gigantic thinking machine, approaching the connectivity level of the human mind.
Last Tuesday, I was part of the Velocity 2009 Keynote, where I gave a talk entitled, “Fixing Twitter”. I covered the last year or so of work in improving Twitter to deal with the massive traffic and user loads we’ve been under and how we use metrics to destroy the fail-whale.
Frequently you want to predict when things are going to happen, and if it’s not the end of the world, it might be something occurring a bit sooner, such as your disk filling up.
First capture some data, with cron. We’re going to capture the free space in our database, once a day, so we’ll put something like this in cron, and set it for every night at midnight:
Wait a few days. We were looking at daily file growth, so we waited a full week to collect data. We had the luxury at the time.
Now you’ll have a file with a series of ls entries in it. Run those through awk, and capture the sizes of the files.
cat /tmp/somefile_log | awk '{ print $5 }'
At this point, it’s time to fire up Mathematica. Mathematica is a stunning piece of software by Wolfram Research, used for data visualization, scientific work, and in a number of industries.
First, let’s load the data into Mathematica.
What we’re going to do now is to copy the 1st data point into the Fit, and create a function that will allow us to predict the future.
(* Fit data to a curve using a polynominal model, make sure you insert the 1st data point or the curve fit will be bad *)
result = Fit[data, {336660004864, x, x^2}, x]
(* current free space on our partition (312334824), refunding the 1st data point as that \
space is already alloc'd *)
diskfree = (312334824 * 1024) + Take[data, 1]
(* use the fit function to find doomsday *)
diskfreefunc = diskfree - result
(* when x = 0 , we are dead *)
deathday = NSolve[diskfreefunc == 0, x]
deathday = Take[x /. %, -1]
deathday = deathday[[1]]
DatePlus[datastartdate, N[deathday]]
So now we know when this data set will hit zero, we have the date of that failure, and an ability to graph when that will happen.
For this example, my data set turns into:
We now know that in ~64 days, we’ll run out of disk space. Prediction is pretty nice, huh?
For you stats types, you’ll want to know how good the fit is for this curve, and for that, we look at R2.
I posted this in response to a post on GigaOM, but it was such a long comment, I felt that it was worthy as a post on it’s own.
The workloads of social networking sites fall mostly into the ‘read lots, write once’ class (most of the web exists within this paradigm.) Regardless of the database company that’s responsible for the software, the main idea in scaling this read heavy workload is to remove the burden from the database and move it to distributed memory stores.
As an engineer, you want applications to pull from the same cache pool to reduce I/O pressure. To ensure that every machine isn’t replicating data in individual caches, you have to go distributed. That’s the win with memcached.
Putting a distributed cache between the application and the database increases performance and shares data across your application servers, something that the database cannot do on it’s own. The database has on-disk and in memory caching, but eventually you’ll run out of memory on a single host if your working set exceeds the host’s memory.
Memcached also covers up replication lag (MySQL is terrible at replication, Oracle not so much) in large environments by putting data into the distributed cache (Write-through caching) before the slave database has finished it’s writing. Data is available immediately to clients, before the replication has completed.
It will also provide a large amount of savings when you’re constantly executing that O(n x m) query to find out who is friends with whom on your social networking site.
This comes with a cost, though. Relational database functions, like joining across large data sets, and atomic operations, become very difficult to execute. Memcached becomes the central server, and there is always a fear that an important key will drop out of cache because of a random eviction.
It’s not without risk, either. Dependence on the cache can hurt you severely if lots of memcached servers fail (and they do fail), Leaving you in a ‘cold cache’ situation where it can take hours to repopulate your working set back into the cache pool.
Don’t question MySQL’s performance — relational databases are great, but they are not the only solution to storage problems. the two problems that are being solved here are, highly orthogonal.
I’d also like to state that the majority of alternate key-value store databases listed in Richard Jones’ article and in Lenoard Lin’s blog are really not ready for high production loads (with maybe the exception of Tokyo Cabinet, HDFS, and Cassandra). There is still a ton of ’secret sauce’ the large sites are keeping quiet about in order to make these into effective data stores.
Lin states this in his review as well: “Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.”
I’m announcing the release of mod_memcache_block, a distributed IP blocking system for Apache, with rate limiting based on HTTP request code.
For many years I’ve had a need for a module like this — A distributed blocking system which could operate across large web serving clusters and register hits in a central store. With rate limiting, incrementing counters on a single host is fairly useless when you have hundreds of servers behind a load balancer.
An attacker could hit many machines within the limit period before being detected, because there would be no central count. By keeping the counts in a memcache pool, all servers share the same data.
It won’t defend against attacks coming from random proxy addresses (say, Tor), and might unfairly count hundreds of users who live behind a single proxy (like corporate NAT), but it offers some protection against attacks coming from a single source IP.
The software is released under the Apache 2.0 Open Source License.
From the docs:
mod_memcache_block is an Apache module that allows you to block access to your servers using a block list stored in memcache. It also offers distributed rate limiting based on HTTP response code.
FEATURES
Distributed White and Black listing of IPs, ranges, and CIDR blocks
Configurable timeouts, memcache server listings
Support for continuous hasing using libmemcached’s Ketama
Windowded Rate limiting based on Response code (to block brute-force dictionary attacks against .htpasswd, for example)
REQUIREMENTS
libmemcached-0.25 or better
Memcached server
Apache 2.x (tested with 2.2.11)
There’s a small interview with me in today’s O’Reilly radar, where I talk about some of the things that I’ll be presenting as part of my Velocity 2009 talk. You can listen to, and read the transcript here:
A bit of last-minute news, but I’ll be on a panel at SXSW Interactive: “Using GPS & Location to Enhance Social Networking”.
First there were social networks, and then there were location-based social networks, and now GPS and navigation-enhanced mobile social networks. This panel will explore how these emerging platforms integrate with existing social networks (facebook, twitter, etc), leverage GPS navigation functionality, and take location-aware social networking to the next level.
It’s at 5pm on Tuesday, the 17th of March.
More details at available on the panel’s schedule page.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=105562ad-411c-40df-be86-3f4317c85bd6)
