It’s a very small, 50MB distribution of linux that is not that easy to install if you don’t have a linux box to start with. Despite the large number of Linux boxes that I once owned, I’ve replaced all of them with Mac OS. Fortunately, there is a simple way to install DSL on a USB boot drive from Mac OS, using VMWare Fusion.

You’ll need a 1GB USB drive. The smaller the drive is, the better. Some systems cannot boot off of very large USB drives. 512MB or 1GB is recommended.

Here’s the how to…

Download the latest copy of DSL from a mirror.

Do not use syslinux, but get the ISOLINUX version. This is the ISO named dsl-x.x.x where x.x.x is a version number. I used dsl-4.4.10.iso, although in retrospect I should have used the VMWare VMX Image. Either works.

Insert the pendrive and format it.

Mac OS will auto mount the pendrive. Open Disk Utiilty.
Format the device as MS DOS FAT and MBR.
One Partition.

Important: Unmount the pendrive before starting VMWare Fusion.

Create a new VMWare Fusion Virtual Machine

The general idea here is to create an empty VM using fusion, and to boot off of the ISO you have just downloaded.
It can be reasonably small. You do not need to allocate disk space on your HD.
Configure the VM as Linux/Ubuntu and to mount the CD.

Mount the pendrive as an “Alcor Micro Mass Storage” device. It will appear to your VM as /dev/sda.

Install DSL

When DSL boots up, at the “boot:” prompt, type “install”
Then, select “5″ for “USB Pendrive HDB boot install”

Answer yes to all prompts, and the install will (hopefully) complete.

Now What?

You’re done. DSL is great for password recovery, emergency repair, or system rescue. So long as your system supports booting from USB drives, you’re good to go!

The workloads of social networking sites fall mostly into the ‘read lots, write once’ class (most of the web exists within this paradigm.) Regardless of the database company that’s responsible for the software, the main idea in scaling this read heavy workload is to remove the burden from the database and move it to distributed memory stores.

As an engineer, you want applications to pull from the same cache pool to reduce I/O pressure. To ensure that every machine isn’t replicating data in individual caches, you have to go distributed. That’s the win with memcached.

Putting a distributed cache between the application and the database increases performance and shares data across your application servers, something that the database cannot do on it’s own. The database has on-disk and in memory caching, but eventually you’ll run out of memory on a single host if your working set exceeds the host’s memory.

Memcached also covers up replication lag (MySQL is terrible at replication, Oracle not so much) in large environments by putting data into the distributed cache (Write-through caching) before the slave database has finished it’s writing. Data is available immediately to clients, before the replication has completed.

It will also provide a large amount of savings when you’re constantly executing that O(n x m) query to find out who is friends with whom on your social networking site.

This comes with a cost, though. Relational database functions, like joining across large data sets, and atomic operations, become very difficult to execute. Memcached becomes the central server, and there is always a fear that an important key will drop out of cache because of a random eviction.

It’s not without risk, either. Dependence on the cache can hurt you severely if lots of memcached servers fail (and they do fail), Leaving you in a ‘cold cache’ situation where it can take hours to repopulate your working set back into the cache pool.

Don’t question MySQL’s performance — relational databases are great, but they are not the only solution to storage problems. the two problems that are being solved here are, highly orthogonal.

I’d also like to state that the majority of alternate key-value store databases listed in Richard Jones’ article and in Lenoard Lin’s blog are really not ready for high production loads (with maybe the exception of Tokyo Cabinet, HDFS, and Cassandra). There is still a ton of ‘secret sauce’ the large sites are keeping quiet about in order to make these into effective data stores.

Lin states this in his review as well: “Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.”

Tread lightly.

How do we find the user in question so we can thump them on the head to make them stop?

This is a basic exercise in information gathering. For the most part, we’ll have the user’s IP address, and we’re a mac shop with many users running iTunes. If the user is sharing their library, you can use iTunes as a covert means of determining a user’s name, as iTunes will use the local computer’s name as the library name.

Telnet to the machines DAAP port, and issue:

John-adamss-macbook-pro:~ jna$ telnet x.x.x.x 3689
Trying x.x.x.x...
Connected to x.x.x.x.
Escape character is '^]'.
GET /server-info HTTP/1.1
Host: x.x.x.x
Client-DAAP-Version: 3.7
User-Agent: iTunes/8.0.2 (Macintosh; N; Intel)
Accept-Language: en-us, en;q=0.50

HTTP/1.1 200 OK
Date: Tue, 13 Jan 2009 21:26:38 GMT
DAAP-Server: iTunes/8.0.2 (Mac OS X)
Content-Type: application/x-dmap-tagged
Content-Length: 280

msrvmstt?mproaproaeSVaeFPatedmsedmsmlmsmOk?[minmUSER NAME’s LibrarymslrmstmsalmsasmsupmspimsexmsbrmsqymsixmsrsmsdcmstcImmsto???

Other options for this include attempting to sign on to the server with Apple-K if AFP on TCP port 548 is active (which will reveal the computer’s name) and using nmap with service detection to glean information about the host.

In my previous Apple Macbook Pro to Local network host (Mac Mini) testing, my top connection speed was around 2.4 Mbps. After the upgrade, it’s between 4.65Mbps and 7.5Mbps. Nothing near the promised speeds of 802.11N (300Mbit/sec), but I suspect that this is because of an incompatibility between Apple’s hardware and Netgear’s Hardware.

------------------------------------------------------------
Client connecting to 10.1.1.15, TCP port 5001
TCP window size:   129 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.70 port 51617 connected with 10.1.1.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.59 MBytes  7.19 Mbits/sec

Update:

After disassociating and reassociating with the AP, speeds went way up:

retina:~ jna$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.1.1.15 port 5001 connected with 10.1.1.70 port 52865
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  34.8 MBytes  29.1 Mbits/sec

With other devices on the WLAN, speeds go down. My current theory is that 802.11g devices on the same wireless network (such as the older Macbooks that we have) drag 802.11n speeds down, but I’m yet to be able to prove that.

There’s two things. First, on outgoing mail, we convert the address to a special VERP address. An email address of “user@example.com” becomes “twitter-welcome-user=example.com@postmaster.twitter.com”. We usually replace “welcome” with something topical so we can track the origin of the message.

Why VERP?

If the mail bounces, the mailer-daemon on the other end will bounce the message back to the postmaster machine, which is a transport method in our inbound Postfix server pool. If an MTA along the transit path destroys the original destination email address, we can recover it from the To: address on the bounce. DJB first devised this for use with qMail, but it works great here.

That server runs a Ruby (originally written in perl) script, that pushes data to an internal API that increments the email address with a bounce score . Get too many points in the bounce score table, and we disable sending to your account. If you change your email address in our database, or ask us to try again, we clear the bounce score. We also clear the bounce score after 30 days or by admin request.

We identify and score bounced emails through the use of a simple regexp based scoring mechanism and/or use the DSN if a Delivery Status Notice is available. A “5″ indicates “hard fail, “2″ = soft fail, and “1″ = we don’t know. Anyone scoring over five is disabled. If it’s the “welcome to twitter” message, we mark it as 10 immediately. Your first mail has to go through, period.

Disabling is accomplished through a reverse blacklist through Postfix’s smtp_recipient_restrictions configuration directive and MySQL via proxymap.

The directives to do this under Postfix 2.5 are pretty simple.

In main.cf, On the central outgoing server(or servers…), Add:

# bounce handling
bouncehandler_destination_recipient_limit = 1
transport_maps = btree:/etc/postfix/transport

smtpd_delay_reject = yes
smtpd_recipient_restrictions = \
  check_recipient_access mysql:/etc/postfix/reject-bouncing.cf, \
        permit_mynetworks, \
        reject_unauth_destination

/etc/postfix/reject-bouncing.cf:

user = your_db_username
password = your_db_password
dbname = your_db_name
hosts = your_db_host
query = SELECT 'DISCARD in_bouncers_table' FROM bouncers WHERE email='%s' AND score >= 5

In master.cf, on the incoming server, add:

bouncehandler  unix  -       n       n       -       -       pipe
  flags=DRhu user=nobody argv=/etc/postfix/bouncehandler.pl

I use the following function to calculate severity of a bounce. It’s not perfect:

sub get_severity {
    my ($es) = @_;
    my $body = $es->body;
    my $subject = $es->header("Subject");
    my $score = 1;

    # DSN Failures
    if ($body =~ /Action: failed/) {
	$score = 5;
    }

    # check through body of message and try to score the bounce.
    # temp failures.
    if ($body =~ /connection refused|host unreachable|host or domain not found/mi) {
	$score = 5;
    } 

    if ($body =~ /quota|mail(box|folder)* is full/mi) {
	# mailbox full, not as bad, we'll try up to 4 times.
	$score = 2;
    } 

    # wierd temporary failures
    if ($body =~ /(junk mail|spam) try again later/mi) {
	$score = 1;
    }

    # temp.
    return $score;
}

I leave it to you to write bouncehandler.pl, including the above code, and the database schema. It’s very simple. Parse the message, create rules, insert/update a record in your database. I used Email::Simple for parsing, and DBI/Mysql for database access.

The DB Schema needs to contain these columns (at a minimum): The email address, a score column, created_at, and updated_at.

My queries look something like this:

# prepare SQL statements for later use
# last times are based on our receipt, not theirs.
$findemail_sth = $dbh->prepare('SELECT id, score FROM bouncers WHERE email=?');
$insert_sth    = $dbh->prepare('INSERT INTO bouncers (email,score,updated_at,user_id) VALUES (?,?,now(),?)');
$update_sth    = $dbh->prepare('UPDATE bouncers SET score=?, updated_at=now(),user_id=? where id=?');

I turned to CAIDA, the Cooperative Association for Internet Data Analysis, which has long been a provider of excellent network performance tools. Their research focuses on developing tools to measure the Internet in many amazing ways, such as this map of interconnections between the all Autonomous Systems (AS) of the Internet. 

There’s many tools available, but the unofficial standard for bandwidth measurement is iperf. It’s a simple tool to show the maximum possible bandwidth between two points. One machine runs a server, using ‘iperf -s’. The client connects to the server (using iperf -c server) and as much data as can be sent in a single interval is sent. 

Between two of my machines running OS X 10.5.5, I get great results:

retina:/tmp jna$ ./iperf -c hackintosh -i 1
------------------------------------------------------------
Client connecting to hackintosh, TCP port 5001
TCP window size:   129 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.15 port 52150 connected with 10.1.1.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    107 MBytes    894 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec    109 MBytes    912 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec    107 MBytes    901 Mbits/sec

The situation is not so wonderful between my laptop, An Intel Macbook Pro, on 802.11N wireless via a Netgear WNR3500.

dhcp-102:iperf-2.0.4 jna$ src/iperf -i 1 -c hackintosh
------------------------------------------------------------
Client connecting to hackintosh, TCP port 5001
TCP window size:   129 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.102 port 49518 connected with 10.1.1.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  2.45 MBytes  20.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  1.0- 2.0 sec  2.59 MBytes  21.8 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 3.0 sec  2.36 MBytes  19.8 Mbits/sec

I don’t believe wireless sales materials anymore (when did I ever?) regarding the maximum speed of these devices. I have five bars on Apple’s wireless icon here, the Macbook Pro supports 802.11N, and I can’t get more than 21.8 Mbits/sec to local machines here. On my Comcast cable, my maximum download speed is around 13 Mbits/sec, so I probably have the best speed possible for downloading from the Internet, but moving files across the local LAN via wireless is a different story. It’s much slower and I’ll go directly to the gigabit, hard wired connection for movies and music.

Interestingly enough, 21.8 Mbits/s is well within 802.11g’s allocation of 25 Mbits/s per client. My 802.11N configuration is no better than 802.11g, even though Apple’s Network Utility reports a link speed of 130Mbits/second and that I have the 802.11 a/b/g/n Network adapter installed.

I’m unsure as to why the network configuration disagrees with the achieved bandwidth, though. It’ll be something I research in the next few days.

I’d also like to take a moment here and condemn Belkin’s entire line of wireless devices. They use a piece of Javascript with breaks the RFC standards in so many ways. If you attempt to configure these devices using CIDR, such as “10.1.1.0/24″, which is what my home network is, their devices force you to a netmask of “255.0.0.0″ because the javascript in the setup form sees 10.0.0.0/8 as a class A network. Classless notation in network allocation is the standard these days, and Class A, B, and C notation is a thing of the past.

This broke my network for hours until I was frustrated enough to bring the device back to Best Buy and purchase the WGR3500. At least I didn’t have to deal with an online return!

From cabling issues to configuration problems, here’s some tips so that you never have to go through the troubles that I did.

1. Adventures in Layer one:  Don’t use home made cables!

If you must make them, ensure that you are crimping the ends of the cables to the same spec and that all four pairs of the cable are working correctly. I had a run coming from the studio, far away from where the main router is with a bad crimp. One dead pair caused the connection to fall back to 100mbit/s instead of 1000mbit/s, eliminating any advantage I’d gain from the new hardware.

Always purchase good quality, store bought, category five cables wired to the TIA 568A or B spec.

2. Always check the lower levels of the network before blaming higher ones.

In order of verification, your checks should go:

1. Layer one – Physical. Check your wiring.
2. Layer two – Ethernet. Make sure you have link, and that both sides of the link are at 1000 mbit
3. Layer four – IP: Make sure you have the right IP addresses in your network, and netmask. Ping hosts on your local network first, then try the router, then try internet based hosts. If you can get to the local network and not the Internet, check your default route.
4. DNS. Can you look up a hostname? Does it return the right address immediately without lag? If not check DNS settings and try again.

Once you’re sure the foundation of the network is fully functional, move on to higher level apps, like your browser.

It’s the only piece of software I know of that has ‘–thpppt’ as an option (Install it, and you’ll see.) It’s an optimized replacement for many of your favorite awk | grep | sed combinations, and includes syntax highlighting.

In other news, I’ve resolved a number of bugs and issued new code for running mod_telemetry on 64-bit Linux. Check out the SVN trunk for the latest branch. The data that this module has been providing to me has been invaluable for researching slow points in the back end.

The CERT advisory is here.

A partial overview, from the PDF released by Secuonis…

On July 8th, technology vendors from across the industry will simultaneously release  patches for their products to close a major vulnerability in the underpinnings of the Internet. While most home users will be automatically updated, it’s important for all businesses to immediately update their networks. This is the largest synchronized security update in the history of the Internet, and is the result of hard work and dedication across dozens of organizations. 

Earlier this year, professional security research Dan Kaminsky discovered a major issue in how Internet addresses are managed (Domain Name System, or DNS). This issue was in the design of DNS and not limited to any single product. DNS is used by every computer on the Internet to know where to find other computers. Using this issue, an attacker could easily take over portions of the Internet and redirect users to arbitrary, and malicious, locations. For example, an attacker could target an Internet Service Provider (ISP), replacing the entire web — all search engines, social networks, banks, and other sites — with their own malicious content. Against corporate environments, an attacker could disrupt or monitor operations by rerouting network traffic traffic, capturing emails and other sensitive business data. 

Exact details on this are being withheld for the safety of the Internet; I prefer full disclosure, but that doesn’t seem to be the case here given that the hole is so large and vulnerability so widespread.