Thoughts.

Brewed in the hoppy city of San Francisco

0 notes &

A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.
John Gall

0 notes &

Quick hack to deploy to EC2.

Some people argue that you can do everything in python.

As much as I love python, I’d argue that for each task you try to accomplish, there is always a tool better than another, and sometimes, python is not that one. 

14 notes &

Why shit breaks.

So yesterday was my one year anniversary at Formspring.

We built a lot of stuff in the past year and it’s been a hell of a ride.
 
One of the first features I built was the ability to ask a question to all of your followers. We wanted to get the site ready to be used by high profile users. You know - the ones that get a lot of followers.
 
In order to understand this, picture your Facebook, you have maybe 400 or 500 friends on there, but the maximum number of friends you can get max-out at 5000.
Some of our accounts have over 1 Million Followers. That’s 200 times what you can get on Facebook. Yep that’s right.
 
So now picture that new feature. You can ask a question to 1 Million person. Have you ever sent an email to 1M person ?
Let’s say you have a Million friend (lucky you) where would you be storing this information ? Have you ever opened a spreadsheet with a Million rows ? It kinds of make your laptop look like it went from the latest build of Mac Os X to Windows 95.
Now imagine you have 27 Million spreadsheets (one per user of course), and all of them constantly add friends, remove them, that’s a lot of editing ! 
 
So obviously, all this data is stored on several machines, because
  • a) it doesn’t fit on one machine
  • b) you don’t want to lose that data so you have to replicate it on other machines.
 
Still following me ?
So we basically know that it takes a lot of machines to be able to store that data accurately, but now you need to actually process the message that you want to send to all the people you follow, and again, how does this work ? 
 
Well it’s being stored in another spreadsheet ! That’s right. It’s a lot of spreadsheets. With a lot of information.
 
Because every time one of these accounts ask a question to their 1 Million followers, we need to go through the spreadsheet with the information about their friends, and for each friend, open their spreadsheet and add a new question to it :) 
 
Wait - What - did you just add a line to 1M different spreadsheets ? 
 
Yep that’s right. And so forth and so on every time people ask questions on the site.
For your to get an idea, the order of magnitude of spreadsheets edited per minute is in the “tens of thousands”. 
 
In order to accomplish this, we have a lot of machines to host that information, process it, send emails etc. etc.
These machines must be running 24/7. You can never turn then off, because well - if you do, they stop doing their work and people do not get their messages in their inbox :) 
 
So again - for you to understand this, we’re not talking about a couple of machines processing that information, we’re talking about hundreds of them.
 
Now ask yourself this question - in the past year or so, has your laptop/computer ever act up ? You know like - slowing down, freezing, lost connection to internets. That kind of stuff.
 
To make this easy, let’s say we have 365 servers running at Formspring. On average a computer that runs 24/7 acts up once every two years.
The site has been up for over 2 years now. Which means that every single of these server actually failed at one point.
That’s a server acting up every 2 days.
 
We currently have 7 engineers on rotation to take care of the site 24/7. No matter what time of the day/night one of these server goes down, the designated engineer has to fix it with as little impact as possible for the users.
I can tell you that when you’re responsible for the whole infrastructure of a website with 27 Million users, and something breaks, you suddenly feel very tiny, and quite lonely.
 
I realize this is very very abstract (no we don’t actually use spreadsheet to store our data :) ), but I hope it gives people a good idea of what’s going on behind the scene. 
 
At that scale, the smallest changes can make big differences. but keep in mind that the more you grow, the more likely you are to go down.
 
Martin

Filed under Formspring Reliability Servers The Internets

0 notes &

intoxicated-veins-deactivated20 asked: Hey (: *follows*

hihihi lol lol im cracking up

0 notes &

youre-first-and-foremost-deacti asked: I got your blog through formspring...LMAOXD

hihihi lol lol im cracking up

0 notes &

Berlin at its best. 

Amazing acting and sick soundtracks by Paul Kalkbrenner

11 notes &

Random lookup algorithm #2

Thanks to @pims, I’ve changed the code from this morning to use generators instead of regular lists loaded in memory.

Here’s the output:

With lists:

 

With generators:

and the available RAM that I have on my machine right now :

I grabbed the snapshots right during run-time, and it takes the activity monitor about 2 seconds to update its info, which gave me enough time to capture the amount of memory used by the different scripts.

The source code for the generators is available here: https://gist.github.com/918579

Filed under python, generators lists algorithms