Foo Hack » Broken Isaac Schlueter on Web Development Tue, 03 Nov 2015 06:34:16 +0000 en How Not to Get Real (and die trying) Tue, 11 Nov 2008 22:36:17 +0000 Isaac ...Read More]]> Recently, Auston of the Dashboard Leadlog startup in Florida posted a somewhat snarky blog post about one of the responses to a job opening at their company. I more or less said the same things that appeared on Hacker News. Eventually my comment may show up, but it’s buried at #20, so I thought I’d post it here, as well.

Auston, this is for you.

While his tone was a little condescending, I completely see where this person was coming from. I have all of these skills, and if I was on the market, I’d skip right over your ad. I wouldn’t even consider it, and I probably wouldn’t be kind enough to tell you why. If anything, he wasn’t harsh enough, and you’d be wise to find the grain of truth in his criticism.

Money isn’t just important as a way to pay the bills. I could pay my bills on a lot less than I make today. Money is important because it shows how much an employer values me and my skills, which is also an indication of the work environment. Moreover, labor is a market. I’m not even going to consider an employer who can’t pay what the market will bear. Put another way, why would I want to work for anyone other than the employer who values me the most?

When you hire someone, however you’re compensating them, it’s more than they could get anywhere else. Think about that for a moment. If you get a Porche at a Honda price, don’t be surprised if the motor doesn’t live up to the salesman’s promises. You offered the highest price he could get for it, and it wasn’t a very good price. Maybe the other buyers know something you don’t.

Of course, salary isn’t the only kind of compensation. If they really believe in your company, they’ll push for equity, because they’ll believe that the equity is more valuable than the paycheck. If that’s the kind of belief you want, then you need to structure the compensation to be competitive to people who feel that way.

Your ad describes an expert frontend/backend guru-level ops/designer/architect (who also doesn’t mind slogging through maintenance tasks). And to compensate them, you’re offering…. a big monitor? Ping pong? JOKES!? Are you SERIOUS? Talking about a “comfortable and fun work environment” is a red flag. If you have to go out of your way to keep your engineers entertained and comfortable, that usually means that your product very boring and unpleasant. No thanks.

It sounds like you want a CTO or chief architect. If you can’t afford several hundred thousand per year in salary (which most startups can’t), you’d better be prepared to hand over a sizable chunk of your company to them. Skip the stuff about your killer break rooms, they probably have better chairs and toys at home. Sell them on the concept and the business plan instead. Anyone with those skills knows what they’re worth. You’re recruiting a partner, not an employee, but your ad clearly shows that you don’t get that.

If you balk at giving up so much, then ask yourself this: why else would someone with with that value choose to invest it in YOU, when they can do half the work and make 4 times as much at Google or Yahoo, or probably start their own company and compete with you directly? You’re hunting tigers, and frankly, it doesn’t sound like you’re up to the task.

If you don’t need that level of expertise, then don’t ask for it. When you want a Honda, and you can afford a Honda, you go to a Honda dealership. Job ads are not the place to shoot for stretch goals. This is the equivalent of the resumes that list every buzzword under the sun, when the person has real experience in maybe two or three.

We’re two biz dev guys, a ux guru and a hacker with solid business experience.

So, you’re 4 people, and only one of you writes code? Good luck with that. No wonder you need so much help. One of the biz dev guys should probably sell their equity to the other three, and go “dev” some other “biz”. Then maybe you’d be able to afford the CTO you need. It’s going to be a tough sell, though, given the childish attitude towards compensation.

Routers and Ethics Fri, 25 Jul 2008 17:35:46 +0000 Isaac ...Read More]]> I just moved into a new apt, and my neighbor has a wireless router that is completely open (no MAC whitelist, not even WEP or WPA). shows me the Netgear login screen, and the default Netgear password works. Curious and a little concerned, I logged in and poked around.

They have all the default settings, and the logs just show the same two computers connecting over and over again. Clearly, they bought it, plugged it in, it worked, and so they stopped caring.

We all know what the unethical course of action would be. You could easily route every bank’s IP address to a phishing site. Or, just install a packet sniffer on the router that will faithfully log any POST requests that they make, then look for anything looking like a password. If they use the same password twice, try that password everywhere. Once you have access to their email, game over.

If you just wanted to be annoying, you could block access to all the most popular websites for the 15 minutes when they always seem to connect, so that it’ll start working seemingly randomly just about the time that they’re calling the cable or DSL company in a huff. Or, expose their computer to the internet and just wait for various bits of malware to wriggle in.

Those are all of course completely evil. The real question is, do you protect them from their own ignorance? Do you leave their wifi network completely open, or lock it down? And, do you change the password?

I decided to leave their network open. If they want to share their internet connection with the world, far be it from me to tell them they can’t. That’s a lovely thing to do for the universe. And I don’t see a problem with using it once in a while until I get my own set up.

I did change their password. Clearly, that doesn’t matter to them. I know better, they don’t, I’d want someone to do that kind of thing for me. It’s a little presumptuous, but it also might keep them from being victimized by identity theft.

There’s an interesting lesson here. No one likes your control panel. Most users will use the defaults, always. These kinds of things are nice when you really want fine-grained control, but completely annoying, complicated, and tedious for most users. They just wanna check their email, play some scrabulous, download some porn, and go back to their normal non-technical lives.

We can of course blame the victim. How could they be so stupid to have their router open to the world. Don’t people think?!? But it would be very hypocritical for me to say that, and I don’t think I’m unusual.

  • I have never read the owner’s manual of a car I’ve driven.
  • I have only a few times read the full drug information on any medications I’ve been prescribed, and then only because I was really bored.
  • I plug it in. I turn it on. I only mess with it if it’s broken (or interesting.)

The difference is that, in any of these cases, the default value is not likely to be harmful. If you don’t change your oil at the right time, your car’s performance will suffer; but as long as you change it once in a while, it’ll keep moving. If a doctor is prescribing a medication, it’s probably not going to kill you, and if there’s any serious risk, they’ll usually tell you what to watch out for. Most electronic devices don’t have access to your bank accounts.

As software and hardware engineers, if our defaults put users in an unsafe situation, where their credit and savings are placed at risk, then we’ve failed them, and we’ve acted unethically.

Hacking the Google Favicon Sat, 07 Jun 2008 01:36:37 +0000 Isaac ...Read More]]> There’s been a lot of nerd rage about the new google favicon. I joined in with a rationalization for my distaste for their new design. After a few days of the lower-case “g” on a gray background, I decided today that I just couldn’t take it. Here’s how I fixed it, so my Mac shows the old logo.

Certainly, any company ought to think long and hard about any branding change so fundamental and wide-reaching as a change in logo. If you’re a web company, then your favicon IS your logo, and everything else is secondary. For Google’s sake, I sure hope they understand just how significant this change was, and weighed the pros and cons carefully.

While I recognize my position deep in peanut-gallery territory, I simply cannot accept that it was a good move. A favicon needs to be:

  1. Far from gray. Browser tabs are gray, and the purpose of a favicon is to let the faster right-brain processes identify your site in a list of tabs or bookmarks. If I can’t see it, it’s useless. FAIL
  2. Mostly not transparent. If you’re going to use a gray logo, fine. Put it on a white background, so you know it will stand out. A small logo in a transparent background is invisible. FAIL
  3. Evoke your corporate branding. I should be able to associate your favicon with your brand without thinking about it. If, for example, your corporate branding is focused on lots of bright colors on a white background, your favicon should echo this. FAIL
  4. Use big, simple shapes. Fine details (like those in the double-curl style of lower-case “g”) get lost at 16×16 size. FAIL

Poor contrast.  Not enough pixels.  Not "google" enough.  Prettier - Who cares?  FAIL

Yahoo and Google are effectively equivalent in terms of the objective quality of their search results. Scores of user-studies have shown that our perception in SERP quality is due primarily to brand loyalty and nothing else. If you’re a Yahoo searcher, and you see a red Y, your comfort filter tells you it’s good, no matter where the results actually came from. You can’t tell the difference, because, for the most part, there isn’t one. Most of the time, Yahoo and Google search pull up exactly the same sites, or close to it. They’re both very very good, and search is a case where there is a right answer, even if the method of finding it is a bit fuzzy.

For some searches, it seems that Yahoo is even significantly better than Google. Compare for yourself. Somewhat humorously, the results for “yahoo is better than google” seems to yield much better results on Yahoo. I tried the reverse search, expecting to see some kind of amusing self-bias, but actually Yahoo did that better, too. From my cursory non-scientific view, it seems like Yahoo puts a bit more weight on the title, and there are probably other slight differences. But the point is, the differences are slight enough to be overlooked. They’re both pretty damn good at finding what you ask for.

I work at Yahoo. But I use Google for search. I perhaps ought to bleed a little more purple than I do. I also own stock in Yahoo, so in a small way, using Google for search is not a financially rational move. So why do I do it?

Brand loyalty. I was using Google ever since it’s been around, and I’ll probably keep using it for a long time. I imprinted early on that big G in the white square, and some very low-level neural systems tell me loud and proud that, when it comes to searching, that G means Good. Frankly, rewiring all that stuff isn’t worth the effort.

I think that brand loyalty explains a lot of the nerd rage over the new favicon. Yes, it doesn’t affect the results. Yes, it’s just a case of cheese moving, and not even a very relevant one. And, face it, you’re not going to stop using Google over it if you’re already a Google user. But you may be consistently annoyed for several months while the brain circuits rewire themselves to imprint on the new design.

Unacceptable. I’m a web geek. I’m a hacker. There is no way in hell that I should have to put up with something I find even slightly distasteful in my regular internet experience. I refuse to be subjected to some marketing exec’s idea of what is best for me, when I clearly know my needs better. I don’t even care if he went to Stanford AND coded a rocket ship in Python.


If you use any Google services that require a login (such as GMail, Groups, Orkut, or pretty much anything other than Search), then you’ll need Apache 2. I tried really hard to make it work with Apache 1.3, but no dice.

The good news is, Apache 2 comes standard with Leopard. The bad news is, it doesn’t come standard with Tiger.

Since I was using Tiger, and Apache 1.3, this was a much more involved hack than I’d anticipated going in. Luckily, it’s just a matter of change one line in the /etc/hosts file to turn it off, which is good, because I had other things to do before I could fix it. I use MacPorts for a bunch of stuff, so I figured I’d just install Apache 2 that way.

Easier said than done, at least in my case.

I already had PHP and MySQL running a local copy of this site, and some other stuff. MacPorts really shines if you use it for everything—try to run Apache on MacPorts and run PHP and MySQL from some other location, and you’re in for a world of hurt. It took a lot of SFTW and learning about reconfiguring to get it all working properly.

I’m not the best guy in the world to critique package management systems, but I’ve got some end-user experience. I use apt-get on my web server, MacPorts on my laptop, and of course, yinst on all the Yahoo! machines I work on. Of course, that’s not counting the Registry and Add/Remove programs dialog from Windows Land (which is hell), and the much more user-friendly “Applications” folder on the Mac (which is lovely, but not really a full-featured package manager). Of these, yinst is the best, hands down. Powerful, simple, effective, and sadly, 100% closed-source and proprietary. If you’re not at Yahoo, you’re not using yinst.

But enough about installing stuff. Long story short, you need Apache 2, or don’t go any further. If you primarily use Firefox, you can just follow Paul’s advice, and install this Greasemonkey script.

I still recommend doing this, especially if you write code for the web, and you’re not normally a “back end guy”. It’s good to tinker with this stuff, and even break it once in a while in your own little sandbox. Unless you tinker, you never learn anything new. Newton did revolutionary work in optics, largely motivated by his need for a better telescope to look at the planets. It’s good to have a working knowledge of the tools of your trade, even if they’re just a means to an end.

So, fixing the G favicon…

mod_proxy to the rescue!

The premise of this hack is pretty simple:

  1. Edit the /etc/hosts file to point “” to your local machine.
  2. Use mod_proxy to transparently send all requests for “” on to Google’s actual IP address, except for /favicon.ico
  3. Point /favicon.ico at the old favicon file, sitting on your local machine.

The interesting thing is that you can do this for any file or url out there that you want to swap in your local machine. Don’t like the favicon that a given site uses? Change it! You have the power! That’s the beauty of the web.


For this hack, you’ll need:

Apache 2 with mod_proxy, mod_ssl, and mod_rewrite

If you have a Mac with Leopard and the developer tools stuff on it, you probably already have this, and might just need to enable them. I’ll tell you how.

If you don’t already have Apache 2 and/or mod_proxy installed, then that’s outside the scope of this post, like I said above. Sorry.

About 15 30 minutes

Yes, it’s really that easy. It’s fairly easy, but a bit trickier than I’d originally thought.

I use a Mac, and these instructions are fairly mac-centric. If you use a Linux or Unix machine, then most of it will still make sense. If you use a Windows machine, then a lot of it won’t apply, but the basic idea should still be possible if you use Apache and install mod_proxy.

Some Apache setups install the Apache Control script as apache2ctl, and others call it apachectl. Since I used MacPorts, I’ve got apachectl on my machine. Substitute as required for your setup.

If you run into any sticky points you can’t get out of, post a comment. I’d like to hear it. If you run into any sticky points and DO get out of it, then definitely post a comment! I’ll update this post with your info.

Get the old favicon

I couldn’t find the actual ico file, but I found some screenshots on the internet talking about the favicon change. I got a pretty good png, and lifted out just the 16×16 piece from the address bar in the screenshot.

Then, I used png2ico to convert that png to an ico file. It’s a great tool that I use all the time to make favicons. It lets you embed multiple pngs into the ico, but in this case, all that I really care about is the 16×16 that shows up in my address bar and on the tabs in Firefox and Safari.

All in all, that took probably less than 10 minutes. But you can do this step even faster by just downloading the Old Google Favicon. Extract it in Terminal or iTerm by doing this:

tar -xzvf old-google-favicon.tgz

You’ll get a file called old-google-favicon.ico. Put that under the “Sites” folder in your home directory. If you just un-tarred it using the command above, then it would be:

mv old-google-favicon.ico ~/Sites/

Turn on Apache

If you already run web pages and whatnot on your local machine, then you are already set. Skip to the next step.

To see if Apache is running, fire up a web browser and go to localhost. If you see the default Apache startup screen, then you’re all set. If it redirects to or a web search, or you get a “could not connect to server” message, then you need to enable Apache.

If you’re running Leopard, open up System Preferences, and click on Sharing. Click the little lock in the lower-left corner, and enter your password. Check “Personal Web Sharing”.

Try hitting localhost again. If it still doesn’t work, then do this in a terminal:

sudo apachectl graceful

If that doesn’t work, try:

sudo apache2ctl graceful

If it’s still not working, sorry, don’t know what to tell ya. Good luck.


At this point, you should be able to see the desired favicon at http://localhost/~YOUR_USER_NAME/old-google-favicon.ico, where YOUR_USER_NAME is the username that you log in with. (If you’re not sure what that is, open up a terminal, and type whoami, and it’ll tell you.)

If you’ve configured Apache to have some different document root, then adjust the path accordingly. For example, since my Mac is effectively a single-user machine, I just have the default document root pointed at /Users/isaacs/Sites/, so the favicon is at http://localhost/old-google-favicon.ico. Your setup may vary.

Ultimately, it doesn’t much matter where you put it, since we’ll be telling Apache exactly where to find it later.

Enable Apache Modules

Fire up your favorite text editor, and open your httpd.conf. By default, this file is in /etc/httpd/ or /etc/apache2/; if you installed with MacPorts, then it’s in /opt/local/etc/apache2/. (You’ll have to save it as the super-user, since it’s a root-owned file, so that means vi and emacs users will have to open it with sudo. GUI editors will usually prompt for your password when you try to save the file.)

Each of these lines have to be in there somewhere. If they’re missing, add them. If they’re commented-out, un-comment them.

LoadModule proxy_module modules/
LoadModule proxy_connect_module modules/
LoadModule proxy_http_module modules/
LoadModule ssl_module modules/
LoadModule proxy_balancer_module modules/
LoadModule rewrite_module modules/

If you notice that these rules look “out of place” for the stuff that’s already there, then modify to match. For example, if the other rules are all loading the .so files out of “libexec”, then change “modules” to “libexec”.

Restart apache by running this command in a terminal:

sudo apachectl graceful

If it tells you that the server couldn’t be restarted, try this to get a more helpful error message:

sudo apachectl configtest

If it’s not working, sorry, don’t know what to tell ya. You may need to install some of the required modules using the apxs utility.


Because we’ll be proxying an SSL (ie, https://…) connection, your server will have to be able to do SSL so that it can maintain the level of encryption.

I followed these instructions to create SSL keys. It directs you towards a script, and finding that was a bit tricky. For your convenience, I’ve provided it here.

Also, I put the ssl.key folder in the same folder as the other Apache stuff, the location of the ServerRoot directive. That way, I can refer to it by a relative path. If you prefer to have it in a separate location, that’s fine, but you’ll need to refer to it by the complete path in the configs below.

Get Google’s IP Addresses

In a terminal, run this:


You’re looking for the “Address” lines, not the “Server” line. In my bash profile, I have a handy shortcut for this, so I just type getip to see the results from both nslookup and ping.

Make a note of those IP addresses. As of this writing, from my location, I got these:

We’re actually going to define a load balancer to distribute your requests between these three. While that’s probably overkill, part of the benefit of this is educational, and that’s what a proxy would normally do.

Host File

Open up your /etc/hosts file in a text editor, and add this line:

Then, in a terminal, run this command to refresh your IP caches:

lookupd -flushcache

Bring up in a web browser, and you should be looking at that default Apache page again. If so, you’re on track. If not, figure out what went wrong. You can see if Apache is in trouble by running this command and then hitting the url again:

tail -f /var/log/apache2/error_log

MacPorts users: tail -f /opt/local/apache2/logs/error_log

You may need to comment that line out with a # and do the lookupd -flushcache if you want to search google while doing this. Or you could just use Yahoo! Search temporarily. Maybe in the process, you’ll get used to Yahoo search and even find that you like it better. (See? I’m a good little purple-blooded corporate monkey!)

Set up mod_proxy

So, now let’s take your new and un-improved, and turn it into a proxy for the actual

If you have a separate conf file where you keep your Apache customizations (which I highly recommend), then add this stuff in there. If you don’t, then you can just add it to /etc/httpd/httpd.conf or /etc/apache2/httpd.conf or /opt/local/apache2/conf/httpd.conf. Either way should work, as long as Apache finds out about it somehow.

Stuff that you may need to change wrapped in strong tags.

#google proxy stuff
Listen 443
SSLPassPhraseDialog builtin
SSLSessionCache dbm:/var/run/ssl_scache
SSLMutex file:/var/run/ssl_mutex
SSLRandomSeed startup builtin
<VirtualHost *:80>
# the place where you put the .ico file.
DocumentRoot /Users/isaacs/Sites/
ProxyPreserveHost On
ProxyRequests Off
<LocationMatch .*>
Order deny,allow
Allow from
SSLProtocol all -SSLv2
SSLSessionCacheTimeout 300
AllowCONNECT 443 563 80
SSLEngine on
# enable SSLv3 but not SSLv2
SSLProtocol all -SSLv2
SSLCertificateFile ssl.key/server.crt
SSLCertificateKeyFile ssl.key/server.key
# correction for browsers that don’t always handle SSL connections well
# You don’t really use MSIE on your own machine, though, do you??
SetEnvIf User-Agent “.*MSIE.*” \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
SSLProxyEngine on
<Proxy balancer://googlecluster>
# the IP addresses you fetched before.
# Browsing History seemed broken unless it proxies back to https.
# This is probably safer anyhow, for privacy.
ProxyPass /history
ProxyPassReverse /history
RewriteEngine On
RewriteRule /favicon.ico /old-google-favicon.ico [L]
ProxyPass / balancer://googlecluster/
<VirtualHost *:443>
# the place where you put the .ico file.
DocumentRoot /Users/isaacs/Sites/
ProxyPreserveHost On
ProxyRequests Off
<LocationMatch .*>
Order deny,allow
Allow from
<Proxy balancer://googlecluster>
# the IP addresses you fetched before.
# note the “https” on these ones.
SSLProtocol all -SSLv2
SSLSessionCacheTimeout 300
AllowCONNECT 443 563 80
SSLEngine on
# enable SSLv3 but not SSLv2
SSLProtocol all -SSLv2
SSLCertificateFile ssl.key/server.crt
SSLCertificateKeyFile ssl.key/server.key
# correction for browsers that don’t always handle SSL connections well
# You don’t really use MSIE on your own machine, though, do you??
SetEnvIf User-Agent “.*MSIE.*” \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
SSLProxyEngine on
RewriteEngine On
RewriteRule /favicon.ico /old-google-favicon.ico [L]
ProxyPass / balancer://googlecluster/

Save the file, and then restart Apache:

sudo apachectl graceful

Again, if it doesn’t work, try this to figure out what’s wrong:

sudo apachectl configtest


Now, go to You should see the google home page, complete with the old lovable [G] favicon. If you see google, but not the old favicon, go directly to the favicon.ico and refresh a whole bunch. Clear your cache, etc.

If you get the new favicon, try running some tests. Search, go to your account. Make sure everything works.

When you hit an https address, you’ll get a prompt that the certificate is issued by an untrusted authority. This is good! Check the cert details, and make sure that they’re what you typed when you created the certificate. Now, the question, do you trust yourself? (I’d already added myself as a trusted authority, so didn’t get this problem in Firefox, but I did see it in Safari.)

If it doesn’t show you the new favicon, then you might have put the virtual host config someplace where it doesn’t belong, or where it won’t be picked up. Tail the error_log file and see if Apache tells you anything is wrong. Note that some Apache configurations use “apache.conf” instead of “httpd.conf”, so maybe try that.

If it doesn’t load at all, tail the apache logs again, and see if anything is barfing.


You may now resume your google usage as normal, before the disruptive new favicon appeared.

Note: The Google Notifier won’t work, because it checks the certificate, and doesn’t give you the option to trust another authority, which is, in my opinion, very bad design. Also, the Google Notifier sends your password and username out in clear text by default, so it’s very insecure. You shouldn’t be using it in the first place.

Instead, you can use the RSS feed to get inbox alerts without exposing your info.

Going Fast, Frankenstein, and Refactoring Mon, 21 Jan 2008 17:00:48 +0000 Isaac already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs. That's right. We haven't released anything, and there's already enough bloat to justify spending at least 2 weeks cleaning it up. ...Read More]]> Very soon now, the project that has consumed my days for the last 6 months will go into public beta. It’s too early to pop the champagne, and that’s not what this post is about. However, the newness of this project is worth pointing out for the following reason: We’re already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs.

That’s right. We haven’t released anything, and there’s already enough bloat to justify spending at least 2 weeks cleaning it up.

Of course, due to the pressures of market, and the morale effects of delaying the release, we’re buttoning up what we have now as much as possible, and planning to work on the refactor once the beta is out there. It may actually help a lot, since the gigantic firehose of traffic that Yahoo! points at sites has a way of helping even the most egotistical of developers realize that their clever baby needs some serious surgery.

This post is going somewhere, I swear, but I can already tell it’s going to take me a while to get there. Go get some coffee and a snack and come back. I’ll wait.

Build Fast

In my opinion, the core of good software development methodology comes down to two basic questions that should be asked daily:

  1. What portion of your time is spent focused on the task of creating software?
    (That includes planning, discussion, and thinking as well as coding, provided that these non-coding activities are focused and aimed at creating software.)
  2. Does it work today?
    (If you don’t know, then you are full of Fail. Get on that right away.)

I call this the Go Fast method. The goal is to efficiently create software with minimum long-term cost and maximum long-term benefit, without driving away your best talent. (Have I got the attention of the business and product people in the audience? Good. Please stay for the rest of the show.)

I know that someone out there is bound to point out the obvious parallels between my “go fast” methodology and Agile, and maybe they’re right. To the extent that Agile accepts these premises as important, it is correct. But I’m claiming that everything else stands or falls based on how it affects these questions. Stand-ups, planning meetings, org charts, IDEs, frameworks; they’re all only as good as their effect on the principles of speed and stability in development.

If you’re developing software properly, you should be creating software that does something as soon as possible, and keep it up as much of the time as you can. Until there’s something to look at, you’re just spinning your wheels. Nowhere is this more true than on the web. Photoshop comps don’t break in different browsers. Product presentation decks don’t “feel klunky” when pieces load out of order. Grand visions promise to solve all your problems and make you biscuits for breakfast, but they don’t deliver.

You make software for a living? So make software, already. You wanna build a website? What’s stopping you? You don’t have designs yet? Build the parts that you do have. Always be releasing, even if your actual release date is months away.

Build the core, and grow the other pieces up around it. Every minute spent waiting, instead of actually building something, is worse than wasted. When you live in the code day by day, it becomes a part of you. You learn how the pieces work, and the knowledge is automatic. When you get too far from it, it becomes foreign, like driving a car for the first time.

The agilistas call this “shippable code.” In my opinion, that’s a horrible term. The vast majority of code that I’ve seen released into the wild was not even close to what I’d call “shippable,” but it got shipped just the same. Most of the time, the prototypes leading up to the release were significantly worse. Some agilistas are quick to point out that “shippable” doesn’t always really mean “that you would actually ship it.” Sorry, but drivable means that you can drive it, and drinkable means that you can drink it, and flyable means that you can fly it, so if “shippable” doesn’t mean “you can ship it,” then they picked a bad term.

I suspect the ambiguity is intentional on some level. I think the claim that “Agile makes software faster” is a myth. Agile methodologies, used in moderation by a well-balanced team of dedicated and talented individuals, do tend to result in higher quality software, greater management visibility, and a happier bunch of people than any alternative, but that means that the overall development time can be significantly longer. And if you’re not on a well-balanced team of dedicated and talented individuals, then no methodology is going to save that sinking ship, so dust off the CV and bail out ASAP. But I digress, and besides, you already know that I think Agile sucks.

I prefer to call it a running prototype, or an integration environment, or anything else that’s more, well, accurate. “Shippable code” tells overeager managers that they can just press “stop” at any point and ship what they’ve got. Nice dream land, hot shot, but it doesn’t actually work quite like that. Nevertheless, a good integration development environment, where everyone’s code gets placed and which is subjected to regular updates and testing—that’s important, almost as important as CVS and a decent build script, and for a lot of the same reasons. If the team’s code isn’t conversing regularly, then the team may as well not be, either. The best way to prevent blockages is to prevent assumptions; the best way to prevent assumptions is to check them every single day. So: Does it work today?

Completion feels good. Getting something that works feels good. That’s why we got into this business. Because, despite all the pain and frustration and work that goes into developing software, The High here is about as good as it gets.

You don’t get that waiting. You don’t get that talking. You get that when you are immersed in building something.

From a team dynamic perspective, I can’t even find words to express how it makes people gel when we each do our little pieces, and plug them in, and see a site come up with data and styles and behavior and images and everything. It’s a pretty marvelous event that makes everyone even more eager to get to work. If you’re a web developer, don’t wait for your designer or back-end engineer to give you what you need. Make an unstyled page with dummy data, if you have to, but make something on day 1 if possible. Then make a list of what you need. When everyone can see their effect on the product, it’s amazing how fast they tend to deliver.

Going fast and pushing to milestones also forces everyone on the team to prioritize. There may be plenty of features that might be nice to implement, but if speed is a priority, then it forces you all to work on the things that really need to be implemented. By raising red flags whenever something doesn’t work, the whole project stays in sync most of the time, and assumptions never get too far out of whack. But, of course, that’s not the whole story.

Gene Wilder as Victor FrankensteinSoftware development is a lot like building Frankenstien’s monster. You start out with a pile of useless ugly pieces, and try to turn it into something beautiful. Along the way, it’s a monstrosity, and tends to get uglier as you tack new bits onto it. Then it kills people.

In Mary Shelly’s classic, everyone Victor loves is killed by the eloquent and sensitive creature, who was turned evil by his creator’s neglect and hatred. If software could walk and talk, how many developers would still be alive today?

Slow Down

Don’t build Frankenstein’s monster, or at least, don’t figure you’re “mostly done” once it walks and talks, as the strict waterfall process proposes. Going fast is important, and necessary. You can’t fix pure ideas very easily, because it’s hard to see what’s wrong with them. Write the code. But know that it’s going to have to be rewritten, possibly several times. Plan for it, and keep it in the back of your mind.

The hazard of moving quickly is that we tend to go with the easy choice, rather than the good choice. Adding one more method onto the class I’ve already got is easier than creating a whole new class; nevermind that it doesn’t really “fit” with what this type of object is supposed to be doing. Do that once, and it’s probably a lot simpler and clearer than any alternative. Do that 50 times, and you end up with a theological problem, or something else equally pungent.

This will happen. Every time. If you don’t have some downright embarrassing shit-tastic WTF-worthy code lurking in your project—I mean, the kind of thing where you see a bug report go to your coworker, and you say to him, Oh, I’ll take that one, I know that feature, but really, you’re saying Please don’t look at that code, or at least, if you do, please don’t judge me—then you probably weren’t going fast enough. The remarkable thing isn’t that this team has produced a product that “already” needs a cleanup; the remarkable thing is that we all seem to recognize that we need to clean things up, before it’s blown up in any serious way yet. That’s why I’m thankful to be on the team that I am. I’ve heard it said that the difference between a good programmer and a bad programmer is that a good programmer feels pain when he looks at ugly code he’s written, while a bad programmer thinks everything he’s done is great.

I used to be proud of every bit of code I’d written, like a two-year-old who just learned how to use the potty. Then I grew up a little bit, and realized that my shit stinks, and hoped no one would ever see it. I’ve grown up a little more, and realized that everyone shits, so there’s nothing to be ashamed of. I try to work towards a point where my actually shit doesn’t stink at all, but I’m not all that hopeful about that. The fecality of this line of metaphor, while a bit disgusting, is meant to highlight an important point.

You want to create code that is nice to be around? Do your best to build quality in from the start. The more you learn, the better you’ll be at it, especially if you manage to code yourself into some truly awful maintenance nightmares, where you can’t walk away and can’t blame anyone else. But no matter how good you get, it’s still gonna stink most of the time. Everyone writes bad code. So fix it.


Wikipedia says:

Refactoring neither fixes bugs nor adds new functionality. Rather it improves the understandability of the code or changes its internal structure and design, and removes dead code, to make it easier for human maintenance in the future.

To relate to the twin principles of software development, “Go fast” and “Does it work?”, refactoring is not important early on. Surely, doing things as close as possible to The Right Way is usually a big help. But The Right Way almost always flies in the face of the “Go Fast” maxim.

I’ve been privy to a few refactoring projects, I think I can safely say that almost every single one was an utter disaster. Without naming names, here’s how a few turned out:

  1. Lead engineer spent 3 months internally redesigning a feature with the intent of making it more extensible in the future. Released. Numerous bugs and problems. When I left the company, it was still unstable. FAIL
  2. The product has grown into an amalgam of junk, and no one knows how to maintain all the disparate pieces. Can’t add features easily. Needs to be re-architected. [[numerous planning meetings]] OK, management says we have 2 months, so let’s just add another layer on it, change how it looks, and call it a day. Result was even more disorganized and difficult to extend. FAIL
  3. #2, again, with new people driving it, who insisted that the last ones were all wrong. Same result. DOUBLE FAIL
  4. Repeat, again. TRIPLE SUPER DUPER FAIL
  5. Last, but oh, so certainly not least, there were all the refactorings that didn’t even make it to a real execution. The times when the development team was so fed up that they pushed to change things, and were told repeatedly that they could get to that just as soon as the next project got done—there was always a “next project.” It works now, what’s the matter with it? To the best of my knowledge, of three teams where I saw this pattern, (a) one company no longer exists, (b) another company lost all their best talent, and (c) in the third case, the team dissolved and everyone moved onto other projects (it was at Yahoo!, so it’s not like they were going to go under.) For failing before they’d even gotten started, these refactoring projects are QUADRUPLE UBER KAMEHAMEHA FAIL!!!

I strongly doubt that any of these cases were rare. In fact, I think they’re the norm.

It’s easy to cite bad management, and in a few of the above cases, management was to blame. They put a high premium on getting new features, without recognizing the hidden costs down the road of dealing with the increasing instability. However, the fact that I’ve come to realize is that most developers don’t really understand maintainability, and thus, tend to completely miss the point of what the purpose of refactoring should be. As a result, their attempts at refactoring end up being worse than failures, because they are interpreted as success.

I did get to see one significant refactoring project that I’d consider a success. It was done in an under-the-radar sort of way, simply because it was the only way to add new features to a product that had grown frightfully unwieldy. Also, I’ve seen and done a lot of small scale refactorings that were executed well and increased the quality of the product as a result. (These, as the Wikipedia article mentions, are usually referred to as simply “cleaning up” the code base.)

Refactoring can only be justified in the “Go Fast” methodology on the condition that it makes a more stable product, thus enabling developers to more efficiently create working software. Elegance is not an end in itself. The end goal is a working product, and code is the enemy. The vast majority of software development is about modifying an existing product. The short-term memory of a human is limited and fleeting. The less information that must be held in mind to understand a given section of code, the more quickly code can be modified, the lower the likelihood of introducing bugs with any given change. So, there is, ultimately, only one purpose that Refactoring should serve:

Be more obvious.

Love Your Monster

Inside cover of the original 1918 Frankenstein by Mary Shelly.Victor was so consumed by his desire to create something new that he ended up rushing the job and not thinking ahead. When his creature opened its eyes, he feared it, and fled, and tried to hide from his sin. Shunned from the world, the creature asked Victor to make him a companion. When his request was denied, he proceeded to destroy everything Victor loved.

The moral of the story: Don’t ignore the ugliness you create. It will find you.

Refactoring should make the code simpler. Complexity is the demon that we fight in writing code. It is the enemy. It opens the doors to bugs; it increases the difficulty in fixing problems and adding features; it raises the learning curve for new developers. Sadly, a lot of refactoring seems to make code more complex rather than less.

In my opinion, there are a few guidelines that work well when changing around existing code. These guidelines apply equally well when creating new code, but since it’s harder to see the whole picture in that scenario, mistakes and shortcuts are more forgivable.

  • There should be as few layers as possible, and no fewer.
  • There should be as few objects as possible, and no fewer.
  • Each object should be as small as possible, and no smaller.
  • Each object should know as little as it needs to, and no less.
  • Each piece should have a job, and should stick to it.
  • Comments should be mostly unnecessary.

Refactoring isn’t about using the coolest object oriented tricks that you just learned. It’s not about making the code “more abstract.” Abstraction is a necessary evil, in my opinion, not a feature. Like Victor, it’s easy to get wrapped up in testing out new discoveries. Most of us got into this job because we like playing with new puzzles, and that’s a great thing. But one of the most satisfying puzzles is to reduce the complexity of an implementation without reducing the problems that it can solve.

I’m a big fan of “comment-driven development”. That is, write out the pseudo-code in comment form first, stubbing out the functions that need to be implemented. It helps to separate the design and implementation phases, since it can be hard to do both simultaneously. As implementation kinks are getting worked out, the design might change slightly, but it helps to have the comments there as a guide to what I was originally thinking. Once a module moves into a more stable place, the comments are a liability. At best, they’re clutter; at worse, misleading.

I like using those comments to clean up the code as well. The process goes like this:

  1. Read through the code, starting at the entry point, and opening each file as it’s referenced.
  2. If a section of code has a shit-ton of comments, then it needs work. If it’s clean, just make sure it makes sense.
  3. Make sure the comments match what the code is doing.
  4. Remove any incorrect comments (comments that are lying.)
  5. If the code isn’t understandable without the comments, then dig in and clean it up. Rename methods, sort things better, etc.
  6. Remove any and all implementation-related comments. Those are dangerous.
  7. Repeat until the file is almost comment free, and understandable.

I’m not so certain on the common refactoring claim that it should not change any existing functionality. Sometimes, this is of vital importance. However, there are plenty of times where the feature set of a bit of code simply needs to be trimmed or revised. For public-facing APIs and products, either the functionality should not be changed, or interfaces should be publicly deprecated long before they are actually removed.

For internal code, or APIs that are only used by a small number of consumers, my favorite approach is to just change it, and then see what breaks. This should never be done in a way that affects customers, of course, but in a development environment, it’s quite appropriate to just change the back-end, and then update the front-end once it’s fixed. Defensive coding is a good practice, but overly defensive coding, where errors are not surfaced at all, can hide problems that may turn up later in unexpected ways.

Don’t bother refactoring something you just wrote. You’ll only make it worse. You have to wait until you don’t remember how it works before you look at it again, or else you won’t be able to appreciate the difficult bits. If you don’t have time for that, then make someone else do it, and tell them to be as harsh as possible.


As they say, No matter what the problem is, it’s always a people problem. The first and most important step in any refactoring effort is to detach our egos from our code. This is even more of an issue if something you wrote is being refactored by someone else, or worse, if you are refactoring someone else’s code.

I’ve worked hard to develop a pretty good understanding of software development. I’ve found that the technology side is easy—on the human side, I still have a lot to learn, and probably always will. Like a lot of geeks, I didn’t really get into the whole “social” thing in a serious way until I was almost an adult, and even today, I tend to focus on The Project and keep my distance from the rest of the world. So, I suppose I don’t know the best way to handle this part of the problem.

TBL’s classic recommendation to be loose in what you accept, and strict in what you send seems to apply well to this case. On a healthy team, everyone is committed to a successful product, and leaves their egos at home when they come to work. Code is passed around and everyone seeks as much input as possible. On a dysfunctional team, everyone’s sense of self-worth is tied up in the appraisal of their work by others—which has the twin effect of making them overly sensitive to criticism, and overly critical of their teammates.

It’s natural to put up resistance to the prospect of your code being modified. After all, you wrote it that way for a reason. Sit down with the other developer, and try to hash out the best approach. If arbitrary decisions have to be made, explain that the best approach is to make them consistently. Don’t focus on the problems in their code, but rather on the need for consistency and abstraction throughout the product.

If someone won’t listen to reason, frankly, the only solution I’ve ever found is to either leave or wait for them to. Hopefully they’re a contractor or something, so you can just wait, and do your best to isolate and minimize the damage as much as possible in the meantime. If not, then the issue should be escalated, if only because it’s polite to tell your manager that their ship is in danger of sinking.

Oh, right, that book…

Refactoring, by Martin FowlerI read Refactoring about 5 years ago, but I’ve been meaning to pick it up again, since I’ve gotten a fair amount of real-world development experience since then, and have seen my share of dismal failures. But this isn’t supposed to be a book review, so I won’t review it in any detail. It’s a good read, though, and I highly recommend it, even though it is mostly in Java, and seemed (to me at least) to simply highlight a lot of design problems inherent in the Java language that lead inevitably to bloated over-sized code. But that’s a post for another day.