I am in love with the Bourne Again Shell. Bash is a fantastic terminal environment, and a very interesting programming language in its own right.

I grew up using DOS. Unix shells always seemed scary and foreign, for some reason. I think a big part of the reason that I stayed on Windows for as long as I did was that I knew DOS, and could always resort to it in times of need. As a support tech in college, I got very familiar with DOS. It was comfortable and handy.

When I got into web programming, I had no choice but to learn at least the rudimentary basics of a unix-style environment. At that time, I was as a support tech at a software company whose product uses VisualBasic ASP over IIS, and had started playing around in that environment. However, I had no money, and web hosts that support ASP cost money.

By some stroke of luck or persistence, I managed to find a host that was free, Beigetower. That link doesn’t work any more, of course. I don’t know how long that server’s been down. Cuong Nghiem’s still listed as the owner on whois, though.) On top of that, they provided shell access. Not being able to resist the temptation to tinker with a real live web server, I learned enough Bash to poke around. As I learned more PHP, I picked up more bits and pieces along the way.

When I started at Yahoo, it was sink or swim to some extent. I was issued a FreeBSD dev box, and a windows desktop. I did most of my development in my safe windows world, but there are *a lot* of things you have to do on your dev box as an engineer at Yahoo that simply can’t be done in any other way. Bash was a regular part of my daily routine almost from the first day.

Eventually, I fell so head over heels in love with Bash that I had to say goodbye to Windows and get a Mac.

There are many good reasons to get a Mac instead of a PC. They’re easier and more intuitive in a lot of ways. They certainly look a lot nicer on almost every level. But I’d be lying if I said that having Bash right there all the time wasn’t the ultimate dealmaker.

One of the best things about Bash, in my opinion, is the fact that you can add shortcuts to your .profile (or .bash_profile or .bashrc, of course) that will be loaded every time you log in. As soon as I learned this, I started adding stuff.

Eventually, I had my profile so tricked out that it was hideously painful to have to work on a machine that didn’t have my shortcuts. Without my customized prompt, I’d get really confused about which directory I was currently in, and have to type pwd every three seconds just to not get lost. The sequence would usually go like this:

$ la
-bash: la: command not found
$ la
-bash: la: command not found
$ la
-bash: la: command not found
$ alias la="ls -laF"

Those first few entries of la were accompanied by plenty of growling and confusion, every. single. time.

So, I got into the habit of using scp or rsync to send my bash profile to any machine that I’d have to work on. However, some machines have vi, and some don’t. I’d make a change on the profile on my mac, and forget to sync it to my dev box. Machines that I didn’t log into often would be so far out of date that it was the “la” situation all over again. What’s worse, I put some stuff in other shell script files that were called by the profile to do other things, so it was no longer straightforward to keep them in sync. I’d send half the package, and then log in, and be inundated with errors all over the place.

When I switched from FreeBSD to a Red Hat Linux box, half my stuff didn’t work quite the same way. Also, I moved Foo Hack to a great new host, which is running Ubuntu. On the Mac, I use MacPorts, and the Ubuntu slice uses apt-get, and the RHEL of course uses yinst, the Yahoo! package management tool that quite simply makes all others look like crap by comparison. (Sadly, yinst is not open source. There’s this hack to simulate yinst on top of apt, though.) The Ubuntu and RHEL environments used vi as the default editor, but I’d changed that on the Mac to point to mate, the command-line interface to TextMate.

To say the least, a simple sync wasn’t cutting it even a little.

I re-wrote all my bash profile goodies to fit in a single file. Shell scripts were replaced with functions so that they could all live together. The edit command was defined to use the first available editor from a list of alternatives. Then, I put it in a file that was external from the main .profile or .bashrc file, so that it could be synced without overwriting anything that the OS placed in there by default. The only thing to do on a new server was to add . ~/.extra.bashrc to the bottom of whatever profile script it was already using.

I had already added a command called editprof that would bring my profile up in an editor. All that was needed then was a pushprof command that would rsync the .extra.bashrc file to a given host.

I’m releasing it under the WTFPL. There are actually a few files, since some stuff simply didn’t make any sense to have in certain environments. The copy here on foohack.com is symlinked from my home directory, so it’ll stay pretty up to date. Feel free to copy, learn, or, like the license says, just do wtf you like with it.

A word of warning: like government regulation, this file increases in size but rarely gets revised. Some stuff may be completely useless or pointless. Some of it I hardly ever use any more. Figuring out better solutions to these problems is left as an exercise for the reader.

Get my bash extras.

Sadly, it doesn’t work in Tiger, and I haven’t made the jump to Leopard yet. If you’re a TextMate user who’s still on Tiger, and wants this feature, I tweaked Henrik’s AppleScript to play nice with Tiger.

Open in TextMate (for Tiger)

Enjoy!

Update: Just fixed the script to work with folders and files that have spaces in the name.

Well, as of 2008 March 25, it was a secret. This press release broke the news a few days ago, so I doubt I’m giving anything away to say that Yahoo! is busy working on implementing a framework for OpenSocial applications that tie into various sorts of Yahoo! data.

Buzz was a very fun project to work on. After games’ mountain of legacy and the brand universe fiascos, it was really great to be able to have a hand in developing a site from scratch and also have enough space and resources to really create quality. (It has its problems, but they’re being worked out, and overall I’m quite proud of the site we built.)

However, the really interesting parts of Buzz are all behind the scenes. The front end was an exercise in the complicated and delicate art of engineering simplicity and eliminating redundant effort. Even in the face of exacting requirements and changing specifications, we ended up releasing a site that performs well and is relatively easy to work on. On some level, I feel like it was my karma baby for some of the complicated monstrosities that I’ve left in my wake as a developer.

But, with Buzz behind me, creating and maintaining yet another website has gotten a bit old. Nothing wrong with that work, but I’m done with it for now. Time to do something else.

OpenSocial and Mash-ups

Douglas Crockford has called mash-ups One of the most exciting features of WEB 2.x, and I couldn’t agree more. The vision of the web from the get-go was a distributed platform for innovation. As client technology and usage advanced, dynamic features followed naturally. The ability to mix-and-match data and functionality into new applications is a very exciting prospect.

However, like so much potential innovation on the web, this effort is severely hampered by the same system that makes it possible. The browser Dom was designed by committee, and the committee didn’t actually meet for a while, and the members of the committee were trying to kill one another. Throughout the browser wars, IE and Netscape introduced features in a frantic arms race to try to shut one another out of the browser market. Many of those features were not really ready for prime time, and many others probably should have been wiped off the whiteboard and never made it into the code.

On the plus side, browsers and web pages were able to innovate much faster than the w3c’s crawl. Much of the time, the “official” specs only included a feature because a few browsers had already implemented it and it was in widespread use. That speed got us to where we are, but lots of important things were sacrificed or forgotten along the way. Among the missing: a proper permission model. Same-domain restriction works to some degree, but it makes mashups trickier and there are still big holes. In particular:

• Scripts run with the same access permission as the page that invokes them, even if they’re hosted elsewhere.
• XHR calls can’t get data asynchronously from external domains.
• The Dom is one big global that all scripts have access to.
• Iframes restrict permission, but they’re a pain to work with.

Sadly, this means that mash-ups are faced with the choice of:

1. Only accessing public data, lest some untrusted third party get ahold of something it shouldn’t have.
2. Running the risk that a third party might steal some data.

Essentially, mash-ups are thus restricted from really doing anything interesting. PayPal or Amazon or Yahoo—-that is, sites with money and interesting private data—-could never risk exposing their data to a mash-up. Their visibility, and the value of the private data they hold, makes them prime targets for attack. And, since they have money, and visibility, they’re also prime targets for lawsuits if a user was damaged by a mistake.

OpenSocial represents a concerted effort by some very big players on the web to make severely interesting and useful mash-ups not just a reality, but common and relatively easy to create and host. API layers will sit between the developer and the browser, and between the developer and the container page, allowing them to code as if it’s 2099 and the browser is sane. It’s no small task. When Hao asked me to join the team, and told me what they were doing, my initial reaction was Well, it might end up a total failure. But it’ll be one hell of a fun problem to work on. I’m in.

There were hints and rumors that Yahoo was doing something with OpenSocial a while back. (We were.) But, the main reaction to the press release really should be Well, spluh. Yahoo’s all about social; it may have started with indexing the web into searchable categories, but some of its earliest and most popular applications are mail, instant messaging, and groups. Yahoo’s all about open; we invest countless developer-hours into the open source technology that we use. Yahoo has a lot to gain from a more exciting social internet; when the ocean rises, the bigger fish benefit the most.

Warnings Wrapped in Compliments

For a month or so, I’ve been transitioning off the Buzz team and onto the team that’s doing the OpenSocial container stuff. As people found out that I was joining that team, I heard this line a lot from various people: Oh, thank god. They really need a good front-end.

There’s a very flattering element to that, and I’m not trying to brag. Objectively speaking, I know I’m good at what I do. I love doing it, and I work hard at it, and that’s just what happens when you work hard at what you love. No magic at all.

At the risk of sounding like a falsely modest blow hard, though, on a more moment-to-moment basis in my own world, I focus on many of my own failings that others rarely see, and I am always very impressed when someone solves a problem that I was stuck on. I often feel like I’m still new at Yahoo, and on some very insecure level, I’m waiting for them to find out that I’m just playing at this stuff and tell me I’ve fooled the real adults for long enough, and it’s time to go home.

But I am good at what I do. It’s real. These skills are rare, and they’re worth money. When others acknowledge it, though, I kind of feel embarrassed and get all “aw, shucks, twern’t nuttin’” and find myself unable to make eye contact. (A few times, this reaction has evoked a No, seriously! I mean that! which just adds to the effect.)

So, the flip side of that comment… yes yes, I’m a great webdev, blah blah, but say that first part again? It’s a bit like hearing, Oh, thank god they picked you to wrestle that giant raptor. He’s been killing one guy after the next, so I’m really glad that they finally picked someone who can take him down. Why couldn’t it have been That’s great! There’s a lot of great talent on that team, I’m sure you’ll fit right in!

Well, that’s the cards that life deals you sometimes. Impossible problems are the most fun, anyway.

And I’ve found that there really are a lot of talented developers on this team. Some more have joined it recently. I’m really happy to be able to be working in this group.

Meetings

Somehow, I managed to work myself into a role on this team where most of my time is spent in meetings. Some of them were soul-sapping and boring, as meetings can be sometimes, but lately, they’ve seemed to focus more and more on solving front-end problems at an architectural level. After almost every one, someone whips out a phone to take a picture of the whiteboard, lest our valuable work be lost to the eraser.

Additionally, I’ve been attending the YUI team meetings, as they’re interested in knowing what they can do to aid Yahoo’s big bets. Through that, and by virtue of working on a product that promises to push the limits of what the browser can safely do, I’m finding myself consulted because Microsoft is asking Yahoo what we think of the IE 8 beta, and what we’d like to see in it. I’ve been asked to teach a CSS class for the next round of incoming junior/trainee webdevs (Jukus, as they’re called). I’m meeting with other groups and helping to figure out what OpenSocial means for their product’s context, and how they’ll use what we’re building. Today, we’ve got a meeting at the Googleplex.

It certainly looks like success. And the pay is pretty good. The product is fun and open and creative. It’s one puzzle after another.

But, I’m feeling a bit envious of the other developers I talk to who get more time to write code. From Tuesday to Thursday of last week, iCal was booked solid from 11-5 every day, I ended up in ad hoc discussions about things (or dealing with email) until about 7 or 8 each night.

I have tried hard not to become a manager at Yahoo. Perhaps a management path is in my future somewhere, but right now, it just seems like a job I would not enjoy. Programming is a very social activity if you’re doing it right. With a new team and a new product, there’s plenty of architecture and designs to figure out, and I’ll be coding again soon enough.

I hope.

Hey, one more thing…

We have openings for great engineers and webdevs. If any of this sounds like something you’d be interested in, let me know. I’m 100% serious. Javascript and PHP experts, we need you. Yahoo is, in my opinion, one of the best places for talented engineers to work. They generally treat their employees very well, and this is a product that is just full of fascinating puzzles.

I’m thinking specifically chpass, finger, and pw.

I know what you’re thinking. There have been a few forays into this arena. MySpace, Facebook, and Plaxo come to mind, not to mention whatever else some MBA has stuck to a “social network” this week. (It’s just like a regular duck, but this one swims around the lake and lets you put all your friends in a list! I’ll be rich! The sad thing isn’t that he thinks it; the sad thing is that he just might be right.)

The social-for-the-sake-of-being-social sites tragically miss the point. I have resisted getting a Facebook account for a few years now, and even deleted my MySpace account. They seemed to require a lot of time and effort doing basically nothing, and didn’t give me what I really want.

Managing data is not (necessarily) enjoyable

I hate managing contact lists. The worst contact list, the one that is the hardest to manage, is the one in my head. Every day I get older, I get worse at remembering phone numbers, and I like to know who’s calling me. I like to see your picture when you call, see your real name when you email. I want my email program or my phone to know who you are when I start to type the three letters of your name that I can remember off-hand, even (especially!) if you’re someone I don’t talk to often.

That’s why I shelled out $40 for Missing Sync so that my phone and computer can share an address book. I have an Applescript program sitting on my back burner that will sync any contacts I add in Adium into this same collection, and even look up their contact details from Yahoo’s corporate intranet (since most of the time, they’re work mates.) Automated replication is still not great, but it eases the pain of managing multiple lists.

Facebook and MySpace are software platforms designed around the premise that managing a contact list is fun. And it can be in that 12-23 age range where we attempt to define ourselves and carve out our place in the world through our social connections. That’s a key demographic for advertisers. Good for you, Facebook. But if I wanted to spend all this time managing my friendships, I’d have more of them in real life. Ooh, burn! i mean… hey, wait a second..

Plaxo is actually a pretty good approximation of what I’d like to see, at least on the “managing contacts” side of things. Granted, I’ve been spamvited to their service by half a dozen people I hardly know, which is a classic example of “let’s be viral” gone horribly horribly wrong. But their product offering is pretty close. You get one contact list online, and it syncs with other areas. It’s unfocused since they’ve added “Pulse” (basically an RSS aggregator for your other web profiles), but still pretty good.

However, even Plaxo misses a key point, and makes several fatal flaws. I’m actually talking about a profile and contact management system that is much grander.

DRY — What Changed

In a relational database or data map, the idea is to keep a piece of data in only one place, and store the relationships between entities rather than making multiple copies. Most contact management systems, from a little black book to the cell phone contact list to Outlook to Plaxo, fail to implement this simple principle. Instead of making each node in the network keep track of all the data about all the other nodes in which it is interested, instead let each node control its own data, and store links to the nodes in which it is interested.

In the old days of land lines, the phone book was enough. If you knew someone’s name and city, you could get their phone number and, perhaps, their street address by performing a simple lookup. Each user had the option to control how much information was shared with the public. Until the autodialer came to telemarketing, the abuse rate was limited by the cost of using the technology.

Today, each person has many more pieces of contact information, and the cost of abuse is virtually zero. There is no way in hell that I’d let anyone publish my actual cell phone number, and once an email address is exposed, it’s basically useless. Spam fighting is an arm’s race, and an unfair one even for Google and Yahoo to fight.

Why we need those commands on the web

Back to my original point: chpass, finger, and pw.

chpass

add or change user database information

In other words, manage my info in one place.

finger

The finger utility displays information about the system users.

In other words, look up the information that other have made available.

pw

create, remove, modify & display system users and groups

In other words, specify who has permission to what.

Many large companies have some sort of online system like this. At Yahoo, it’s the almighty Backyard, the corp website that started as a list of email addresses and grew into a full-scale intranet with contact lookup and LDAP access. (It also features conference room booking and documentation searching and plenty of other handy things. But mostly, it’s still all about the mega employee contact list.)

You manage your own profile, and make sure that your numbers and whatnot are up to date. No one else ever has to worry about how to contact you, because it’s all in one place. However, that only works because access to the backyard system is tightly limited to current employees, so abusing the system would entail serious consequences for the abuser’s reputation (and career).

In other words, we have finger and chpass, but pw is being done manually by the HR department, which limits the possible size of the network considerably.

Abuse Prevention is Extremely Non-Trivial

The easier it is to use a networked contact management system, the easier it is to abuse. The more useful it is for you and your friends and associates, the more useful it is for spammers and scammers. Already, we have to keep our email addresses hidden from strangers. Imagine how much worse it would be if a PPC pusher could just e-finger “isaac.schlueter” and have my home address, email, phone number, instant messenger alias, birthday, and photo. Yes this is exactly the sort of information that I’d like to easily share with everyone else.

Everything that has been done so far in the area of email spam, while impressive and necessary, is fundamentally inadequate. As long as it remains profitable for a spammer to send out 100 billion emails every day, it will happen. Any attempts to prevent or avoid this behavior run counter to the incentives of the market; which is to say that it’s a bit like building a dam of sand and expecting to stop the Mississippi. Won’t happen. A bigger dam will take longer, but eventually, they’ll all crumble.

In order to truly divert human behavior, the incentive must be dealt with at the source. Direct attacks against the offenders (ie, shutting down their accounts) are not effective in the long run (they just get new accounts.) Negative incentives, such as putting spammers in jail, are not going to be effective in the long run, because it doesn’t push the cost of spamming up high enough. John Q. Spammer doesn’t think he’ll be the one to get caught, and he’s probably right.

I don’t claim to have this bit of the system figured out, not by a long shot. But I have a few ideas.

IRL

In real life, we meet a lot of people, and many of them can and do annoy us by contacting us in unwanted, if mostly harmless, ways. The foul smelling man who stops babbling for a second to ask me for a quarter. The smiling woman who shoves a tract at me and tells me I’m going to hell. Sadly, the list goes on and on and on.

However:

1. It’s easy to size someone up quickly, because:
2. Annoying people build a reputation for being annoying, because:
3. They’re real people and you can see who they are.

There are still, of course, the violent offender and the con man. However, in real life, direct attacks incur a high degree of risk, due to the chances of being caught or retaliated against, and so law enforcement has a relatively easy time keeping serious criminals in check. And those looking to do you harm by gaining confidence and taking advantage of it, well, they’ll always be around, but they’re pretty rare and the reputational aspects keep them somewhat in check as well.

So…

Entrance into this global open personnel file would require that an account be tied to a single real person, who doesn’t have any other account in the system. Accounts that are not “backed” by some kind of reliable identity are only given some kind of limited provisional access—-perhaps they can email a user through the system, but they cannot get the user’s “real” email address, and users would be able to deny access altogether to unidentified strangers if they chose.

Identification is itself a non-trivial task requiring a high degree of trust from the web site. Even if you know it’s 100% secure, being asked for your date of birth, SSN or passport, and a major credit card is a tall order. Without biometrics, it must come down to discrete bits of information at some point, which can be (and often are) faked.

A rinky-dink fly-by-night startup can’t hope to achieve this level of trust quickly. And, without getting a critical mass of users, the value proposition to new users is a lot tougher.

The company to build this system would need:

1. A huge base of existing users and preferably their contact details, too.
2. A strong reputation for protecting user data.
3. Impressive engineering resources and domain knowledge in the areas of spam protection and social networking.
4. A serious commitment to open APIs that help the web as a whole.

If it’s not everywhere…

…then it may as well not be anywhere. The goal of this system would be to revolutionize contact management the same way that email and hypertext revolutionized written communication.

Just as email works in any email client, and web pages work can be viewed by any browser, the APIs provided by this system would have to be completely open. Any application must be allowed to interact with them, both to change data and fetch it.

In order for it to work, and really have the effect that I’m talking about, there must be absolutely no lock-in, no up-sell, and reasonably liberal rate limits.

$$$

How does something like this make money? That’s an open question, and a big one, probably part of the reason why I’m still pushing bits in a day job and not out getting VC to build this thing. I also happen to really like what I do at that day job.

Maybe it would have to be something built under the Apache foundation or some other OSS group, and sponsored by donations of capital and resources from some major players in the online social arenas. Maybe there’s some clever way in which smaller users and early adopters get the API for free, but then charge everyone else.

Who could do this? What’s going on now?

OpenID is a great start, but what we really need is an open profile and open contact list system, and OpenID doesn’t provide that.

Google’s Open Social is an interesting product, but the more I read about it, the more I think it’s not quite low-level enough to really deliver on what I’d like to see here. While it promises to expose social data to third-party applications in an API that could be consistent across social websites, it doesn’t fully address the issue of being able to manage contact data in a distributed way.

As I said above, the company to do this will need:

1. A huge base of existing users and preferably their contact details, too.
2. A strong reputation for protecting user data.
3. Impressive engineering resources and domain knowledge in the areas of spam protection and social networking.
4. A serious commitment to open APIs that help the web as a whole.

Yahoo has all four of these, but that whole China escapade has damaged Yahoo’s reputation in the eyes of many users. Of any company on the web, however, Yahoo has perhaps the most to gain from such a system and a lot of resources and domain knowledge to throw at the problem.

Even if they only share user information when presented with a subpoena, that means that using this system exposes my information to governmental intrusion, which is deeply problematic. In order to be truly trustworthy, a stronger commitment to protecting privacy needs to be in place than just words on a corporate press release. The officers of the company to provide this service should enter into a binding agreement that they will not knowingly expose user information, even in the face of governmental pressure.

Like I said, I don’t have all the answers to this product. But I know that, as a user, I’d be absolutely delighted to see something like this take hold.

It took a lot of work to pull this thing together. Monday night was the craziest month of my web development career, culminating in this tweet posted as I finished pushing the last packages out into production. I managed to ignore the page that came in 5 hours later. I wonder if it was some kind of subconscious self-preservation that made me leave my phone on vibrate when I finally went to bed. Sorry bout that, Trang.

And despite the rather disappointing leak, it seems to have been pretty well received.

We worked really hard on this project, and while it’s definitely still a “beta”, the feature set is enough to be interesting, in my opinion. All in all, this has been a highly rewarding product to be a part of, and I’m extremely proud to have been a part of the team that built it.

I’m in the process of moving onto a different team, and I have mixed feelings about the transition. On the one hand, the team I’m joining is doing some truly ambitious and interesting things, and it’s more in the direction that I want to go. I’ve been starting to get tired of building one web site after another, and this will be an opportunity to do something much more in the “web application” direction. On the other hand, the Orion team is one of the best groups of developers I’ve ever worked with, and it’s sad to leave them.

But that’s the joy of working at a company like Yahoo—if you get bored of doing one thing, then there’s always something else, and no matter which team you’re on, there are lots of top-notch developers among your coworkers.

I’m not a very understanding end-user when it comes to this site. The Foo Hack design team got fed up with my complaints, and decided to give in and make some changes.

Feature creep is a bitch when you have a deadline. For personal projects, there is no deadline, and so a bit of feature creep is fun, within reason. That’s a big part of the reason why I have so many back-burner projects that are almost finished; just as I near the finish line, I decide that it needs a little more, or another project gets more interesting for a while. So, on to what changed.

Fonts

I was a computer geek first, but when my dad replaced our DOS home computer with a Windows 3.1 box, I fell in love with fonts. I can waste an afternoon surfing the web downloading fonts that I’ll probably rarely (if ever) use. Web design is about 90% typography, and it is an art that blends strict perfectionism and fluid acceptance in a beautiful way.

For the main body text, I wanted something subtle and easy on the eyes. Something that would let the writing speak in the voice that I intend, without getting in the way. For the “small” items of diminished importance (asides and things in <small> tags) I wanted a serif font with a very thin ex measurement, so that the text could “feel” smaller without altering the line-height. For the headings, I wanted a rounder, more open font that could evoke a sense of boldness without being too heavy. Subheadings and meta items (the date, “comment” link, etc.) should use the same font, but with less letter space and a subtler color.

For the body text, I decided to go with Helvetica, the classier old-world cousin of Arial. I was initially pleased with the results.

Cliché though it may be, Times New Roman was the perfect choice for the diminished items. It’s very light, yet still fairly readable, and when italicized, it practically disappears. I was already using it for that reason, but since I don’t love TNR all that much, I was willing to entertain other possibilities.

If Helvetica is so great, why do I see Arial?

A problem came up when I noticed that the line-height was getting messed up whenever a TNR inline element would wrap to the next line. I figured out the problem.

Let’s say you have font A and font B. You create a block-level element using font A. Then, you have an inline element using font B, which wraps to the next line.

If A and B don’t put their letters on exactly the same point in the line-block, then the line-height will adjust up or down, as the next line is set by the position of the letters in B instead of the position of letters in A. Since the B element isn’t a block-level element, it won’t create a whole new block, and will have the effect of adding or removing a few pixels from the overall height of the A element.

To correct this problem, and still have 2 fonts sharing the same block-level element, you need to find two fonts that have exactly the same vertical letter placement on the line block.

I’m not exaggerating when I say that I created a test page and exhaustively tested every combination of serif and sans-serif fonts on my Mac. It took about a week. There was only one combination that worked.

Arial and Times New Roman.

Ah, compromise, my old nemesis. You strike again. Not willing to give up the line-height strictness, I gave in and decided to use Arial instead of Helvetica. On the bright side, Windows users were mostly going to see Arial anyway, and they’re both tried and true web-safe fonts. And, at typical screen resolutions, it’s hard to tell the difference anyway. But still, it’s a bit of a sore point.

Gotham, Large and Small

Gotham, the masterpiece homage to urban signage by Hoefler & Frere-Jones, is quite possibly the best font ever devised. It’s open and confident even when it’s not bold, balanced and practical. I went with this as the headings, and it also was the best candidate for the smaller informational bits. (After all, it seemed fitting to put navigational links in a font designed for signage.

The downside of such a perfect font is that it’s not free. It would be fantastic if Apple or Microsoft were to license Gotham for their respective operating systems, but I don’t see that happening any time soon. Those that don’t want to install Gotham will see Century Gothic (if it’s installed, most likely because they got it bundled with Microsoft Office), or something in the Lucida family.

Accentuate the Important, Diminish the Rest

I brought down the contrast a bit, and did away with the background graphic. Some color was sprinkled around to help create a meaningful mental model of each page. I got the idea for the categories above the titles from Rands in Repose, one of my favorite regular reads.

The default set of post meta info that Wordpress provides is far more than necessary. I got rid of everything that didn’t directly benefit the goals of conversing with the world through this blog.

Comments

It seems awkward that my comments should have a blue left border, and reader comments are unadorned. Simply using random colors wouldn’t do. So, I wrote a function that will hash a string into a color value. A simpler hash would have sufficed, but I wanted more control over the range of colors that it selects, and rand() was a good fit. So, your comment will always have the same color (unless you use a different email address, of course) and all the colors will be in a particular mid-range of luminosity that is bold, but not overpowering.

I’ve already run into a few situations on this blog where I felt that threaded comments would have been helpful. However, the threaded forum-style comments would be complex and counterproductive. I grabbed a standard threaded-comment plugin, and tweaked it to replace the interface with a few hyperlinks. Since conversations are more naturally many-to-many, I’d like to implement something along the lines of what Dunstan Orchard used to have on his blog, but doing that the right way means a bit more investment, and this project was dangerously close to being back-burnered forever.

That’s right. We haven’t released anything, and there’s already enough bloat to justify spending at least 2 weeks cleaning it up.

Of course, due to the pressures of market, and the morale effects of delaying the release, we’re buttoning up what we have now as much as possible, and planning to work on the refactor once the beta is out there. It may actually help a lot, since the gigantic firehose of traffic that Yahoo! points at sites has a way of helping even the most egotistical of developers realize that their clever baby needs some serious surgery.

This post is going somewhere, I swear, but I can already tell it’s going to take me a while to get there. Go get some coffee and a snack and come back. I’ll wait.

Build Fast

In my opinion, the core of good software development methodology comes down to two basic questions that should be asked daily:

1. What portion of your time is spent focused on the task of creating software?
   (That includes planning, discussion, and thinking as well as coding, provided that these non-coding activities are focused and aimed at creating software.)
2. Does it work today?
   (If you don’t know, then you are full of Fail. Get on that right away.)

I call this the Go Fast method. The goal is to efficiently create software with minimum long-term cost and maximum long-term benefit, without driving away your best talent. (Have I got the attention of the business and product people in the audience? Good. Please stay for the rest of the show.)

I know that someone out there is bound to point out the obvious parallels between my “go fast” methodology and Agile, and maybe they’re right. To the extent that Agile accepts these premises as important, it is correct. But I’m claiming that everything else stands or falls based on how it affects these questions. Stand-ups, planning meetings, org charts, IDEs, frameworks; they’re all only as good as their effect on the principles of speed and stability in development.

If you’re developing software properly, you should be creating software that does something as soon as possible, and keep it up as much of the time as you can. Until there’s something to look at, you’re just spinning your wheels. Nowhere is this more true than on the web. Photoshop comps don’t break in different browsers. Product presentation decks don’t “feel klunky” when pieces load out of order. Grand visions promise to solve all your problems and make you biscuits for breakfast, but they don’t deliver.

You make software for a living? So make software, already. You wanna build a website? What’s stopping you? You don’t have designs yet? Build the parts that you do have. Always be releasing, even if your actual release date is months away.

Build the core, and grow the other pieces up around it. Every minute spent waiting, instead of actually building something, is worse than wasted. When you live in the code day by day, it becomes a part of you. You learn how the pieces work, and the knowledge is automatic. When you get too far from it, it becomes foreign, like driving a car for the first time.

The agilistas call this “shippable code.” In my opinion, that’s a horrible term. The vast majority of code that I’ve seen released into the wild was not even close to what I’d call “shippable,” but it got shipped just the same. Most of the time, the prototypes leading up to the release were significantly worse. Some agilistas are quick to point out that “shippable” doesn’t always really mean “that you would actually ship it.” Sorry, but drivable means that you can drive it, and drinkable means that you can drink it, and flyable means that you can fly it, so if “shippable” doesn’t mean “you can ship it,” then they picked a bad term.

I suspect the ambiguity is intentional on some level. I think the claim that “Agile makes software faster” is a myth. Agile methodologies, used in moderation by a well-balanced team of dedicated and talented individuals, do tend to result in higher quality software, greater management visibility, and a happier bunch of people than any alternative, but that means that the overall development time can be significantly longer. And if you’re not on a well-balanced team of dedicated and talented individuals, then no methodology is going to save that sinking ship, so dust off the CV and bail out ASAP. But I digress, and besides, you already know that I think Agile sucks.

I prefer to call it a running prototype, or an integration environment, or anything else that’s more, well, accurate. “Shippable code” tells overeager managers that they can just press “stop” at any point and ship what they’ve got. Nice dream land, hot shot, but it doesn’t actually work quite like that. Nevertheless, a good integration development environment, where everyone’s code gets placed and which is subjected to regular updates and testing—that’s important, almost as important as CVS and a decent build script, and for a lot of the same reasons. If the team’s code isn’t conversing regularly, then the team may as well not be, either. The best way to prevent blockages is to prevent assumptions; the best way to prevent assumptions is to check them every single day. So: Does it work today?

Completion feels good. Getting something that works feels good. That’s why we got into this business. Because, despite all the pain and frustration and work that goes into developing software, The High here is about as good as it gets.

You don’t get that waiting. You don’t get that talking. You get that when you are immersed in building something.

From a team dynamic perspective, I can’t even find words to express how it makes people gel when we each do our little pieces, and plug them in, and see a site come up with data and styles and behavior and images and everything. It’s a pretty marvelous event that makes everyone even more eager to get to work. If you’re a web developer, don’t wait for your designer or back-end engineer to give you what you need. Make an unstyled page with dummy data, if you have to, but make something on day 1 if possible. Then make a list of what you need. When everyone can see their effect on the product, it’s amazing how fast they tend to deliver.

Going fast and pushing to milestones also forces everyone on the team to prioritize. There may be plenty of features that might be nice to implement, but if speed is a priority, then it forces you all to work on the things that really need to be implemented. By raising red flags whenever something doesn’t work, the whole project stays in sync most of the time, and assumptions never get too far out of whack. But, of course, that’s not the whole story.

Software development is a lot like building Frankenstien’s monster. You start out with a pile of useless ugly pieces, and try to turn it into something beautiful. Along the way, it’s a monstrosity, and tends to get uglier as you tack new bits onto it. Then it kills people.

In Mary Shelly’s classic, everyone Victor loves is killed by the eloquent and sensitive creature, who was turned evil by his creator’s neglect and hatred. If software could walk and talk, how many developers would still be alive today?

Slow Down

Don’t build Frankenstein’s monster, or at least, don’t figure you’re “mostly done” once it walks and talks, as the strict waterfall process proposes. Going fast is important, and necessary. You can’t fix pure ideas very easily, because it’s hard to see what’s wrong with them. Write the code. But know that it’s going to have to be rewritten, possibly several times. Plan for it, and keep it in the back of your mind.

The hazard of moving quickly is that we tend to go with the easy choice, rather than the good choice. Adding one more method onto the class I’ve already got is easier than creating a whole new class; nevermind that it doesn’t really “fit” with what this type of object is supposed to be doing. Do that once, and it’s probably a lot simpler and clearer than any alternative. Do that 50 times, and you end up with a theological problem, or something else equally pungent.

This will happen. Every time. If you don’t have some downright embarrassing shit-tastic WTF-worthy code lurking in your project—I mean, the kind of thing where you see a bug report go to your coworker, and you say to him, Oh, I’ll take that one, I know that feature, but really, you’re saying Please don’t look at that code, or at least, if you do, please don’t judge me—then you probably weren’t going fast enough. The remarkable thing isn’t that this team has produced a product that “already” needs a cleanup; the remarkable thing is that we all seem to recognize that we need to clean things up, before it’s blown up in any serious way yet. That’s why I’m thankful to be on the team that I am. I’ve heard it said that the difference between a good programmer and a bad programmer is that a good programmer feels pain when he looks at ugly code he’s written, while a bad programmer thinks everything he’s done is great.

I used to be proud of every bit of code I’d written, like a two-year-old who just learned how to use the potty. Then I grew up a little bit, and realized that my shit stinks, and hoped no one would ever see it. I’ve grown up a little more, and realized that everyone shits, so there’s nothing to be ashamed of. I try to work towards a point where my actually shit doesn’t stink at all, but I’m not all that hopeful about that. The fecality of this line of metaphor, while a bit disgusting, is meant to highlight an important point.

You want to create code that is nice to be around? Do your best to build quality in from the start. The more you learn, the better you’ll be at it, especially if you manage to code yourself into some truly awful maintenance nightmares, where you can’t walk away and can’t blame anyone else. But no matter how good you get, it’s still gonna stink most of the time. Everyone writes bad code. So fix it.

Refactoring

Wikipedia says:

Refactoring neither fixes bugs nor adds new functionality. Rather it improves the understandability of the code or changes its internal structure and design, and removes dead code, to make it easier for human maintenance in the future.

To relate to the twin principles of software development, “Go fast” and “Does it work?”, refactoring is not important early on. Surely, doing things as close as possible to The Right Way is usually a big help. But The Right Way almost always flies in the face of the “Go Fast” maxim.

I’ve been privy to a few refactoring projects, I think I can safely say that almost every single one was an utter disaster. Without naming names, here’s how a few turned out:

1. Lead engineer spent 3 months internally redesigning a feature with the intent of making it more extensible in the future. Released. Numerous bugs and problems. When I left the company, it was still unstable. FAIL
2. The product has grown into an amalgam of junk, and no one knows how to maintain all the disparate pieces. Can’t add features easily. Needs to be re-architected. [[numerous planning meetings]] OK, management says we have 2 months, so let’s just add another layer on it, change how it looks, and call it a day. Result was even more disorganized and difficult to extend. FAIL
3. #2, again, with new people driving it, who insisted that the last ones were all wrong. Same result. DOUBLE FAIL
4. Repeat, again. TRIPLE SUPER DUPER FAIL
5. Last, but oh, so certainly not least, there were all the refactorings that didn’t even make it to a real execution. The times when the development team was so fed up that they pushed to change things, and were told repeatedly that they could get to that just as soon as the next project got done—there was always a “next project.” It works now, what’s the matter with it? To the best of my knowledge, of three teams where I saw this pattern, (a) one company no longer exists, (b) another company lost all their best talent, and (c) in the third case, the team dissolved and everyone moved onto other projects (it was at Yahoo!, so it’s not like they were going to go under.) For failing before they’d even gotten started, these refactoring projects are QUADRUPLE UBER KAMEHAMEHA FAIL!!!

I strongly doubt that any of these cases were rare. In fact, I think they’re the norm.

It’s easy to cite bad management, and in a few of the above cases, management was to blame. They put a high premium on getting new features, without recognizing the hidden costs down the road of dealing with the increasing instability. However, the fact that I’ve come to realize is that most developers don’t really understand maintainability, and thus, tend to completely miss the point of what the purpose of refactoring should be. As a result, their attempts at refactoring end up being worse than failures, because they are interpreted as success.

I did get to see one significant refactoring project that I’d consider a success. It was done in an under-the-radar sort of way, simply because it was the only way to add new features to a product that had grown frightfully unwieldy. Also, I’ve seen and done a lot of small scale refactorings that were executed well and increased the quality of the product as a result. (These, as the Wikipedia article mentions, are usually referred to as simply “cleaning up” the code base.)

Refactoring can only be justified in the “Go Fast” methodology on the condition that it makes a more stable product, thus enabling developers to more efficiently create working software. Elegance is not an end in itself. The end goal is a working product, and code is the enemy. The vast majority of software development is about modifying an existing product. The short-term memory of a human is limited and fleeting. The less information that must be held in mind to understand a given section of code, the more quickly code can be modified, the lower the likelihood of introducing bugs with any given change. So, there is, ultimately, only one purpose that Refactoring should serve:

Love Your Monster

Victor was so consumed by his desire to create something new that he ended up rushing the job and not thinking ahead. When his creature opened its eyes, he feared it, and fled, and tried to hide from his sin. Shunned from the world, the creature asked Victor to make him a companion. When his request was denied, he proceeded to destroy everything Victor loved.

The moral of the story: Don’t ignore the ugliness you create. It will find you.

Refactoring should make the code simpler. Complexity is the demon that we fight in writing code. It is the enemy. It opens the doors to bugs; it increases the difficulty in fixing problems and adding features; it raises the learning curve for new developers. Sadly, a lot of refactoring seems to make code more complex rather than less.

In my opinion, there are a few guidelines that work well when changing around existing code. These guidelines apply equally well when creating new code, but since it’s harder to see the whole picture in that scenario, mistakes and shortcuts are more forgivable.

• There should be as few layers as possible, and no fewer.
• There should be as few objects as possible, and no fewer.
• Each object should be as small as possible, and no smaller.
• Each object should know as little as it needs to, and no less.
• Each piece should have a job, and should stick to it.
• Comments should be mostly unnecessary.

Refactoring isn’t about using the coolest object oriented tricks that you just learned. It’s not about making the code “more abstract.” Abstraction is a necessary evil, in my opinion, not a feature. Like Victor, it’s easy to get wrapped up in testing out new discoveries. Most of us got into this job because we like playing with new puzzles, and that’s a great thing. But one of the most satisfying puzzles is to reduce the complexity of an implementation without reducing the problems that it can solve.

I’m a big fan of “comment-driven development”. That is, write out the pseudo-code in comment form first, stubbing out the functions that need to be implemented. It helps to separate the design and implementation phases, since it can be hard to do both simultaneously. As implementation kinks are getting worked out, the design might change slightly, but it helps to have the comments there as a guide to what I was originally thinking. Once a module moves into a more stable place, the comments are a liability. At best, they’re clutter; at worse, misleading.

I like using those comments to clean up the code as well. The process goes like this:

1. Read through the code, starting at the entry point, and opening each file as it’s referenced.
2. If a section of code has a shit-ton of comments, then it needs work. If it’s clean, just make sure it makes sense.
3. Make sure the comments match what the code is doing.
4. Remove any incorrect comments (comments that are lying.)
5. If the code isn’t understandable without the comments, then dig in and clean it up. Rename methods, sort things better, etc.
6. Remove any and all implementation-related comments. Those are dangerous.
7. Repeat until the file is almost comment free, and understandable.

I’m not so certain on the common refactoring claim that it should not change any existing functionality. Sometimes, this is of vital importance. However, there are plenty of times where the feature set of a bit of code simply needs to be trimmed or revised. For public-facing APIs and products, either the functionality should not be changed, or interfaces should be publicly deprecated long before they are actually removed.

For internal code, or APIs that are only used by a small number of consumers, my favorite approach is to just change it, and then see what breaks. This should never be done in a way that affects customers, of course, but in a development environment, it’s quite appropriate to just change the back-end, and then update the front-end once it’s fixed. Defensive coding is a good practice, but overly defensive coding, where errors are not surfaced at all, can hide problems that may turn up later in unexpected ways.

Don’t bother refactoring something you just wrote. You’ll only make it worse. You have to wait until you don’t remember how it works before you look at it again, or else you won’t be able to appreciate the difficult bits. If you don’t have time for that, then make someone else do it, and tell them to be as harsh as possible.

Ego

As they say, No matter what the problem is, it’s always a people problem. The first and most important step in any refactoring effort is to detach our egos from our code. This is even more of an issue if something you wrote is being refactored by someone else, or worse, if you are refactoring someone else’s code.

I’ve worked hard to develop a pretty good understanding of software development. I’ve found that the technology side is easy—on the human side, I still have a lot to learn, and probably always will. Like a lot of geeks, I didn’t really get into the whole “social” thing in a serious way until I was almost an adult, and even today, I tend to focus on The Project and keep my distance from the rest of the world. So, I suppose I don’t know the best way to handle this part of the problem.

TBL’s classic recommendation to be loose in what you accept, and strict in what you send seems to apply well to this case. On a healthy team, everyone is committed to a successful product, and leaves their egos at home when they come to work. Code is passed around and everyone seeks as much input as possible. On a dysfunctional team, everyone’s sense of self-worth is tied up in the appraisal of their work by others—which has the twin effect of making them overly sensitive to criticism, and overly critical of their teammates.

It’s natural to put up resistance to the prospect of your code being modified. After all, you wrote it that way for a reason. Sit down with the other developer, and try to hash out the best approach. If arbitrary decisions have to be made, explain that the best approach is to make them consistently. Don’t focus on the problems in their code, but rather on the need for consistency and abstraction throughout the product.

If someone won’t listen to reason, frankly, the only solution I’ve ever found is to either leave or wait for them to. Hopefully they’re a contractor or something, so you can just wait, and do your best to isolate and minimize the damage as much as possible in the meantime. If not, then the issue should be escalated, if only because it’s polite to tell your manager that their ship is in danger of sinking.

Oh, right, that book…

I read Refactoring about 5 years ago, but I’ve been meaning to pick it up again, since I’ve gotten a fair amount of real-world development experience since then, and have seen my share of dismal failures. But this isn’t supposed to be a book review, so I won’t review it in any detail. It’s a good read, though, and I highly recommend it, even though it is mostly in Java, and seemed (to me at least) to simply highlight a lot of design problems inherent in the Java language that lead inevitably to bloated over-sized code. But that’s a post for another day.

In particular, I was struck by the idea of Mills and Brooks that software development could be done more effectively by a “surgical team.” That is, the experienced lead developer writes all or most of the code, and is supported by a team of people to make him more effective. While this is perhaps an effective way to capitalize on the significant difference between excellence and mediocrity in coders, it seems to me that most of the jobs on the team are today either (a) made obsolete by technological advancements, or (b) best done by specialists who float between teams. For example, Yahoo! has some of the best Javascript and CSS “language lawyers” in the world—but does every team really need one? The need for a full-time toolsmith has been all but completely surpassed by the customizability of workstations and the availability of many high-quality tools for coding and such.

The nature of software development has changed somewhat as well. We aren’t writing assembly code any more; not only do we have high-level languages, in my current project, there are no fewer than 9 different languages in use, 10 if you count regular expressions, which we all use quite frequently. [1] We aren’t all creating the same kind of stuff. Thus, there is a strong need for all of us to be experts, surgeons if you will, in our respective areas. In addition, the non-coding jobs require their own sort of specialized focus.

It occurred to me that this division of labor, where everyone depends on everyone else but everyone has a very different job, is more like a Dungeons and Dragons party than a surgical team. Each member plays a vital but remarkably different role, each owns what they do, and each is the team’s expert in what they do. As a webdev, I get consulted about front-end concerns, and am quick to ask a back-end engineer about things that touch their domain. UI questions are decided by the interaction designer; questions of product features are handled by the product manager, etc. I suspect that this is not unique to software for the web; a similar metaphor exists (or at least, could exist) for application software.

The best D&D parties are a bit formulaic, but that’s simply because a certain distribution of talents tends to be most adaptable to the most different kinds of problems.

Fighter

Not the flashiest or most prestigious role, Fighters have the lowest barrier to entry. They accumulate experience quickly, and don’t have a lot of class bonuses. (Until 3^rd Edition made them into action heroes, that is.) But their simplicity belies a focus and strength that is crucial. The fighter is the guy in the party that protects the more “thinky” types from anything that might get in their way. There can be honor and satisfaction in bashing orcs one at a time.

On a web development team, there are two archetypes that sometimes satisfy this role. The first is the diligent code monkey who might not be the most ambitious or inventive, but is good at getting things done. They are perfectly happy to churn through buglists, knocking down whatever they can. If you show them how to do something, they’ll make it work however it has to through sheer force of will—not always fast or elegant, but functional. When depended upon properly, code monkeys are a great and valuable asset.

Also in the fighter category are the producers who manage the content of the site, use the admin tools, troll for spammers, and do all the other busywork of web site maintenance. When the programmers have moved on to the next problem, they’re still working. Don’t forget them. Whenever possible, make their lives easier. They are the ones that keep your product shiny, so you can continue to be proud to have the URL sitting there on your resume.

Ranger

Rangers are good at surviving on their own. They shun convention and human society. Often a bit wild themselves, they feel more comfortable in the company of nature. Their chaotic bent can sometimes be a liability in an adventuring party, but their wide-ranging knowledge makes them hard to discount. They are fiercely independent and difficult (if not impossible) to tame.

In a programming team, this is the quintessential hacker. Most mashups are created by a single ranger acting alone. They are great at creating new things, and no browser quirk or network irregularity will stand in their way. On the other hand, they don’t always follow convention, and have a tendency to create overly clever code that is unintelligible to anyone who follows in their wake. Thus, they have to be watched closely lest they venture off.

I’ve learned a lot of great tricks and techniques from the Ranger programmers I know. I’m always a bit envious of their free approach to things, but I guess I’m just too timid to abandon convention completely. Make sure that you understand their code before you try to adapt it to your own ends—wild animals turn vicious when handled improperly.

Paladin

The paladin is the character class with the most stringent requirements. They must conform to the strictest of moral codes and demanding attribute requirements. As such, they’re rare, and respected. The paladin’s job is twofold. As a warrior subclass, he protects the party members from outside threats. However, he’s also a healer, and a leader. Charismatic and compassionate, he holds the party together by negotiating internal conflicts. Unshakably positive, he nonetheless keeps everyone going with firm conviction in the mission.

The best project managers aspire to paladin-like grace in their roles. They defend the team from the pressures of management and the monotony of GANT and PERT charts, and make sure that everyone is able to work at their top potential. When there’s a problem, they are approachable and always seek a win-win situation. By being consistently honest and righteous, they earn the respect and admiration of those both below and above them on the management totem pole.

When someone leads like a paladin, you want to get your tasks done and meet your deadlines. Not because you’re afraid of the consequences, but simply because their drive and attitude are contagious.

There are some managers that are decidedly un-paladin-like. Few things are as hazardous on a project as leadership that does not inspire confidence and commitment. Attitude is everything in a leader.

Rogue

The rogue is a highly adaptable character. They can pick locks and disarm traps, making them essential in any dungeon-faring party. Surviving in the streets and cities on his wit and stealth, a rogue comes to know the different factions and groups, and learns to navigate the urban waters. He knows how to make a buck, how to make a friend, and how to sell, and where to buy.

The product manager takes on this role for a programming team. They are often the ones who first envision the product and pitch it to management. Often the only member of a programming team with a business background [2], they settle questions of business goals and product requirements. They make the powerpoint presentations that eventually drive the development of a real software product.

The product manager, like a rogue, also sings the song of the party, talking them up when necessary and interacting with foreign powers [3]. If a product requires buy-in from another team, whether it’s a standards group in the same company or a potential outside partner, these are the people who make the calls and settle the deal. Their day is usually full of meetings.

I’ve known some developers who look down on product managers as being less technical. This is a big mistake. Their job is every bit as intellectually rigorous as ours, due to the huge amount of information that they need to be able to call forth on demand. Since they deal with features and requirements that have not even been fully envisioned, and might not exist in any form whatsoever in the market, they are even more in the world of pure thought-stuff than programmers. Make friends with your product manager, and you’ll find that they often do actually know their stuff. And after all, some rogues will stab you in the back if you cross them

Mage

The mage is the general purpose wizard. They are resourceful and possess a broad understanding of all types of magic. While not specialized in their studies, they are deeply theoretical, research oriented, and possess incredible focus and discipline. Their power comes from being able to summon a wide variety of spells, but they tend to be weak and vulnerable in the chaos of melee, since they must prepare spells ahead of time.

Many experienced back-end engineers fit into this role. They tend to prefer structured approaches and strongly typed languages, and produce exceptionally high-quality code. They often have advanced degrees in computer science and are knowledgeable in the intricacies of design patterns and memory management. They excel in areas that are somewhat regular and predictable, where the environment can be controlled, and the assumptions and edge-cases checked accurately.

Dedicated programming wizards are obviously important on any programming team. However, as much as “laymen” tend to see all software as a form of magic, the truth is that mages are actually pretty rare in the development world. They are the core of the “20%” of developers who truly care about what they create and diligently work to improve their abilities. Their craft is important to them, and they approach each problem with care and precision. It is much more than just a way to pay the bills.

While dedication is important, it comes at a price. Many back-end engineers come to get out of touch with the way that “normal” people think and interact with software. Most are pretty aware of this fact; I once knew an engineer who complained that bugzilla’s user interface was obviously designed by a database expert.

Specialist Wizard (illusionist, abjurer, conjurer, etc.)

These are the wizards that have selected a certain area of magic in which to specialize, at the expense of reduced ability in some other area. They have the same theoretical focus as a mage, but instead of having a broad understanding of every type of magic, they work to gain a much deeper understanding of a single area. When it comes to solving that sort of problem, they are the best—however, in other areas, they may be lacking.

There are many “specialist wizards” in the programming world. Here at Yahoo!, there are teams that focus 100% of their time on developing PHP, or MySQL, or Apache. (Rasmus comes to mind.) The “Paranoids” team sits in the cubes adjacent to ours, and they work to make sure that all the software at Yahoo! is as bulletproof as possible. The exceptional performance team does research to help us webdevs make our pages load and run faster.

When you have a hammer, the world can begins to look like a nail. A MySQL specialist will be tempted to suggest a database schema to solve any problem, even perhaps if the problem would be better solved by some other method. You may be asking for trouble if you ask them to do something outside their specialization. However, if you have a certain task that you’re certain to need, they can be invaluable. Database optimization is a highly developed science, and it’s not always easy or simple. Good programmers are rarely security experts (or even security knowledgeable!)

If you aren’t likely to have a lot of problems in their specialty, it’s often better to consult with a specialist only if and when they’re necessary. That way a single specialist can service multiple different teams.

Sorcerer

Sorcerers are similar to mages in the types of spells that they can cast, but instead of preparing spells from a spellbook, they simply select from the spells that they know, calling them when needed. As such, they tend to be much more spontaneous. However, they do not cultivate the discipline or depth of understanding of the wizards, and so they are limited in the number of spells that they can learn. The sorcerer excels in areas that are uncharted, and their spontaneous approach to magic allows them to quickly adapt to any situation, no matter how unpredictable.

The sorcerer is the perfect corollary to the webdev, aka the front-end engineer. Web browsers are a constantly moving target, with each behaving a little differently and new versions being released all the time. While programming patterns and the “right” ways of doing things can be useful, they are often too bulky for Javascript that has to be delivered, parsed, and executed on demand. Essentially, the webdev creates software in an environment where the rules often make no sense; where the code has to work on different platforms with conflicting requirements; where speed and features are both must-haves.

I’ve only recently heard of any colleges that make any serious attempt to teach courses on web development. From what I have heard, the things they teach are pretty bad—at best out of date, at worst outright harmful. Most professional web developers that I’ve met didn’t learn their craft in school. Like the sorcerer, they simply picked up their craft as they could, and learned techniques by studying “in the wild”; that is, by viewing source and reading websites and trying things out.

I’ve said before, it takes a certain kind of crazy to want to write software in the browser. You have to love the chaos and enjoy pulling out tricks and coming up with new hacks to work around impossibly ridiculous requirements. At the same time, unlike the ranger’s free-wheeling wildness, the team’s sorcerer must have enough discipline to tame the chaos and create software with whatever degree of stability he can, in order to help create a functional and stable site.

It can be a challenge at times to find stability and see the bigger picture and not write code for a single case. Yes, yes, that hack made it work in Safari. Lovely. But it’s also going to be a nightmare to maintain, or you just ruined the accessibility. Trade-offs are always present, and as far as I can tell, only experience and a lot of mistakes can teach you how to choose wisely.

Cleric

Diligent and committed to the ethos of their higher power, the cleric summons heavenly forces to do his bidding. By making himself an instrument of the divine, he can accomplish great things. A cleric’s power comes from their will and common sense (in D&D, this is the Wisdom score), as opposed to the cleverness and wit that Wizards rely on. The cleric has the power to vanquish undead through the force of their convictions, and their healing spells make them a must-have for any party.

In a programming party, this is the role of the QA tech. These noble souls must possess tenacity and diligence far beyond what is normal or sane. They must have the conviction and tenacity to insist that every test case be faithfully executed, even if it’s “impossible” that this-or-that change would have caused a regression. When they see something in the product that doesn’t look right, they must get to the bottom of it, carefully catalog the event, and enter it into the bug queue. They help banish the skeletons by exposing them to the light, no matter where they may try to hide.

Many a programmer, close to a deadline, has remarked with exasperation, Jeez, %$!@# QA keeps making me revisit this thing, I fixed that already!, only to find, when they go through the steps in the bug report, that yes, it is indeed broken.

While they may seem difficult or inefficient or outright stubborn at times, they’re vital. Their wisdom is a balance to the programmers’ wit, and they keep teams from burning up. Budget plenty of time for QA. Start it early. Get someone who knows what they’re doing, and use a proper bug tracking system. Your customers won’t thank you for it, but they’ll certainly come to hate you if you don’t.

Druid

Free from any affiliations or alignment, the druid transcends the simple human existence to become one with the natural world. Unlike the clerics who request their power from a deity, the druid opens themselves up to be nature’s instrument. It is an understatement to say that they know about the wild world; they understand the ways of the animals and plants intuitively and instinctively. Always seeking the way of balance and harmony and simplicity, they do not try to change or restrict the chaos of nature, nor do they rebel against the order that sometimes comes out of that chaos.

Designers are called by many names in our industry. UEDs, Visual Designers, User Interface Architects, HCI Specialists, Interaction Designers, IAs. (Indeed, these roles are somewhat different, as the task of designing the user interface has many facets.) Whatever they’re called, their job is to understand the expectations and habits of people they’ve never met, and shape the product into something that will fit the user’s mental model. At the same time, they must understand how those models change in the face of new problems and new solutions, and be able to adapt their software properly.

Like the druid, the UED must be free from any biases that may impede their ability to find the best solution for the user, even if that sometimes means that they may take issue with business objectives. (If the business goals and the user goals cannot both be met, then the product is in real trouble, anyhow!)

Part of the UED’s task is to map out the competition’s sites, to determine where this new product fits into the current ecosystem and how the target market’s expectations may have already been set. They must sometimes balance the “ideal” way of doing something with the “common” way of doing it, simply because users have been trained in a certain way.

More Similarities

A lot of the same characteristics that make up a good party are also true of a good programming team.

1. A team should have enough members to get the job done, and no more.
2. Each member should know his role, and enjoy doing it
3. Balance is key. Each part of the task should be accounted for, and no one area should have more resources than is needed
4. Druids make so-so healers; UEDs are usually pretty good at QA. A really good sorcerer might be able to handle the magic support if it’s not too intensive; most webdevs can usually throw some kind of back-end support together. But, if a certain area of the project is significant, then it’s worth having someone who knows it best.
5. Multiclassing can make a character more diverse, but they won’t increase in levels as quickly.
6. Treasure should be distributed fairly.
7. If you have more than one mage (or sorcerer, or cleric, or druid, or whatever), they should either divide responsibilities or one should be in charge. Same for the roles on a programming team.
8. It’s hard for a fighter to know how good a wizard is, or vice versa. It takes one to know one. Make sure your team helps interview new hires.
9. Raising levels and carefully assigning skill points will lead to a more powerful character. We min-max in real life, which is why we do it in RPGs.

Like most of the essays in The Mythical Man-Month, the fundamental principles of the surgical team are valid, even if the specific roles are mostly out of date. Within each class on the adventuring party, you can sometimes see a combination of surgeon and co-pilot; sometimes the co-pilot is unnecessary, or there are two surgeons who divide up the work.

The basic idea of thinking about the programming organization in a creative way is also worthwhile, even if only as a mental exercise. I’m sure that there are other metaphors that would work equally well.

In general, I think that the motley crew of adventurers is a lot more fun than the surgical team.

In my last post, I touched on a method to get different browsers to handle the inline-block display style. I decided to use that on a project that I’m working on now that has a few modal dialogs.

Feel free to skip all this and go straight to the example.

In the application world, creating a modal is pretty straightforward. Here are some features that every modal window should have:

1. Interaction with the contents of the parent window should be *impossible* until the modal is dealt with. Scroll-mouse should not scroll the background page, clicking should not click the background page, tabbing should not get you there, etc.
2. When you dismiss the modal, the parent window should be *exactly* how you left it.
3. The parent window should be dimmed, or there should be some other indicator that it is currently not available for interaction. This has the corollary effect of making the modal “pop” a bit more.
4. In a web page, a modal should be constrained within the viewport. Opening a new window just to show a dialog box is ugly as a bucket of slugs.
5. The modal should be placed consistently. If it is not movable, then it should be centered vertically and horizontally. This positioning should be consistent if the parent window is resized or moved.
6. The modal should be smaller than the parent window. If the viewport is resized smaller than the modal, then scrollbars should appear that allow the user to see the rest of the modal.

The rule of thumb: The modal should be visible always and in every situation, the parent should be dimmed and preserved. Almost every modal dialog I could find online broke a few of these rules.

In some, the html and body elements have their overflow set to hidden when the modal is shown, so the current scroll position of those elements is lost. If you were looking through a list of items, and you brought up a modal dialog related to one of them, then you’re back to the start of the list when the modal is dismissed. Annoying.

In others, you CAN scroll the background, but the modal is set to respond to the onscroll event to try to stay in place. Some of the time, they use display:fixed to behave a little nicer in non-IE browsers. At worse, you get a jumpy seizure-inducing experience; at best, you accidentally scroll the background and lose your place without realizing it.

It’s a bit of an edge case, but you do have to handle cases where the modal might be taller/wider than the viewport. As a test, I tried resizing the viewport down on a bunch of modal approaches. Vertical centering often doesn’t do too well unless the contents are smaller than the viewport, so the top and bottom of the modal were cut off. Unacceptable.

It wasn’t terribly difficult to create one that obeyed these rules. Here’s the approach that I took.

1. Build it with tables.
2. Replace the tables with divs, and give them the appropriate display:table, display:table-cell CSS rules.
3. Fret and hack to try to get it to work in IE, unsuccessfully.
4. IM another yahoo who has a reputation for being a great webdev hacker.
5. Adapt what he did to my needs.

For the non-IE browsers, it’s pretty straightforward. Take the scrollbars off of the HTML and BODY elements, and put them on #body instead. #modal is a sibling of #body, so when it is 100% height and width, it will cover it up. Then #modal will overlap the scrollbars of #body, without losing the scroll position. Also, that means there’s no need to do any trickery with a negative right margin to try (unsuccessfully, on the mac) to overlap the scrollbars.

Since .overlay element in the modal has a display:table, it will always expand to fit its contents. .overlay’s parent is overflow:auto, so if .overlay is too big, it’ll let you scroll to see the rest.

What this means is that you can’t cut off the top and bottom of the modal. It’ll be vertically centered if there’s room, and if not, it’ll be flush to the top and scrollable. Perfect.

For IE it gets trickier. Hedger’s example showed 2 great ideas, though. Basically, if you have 2 siblings that are display:inline-block, then, in IE at least, the line-block will expand to fit both of them. So, they both end up being vertically centered to the height of the taller child. If that taller element is width:0, then you essentially have an arbitrary line-height setter for the inline-block row. In this case, the height is set to 100%.

The other child will be whatever height it is based on its contents. If that happens to be taller than 100%, then the line-height will be taller than the viewport. At that point, the overflow:auto parent shows a scrollbar to let you see the rest. Identical behavior to the standards browsers!

The second problem arises with IE’s peculiarities when it comes to percentage heights. For whatever reason, if you have a position:absolute element, it doesn’t like to do height:100% unless some parent has a fixed height value, like 500px. 100% of 100% doesn’t compute, for some reason. Sadly, that means that the element that does the overlay wouldn’t be able to properly do its job in IE. So, instead of using a separate transparent layer with a background image, Hedger used the DXImageTransform.Microsoft.gradient filter on the modal container itself. However, since the gradient filter doesn’t actually prevent interaction with the background, I added a 1 pixel transparent gif as the background image.

I’d very much like to not use this transparent pixel. It feels a bit like a spacer gif to me. But, all in all, it’s a pretty small price to pay.

I didn’t bother to tackle the tab-order focus stuff in this little test page, since I wanted to just isolate the stylistic stuff to get it working across browsers. However, it would be good to put a focus handler on #body, and any time a #body element gets focus, focus the #modal instead. That way, once you tab off the modal dialog, you’d get right back onto it. In the spirit of being friendly to keyboard users, it would also be a good idea to keep track of the currently focused element when the modal opens, and go back to it once the dialog closes.

Here it is for your view-source pleasure.

Please please feel free use this modal. Life is too short to futz around trying to see the “cancel” button that is hiding off the screen so you can go back to reading your page.

Let me know if you can think of any enhancements or find any problems with it. Those of you in the US, I hope you had a great Thanksgiving.

Update

I updated it to add support for focus and blur events. It works in Firefox, MSIE, and Safari. Opera on the Mac has some really strange behavior regarding focusing on links, and I decided that it just wasn’t salvageable. Safari requires a bit of hacking to get the focus stuff to work properly, but I think it’s acceptable.

Inline-block layout solves a lot of problems. It lets you do some cool stuff previously thought impossible with CSS. It makes vertical alignment work properly. And sadly, it’s supported pretty badly.

Mozilla doesn’t support inline-block at all, but they have -moz-inline-stack which is about the same. Fair enough, since no one else understands -moz-inline-block, you can just do this:

display:-moz-inline-stack;
display:inline-block;

If you put inline-block after -moz-inline-stack, then Moz will start using the “right” one when it supports it.

IE supports inline-block, but only for elements that are natively inline. So, if you really want to use inline-block, you’re restricted to spans and strongs and ems, when a list or paragraph would perhaps make more semantic sense (and also degrade more nicely for non-CSS users.)

However, if you trigger hasLayout on a block element, and then set it to display:inline, it magically becomes an inline-block in IE! By using the *property hack (which I love so well), you can hide the display:inline from all non-IE browsers effortlessly.

Here’s the code, in all its brief loveliness:

display:-moz-inline-stack;
display:inline-block;
zoom:1;
*display:inline;

From there, it pays to learn a thing or two about the vertical-align property. It lets you do lovely things like this.

Benefits

Inline block elements can be vertically centered like display:table-cell, but they wrap when they get to the end of their parent. Also, it’s supported across browsers using this hack, whereas display:table-cell is not supported in IE.

This technique allows for some very interesting layout approaches that would have required a lot of very tricky use of floats previously.

Caveats

If an element is inside of an inline block, and lies outside the line box, then it won’t be clickable in Mozilla. Give the child element position:relative to correct the problem.

Elements treated this way will have hasLayout set in MSIE. This is a weird and esoteric aspect of MSIE’s CSS engine that has potentially unforeseen consequences. Beware.

If an IMG element is directly inside an inline block element in Mozilla, it will stretch to the full width of that element. Wrap all IMG tags in a block-level container element to avoid the problem.

Since inline block elements wrap and flow like inline content, that means that they also respect white space like words on a page would. That is, if there is *any* whitespace between two inline-block elements, then a single space will be added between two inline-block elements. If this causes a problem, you can either remove the whitespace or comment it out like so:

</div><!--
--><div>

All in all, the caveats are pretty easy to work around, and the benefits allow for some really cool stuff that would be almost impossible or very difficult otherwise.

The 2 Faces of vertical-align

In a classic CSS blunder, vertical-align can mean 2 extremely different things, depending on whether an element is display:inline-block or display:table-cell.

table-cell

Align the element’s contents according to the element’s vertical-align property. IE, if the cell’s vertical-align is set to “middle”, then vertically center the element’s contents. The height and position of the element itself is determined by the containing display:table element.

inline-block

Align the element according to the element’s vertical-align property. IE, if the inline-block’s vertical-align is set to “middle”, then the element is vertically centered in the line-block. The height and position of the element’s contents are determined by the standard block-level flow rules.

While I personally believe that this was a stupendously bad and confusing approach to take, I believe that the reasoning comes from backwards compatibility. Inline blocks emulate the behavior of the IMG tag, and the vertical-align CSS property thus mimics the old valign attribute. Table cells emulate the behavior of the TD tag, and the vertical-align CSS property thus mimics the behavior of the valign attribute on TDs. In other words, in this way, CSS faithfully reproduces the sloppy errors of HTML. It would have been better to use two different properties to achieve this effect; after all, vertical-align:baseline hardly makes sense for table cells. Perhaps the inline-block type of vertical alignment would have been better called “line-position” or some such, since it is less like a vertical version of the text-align property.

But, you write code with the language you have, not the language you wish you had, and CSS is what it is, at least for the foreseeable future.

Recently, I had to achieve an effect that was extremely tricky by standard methods, but extremely easy using tables. I decided to test out a display:table approach, and then try to hack it into place for IE, since it is the only browser that does not support this approach.

The result uses a fairly large number of DIVs, but still fewer than the straight table approach, and without the semantic rubble of tables. The dialog is vertically and horizontally centered, but if you resize the viewport too small, the dialog will not be hidden permanently, due to the “collapse to fit” nature of the table display style. Doing this will an inline-block would have been quite a bit more difficult.

For IE, I used the 50/50 hack. Create a position:absolute element at top:50%. Then, create a position:relative child at top:-50%. The negative top rule on the position:relative element will be misinterpreted, and result in a vertically centered box. The downside is that IE gets a scrollbar if the viewport is less than twice the height of the dialog. But, that’s a pretty acceptable down side, in my opinion.

Sadly, unlike with display:inline-block, it doesn’t look like there’s any real consistency to support display:table across browsers. (Except, of course, using table tags.) You basically just have to hack something for IE that achieves the same effect, and which approach you use varies on the effect you’re going for. In this case, I exploited an IE positioning bug to achieve vertical centering, but other situations would require different approaches. If you’re doing complex layouts using display:table, which in a perfect world would indeed be a great way to do it, you’re going to have a lot of work cut out for you hacking away at IE.

Let’s pray that IE 8 supports display:table!