This post has a great little droplet that can be added to the Finder toolbar to open either the selected items or the current Finder folder in TextMate.
Sadly, it doesn’t work in Tiger, and I haven’t made the jump to Leopard yet. If you’re a TextMate user who’s still on Tiger, and wants this feature, I tweaked Henrik’s AppleScript to play nice with Tiger.
Enjoy!
It’s a very interesting time for me at Yahoo! right now. I’d been keeping quiet about what’s going on for two big reasons. First, I’ve been too busy working to write about it. Second, it’s secret.
Well, as of 2008 March 25, it was a secret. This press release broke the news a few days ago, so I doubt I’m giving anything away to say that Yahoo! is busy working on implementing a framework for OpenSocial applications that tie into various sorts of Yahoo! data.
Buzz was a very fun project to work on. After games’ mountain of legacy and the brand universe fiascos, it was really great to be able to have a hand in developing a site from scratch and also have enough space and resources to really create quality. (It has its problems, but they’re being worked out, and overall I’m quite proud of the site we built.)
However, the really interesting parts of Buzz are all behind the scenes. The front end was an exercise in the complicated and delicate art of engineering simplicity and eliminating redundant effort. Even in the face of exacting requirements and changing specifications, we ended up releasing a site that performs well and is relatively easy to work on. On some level, I feel like it was my karma baby for some of the complicated monstrosities that I’ve left in my wake as a developer.
But, with Buzz behind me, creating and maintaining yet another website has gotten a bit old. Nothing wrong with that work, but I’m done with it for now. Time to do something else.
Douglas Crockford has called mash-ups One of the most exciting features of WEB 2.x, and I couldn’t agree more. The vision of the web from the get-go was a distributed platform for innovation. As client technology and usage advanced, dynamic features followed naturally. The ability to mix-and-match data and functionality into new applications is a very exciting prospect.
However, like so much potential innovation on the web, this effort is severely hampered by the same system that makes it possible. The browser Dom was designed by committee, and the committee didn’t actually meet for a while, and the members of the committee were trying to kill one another. Throughout the browser wars, IE and Netscape introduced features in a frantic arms race to try to shut one another out of the browser market. Many of those features were not really ready for prime time, and many others probably should have been wiped off the whiteboard and never made it into the code.
On the plus side, browsers and web pages were able to innovate much faster than the w3c’s crawl. Much of the time, the “official” specs only included a feature because a few browsers had already implemented it and it was in widespread use. That speed got us to where we are, but lots of important things were sacrificed or forgotten along the way. Among the missing: a proper permission model. Same-domain restriction works to some degree, but it makes mashups trickier and there are still big holes. In particular:
Sadly, this means that mash-ups are faced with the choice of:
Essentially, mash-ups are thus restricted from really doing anything interesting. PayPal or Amazon or Yahoo—-that is, sites with money and interesting private data—-could never risk exposing their data to a mash-up. Their visibility, and the value of the private data they hold, makes them prime targets for attack. And, since they have money, and visibility, they’re also prime targets for lawsuits if a user was damaged by a mistake.
OpenSocial represents a concerted effort by some very big players on the web to make severely interesting and useful mash-ups not just a reality, but common and relatively easy to create and host. API layers will sit between the developer and the browser, and between the developer and the container page, allowing them to code as if it’s 2099 and the browser is sane. It’s no small task. When Hao asked me to join the team, and told me what they were doing, my initial reaction was Well, it might end up a total failure. But it’ll be one hell of a fun problem to work on. I’m in.
There were hints and rumors that Yahoo was doing something with OpenSocial a while back. (We were.) But, the main reaction to the press release really should be Well, spluh.
Yahoo’s all about social; it may have started with indexing the web into searchable categories, but some of its earliest and most popular applications are mail, instant messaging, and groups. Yahoo’s all about open; we invest countless developer-hours into the open source technology that we use. Yahoo has a lot to gain from a more exciting social internet; when the ocean rises, the bigger fish benefit the most.
For a month or so, I’ve been transitioning off the Buzz team and onto the team that’s doing the OpenSocial container stuff. As people found out that I was joining that team, I heard this line a lot from various people: Oh, thank god. They really need a good front-end.
There’s a very flattering element to that, and I’m not trying to brag. Objectively speaking, I know I’m good at what I do. I love doing it, and I work hard at it, and that’s just what happens when you work hard at what you love. No magic at all.
At the risk of sounding like a falsely modest blow hard, though, on a more moment-to-moment basis in my own world, I focus on many of my own failings that others rarely see, and I am always very impressed when someone solves a problem that I was stuck on. I often feel like I’m still new at Yahoo, and on some very insecure level, I’m waiting for them to find out that I’m just playing at this stuff and tell me I’ve fooled the real adults for long enough, and it’s time to go home.
But I am good at what I do. It’s real. These skills are rare, and they’re worth money. When others acknowledge it, though, I kind of feel embarrassed and get all “aw, shucks, twern’t nuttin’” and find myself unable to make eye contact. (A few times, this reaction has evoked a No, seriously! I mean that!
which just adds to the effect.)
So, the flip side of that comment… yes yes, I’m a great webdev, blah blah, but say that first part again? It’s a bit like hearing, Oh, thank god they picked you to wrestle that giant raptor. He’s been killing one guy after the next, so I’m really glad that they finally picked someone who can take him down.
Why couldn’t it have been That’s great! There’s a lot of great talent on that team, I’m sure you’ll fit right in!
Well, that’s the cards that life deals you sometimes. Impossible problems are the most fun, anyway.
And I’ve found that there really are a lot of talented developers on this team. Some more have joined it recently. I’m really happy to be able to be working in this group.
Somehow, I managed to work myself into a role on this team where most of my time is spent in meetings. Some of them were soul-sapping and boring, as meetings can be sometimes, but lately, they’ve seemed to focus more and more on solving front-end problems at an architectural level. After almost every one, someone whips out a phone to take a picture of the whiteboard, lest our valuable work be lost to the eraser.
Additionally, I’ve been attending the YUI team meetings, as they’re interested in knowing what they can do to aid Yahoo’s big bets. Through that, and by virtue of working on a product that promises to push the limits of what the browser can safely do, I’m finding myself consulted because Microsoft is asking Yahoo what we think of the IE 8 beta, and what we’d like to see in it. I’ve been asked to teach a CSS class for the next round of incoming junior/trainee webdevs (Jukus, as they’re called). I’m meeting with other groups and helping to figure out what OpenSocial means for their product’s context, and how they’ll use what we’re building. Today, we’ve got a meeting at the Googleplex.
It certainly looks like success. And the pay is pretty good. The product is fun and open and creative. It’s one puzzle after another.
But, I’m feeling a bit envious of the other developers I talk to who get more time to write code. From Tuesday to Thursday of last week, iCal was booked solid from 11-5 every day, I ended up in ad hoc discussions about things (or dealing with email) until about 7 or 8 each night.
I have tried hard not to become a manager at Yahoo. Perhaps a management path is in my future somewhere, but right now, it just seems like a job I would not enjoy. Programming is a very social activity if you’re doing it right. With a new team and a new product, there’s plenty of architecture and designs to figure out, and I’ll be coding again soon enough.
I hope.
We have openings for great engineers and webdevs. If any of this sounds like something you’d be interested in, let me know. I’m 100% serious. Javascript and PHP experts, we need you. Yahoo is, in my opinion, one of the best places for talented engineers to work. They generally treat their employees very well, and this is a product that is just full of fascinating puzzles.
It’s been said that the best startups take a popular Unix command and bring it to the web. But there are a few that are poorly represented. I understand that I may be making a bad career move by discussing this openly on a blog, but quite honestly, my desire as a consumer for a satisfying product is enough to risk—-nay, hope—-that someone else makes a million dollars doing this before I get a chance to.
I’m thinking specifically chpass, finger, and pw.
I know what you’re thinking. There have been a few forays into this arena. MySpace, Facebook, and Plaxo come to mind, not to mention whatever else some MBA has stuck to a “social network” this week. (It’s just like a regular duck, but this one swims around the lake and lets you put all your friends in a list! I’ll be rich!
The sad thing isn’t that he thinks it; the sad thing is that he just might be right.)
The social-for-the-sake-of-being-social sites tragically miss the point. I have resisted getting a Facebook account for a few years now, and even deleted my MySpace account. They seemed to require a lot of time and effort doing basically nothing, and didn’t give me what I really want.
I hate managing contact lists. The worst contact list, the one that is the hardest to manage, is the one in my head. Every day I get older, I get worse at remembering phone numbers, and I like to know who’s calling me. I like to see your picture when you call, see your real name when you email. I want my email program or my phone to know who you are when I start to type the three letters of your name that I can remember off-hand, even (especially!) if you’re someone I don’t talk to often.
That’s why I shelled out $40 for Missing Sync so that my phone and computer can share an address book. I have an Applescript program sitting on my back burner that will sync any contacts I add in Adium into this same collection, and even look up their contact details from Yahoo’s corporate intranet (since most of the time, they’re work mates.) Automated replication is still not great, but it eases the pain of managing multiple lists.
Facebook and MySpace are software platforms designed around the premise that managing a contact list is fun. And it can be in that 12-23 age range where we attempt to define ourselves and carve out our place in the world through our social connections. That’s a key demographic for advertisers. Good for you, Facebook. But if I wanted to spend all this time managing my friendships, I’d have more of them in real life. Ooh, burn! i mean… hey, wait a second..
Plaxo is actually a pretty good approximation of what I’d like to see, at least on the “managing contacts” side of things. Granted, I’ve been spamvited to their service by half a dozen people I hardly know, which is a classic example of “let’s be viral” gone horribly horribly wrong. But their product offering is pretty close. You get one contact list online, and it syncs with other areas. It’s unfocused since they’ve added “Pulse” (basically an RSS aggregator for your other web profiles), but still pretty good.
However, even Plaxo misses a key point, and makes several fatal flaws. I’m actually talking about a profile and contact management system that is much grander.
In a relational database or data map, the idea is to keep a piece of data in only one place, and store the relationships between entities rather than making multiple copies. Most contact management systems, from a little black book to the cell phone contact list to Outlook to Plaxo, fail to implement this simple principle. Instead of making each node in the network keep track of all the data about all the other nodes in which it is interested, instead let each node control its own data, and store links to the nodes in which it is interested.
In the old days of land lines, the phone book was enough. If you knew someone’s name and city, you could get their phone number and, perhaps, their street address by performing a simple lookup. Each user had the option to control how much information was shared with the public. Until the autodialer came to telemarketing, the abuse rate was limited by the cost of using the technology.
Today, each person has many more pieces of contact information, and the cost of abuse is virtually zero. There is no way in hell that I’d let anyone publish my actual cell phone number, and once an email address is exposed, it’s basically useless. Spam fighting is an arm’s race, and an unfair one even for Google and Yahoo to fight.
Back to my original point: chpass, finger, and pw.
add or change user database information
In other words, manage my info in one place.
The finger utility displays information about the system users.
In other words, look up the information that other have made available.
create, remove, modify & display system users and groups
In other words, specify who has permission to what.
Many large companies have some sort of online system like this. At Yahoo, it’s the almighty Backyard, the corp website that started as a list of email addresses and grew into a full-scale intranet with contact lookup and LDAP access. (It also features conference room booking and documentation searching and plenty of other handy things. But mostly, it’s still all about the mega employee contact list.)
You manage your own profile, and make sure that your numbers and whatnot are up to date. No one else ever has to worry about how to contact you, because it’s all in one place. However, that only works because access to the backyard system is tightly limited to current employees, so abusing the system would entail serious consequences for the abuser’s reputation (and career).
In other words, we have finger and chpass, but pw is being done manually by the HR department, which limits the possible size of the network considerably.
The easier it is to use a networked contact management system, the easier it is to abuse. The more useful it is for you and your friends and associates, the more useful it is for spammers and scammers. Already, we have to keep our email addresses hidden from strangers. Imagine how much worse it would be if a PPC pusher could just e-finger “isaac.schlueter” and have my home address, email, phone number, instant messenger alias, birthday, and photo. Yes this is exactly the sort of information that I’d like to easily share with everyone else.
Everything that has been done so far in the area of email spam, while impressive and necessary, is fundamentally inadequate. As long as it remains profitable for a spammer to send out 100 billion emails every day, it will happen. Any attempts to prevent or avoid this behavior run counter to the incentives of the market; which is to say that it’s a bit like building a dam of sand and expecting to stop the Mississippi. Won’t happen. A bigger dam will take longer, but eventually, they’ll all crumble.
In order to truly divert human behavior, the incentive must be dealt with at the source. Direct attacks against the offenders (ie, shutting down their accounts) are not effective in the long run (they just get new accounts.) Negative incentives, such as putting spammers in jail, are not going to be effective in the long run, because it doesn’t push the cost of spamming up high enough. John Q. Spammer doesn’t think he’ll be the one to get caught, and he’s probably right.
I don’t claim to have this bit of the system figured out, not by a long shot. But I have a few ideas.
In real life, we meet a lot of people, and many of them can and do annoy us by contacting us in unwanted, if mostly harmless, ways. The foul smelling man who stops babbling for a second to ask me for a quarter. The smiling woman who shoves a tract at me and tells me I’m going to hell. Sadly, the list goes on and on and on.
However:
There are still, of course, the violent offender and the con man. However, in real life, direct attacks incur a high degree of risk, due to the chances of being caught or retaliated against, and so law enforcement has a relatively easy time keeping serious criminals in check. And those looking to do you harm by gaining confidence and taking advantage of it, well, they’ll always be around, but they’re pretty rare and the reputational aspects keep them somewhat in check as well.
Entrance into this global open personnel file would require that an account be tied to a single real person, who doesn’t have any other account in the system. Accounts that are not “backed” by some kind of reliable identity are only given some kind of limited provisional access—-perhaps they can email a user through the system, but they cannot get the user’s “real” email address, and users would be able to deny access altogether to unidentified strangers if they chose.
Identification is itself a non-trivial task requiring a high degree of trust from the web site. Even if you know it’s 100% secure, being asked for your date of birth, SSN or passport, and a major credit card is a tall order. Without biometrics, it must come down to discrete bits of information at some point, which can be (and often are) faked.
A rinky-dink fly-by-night startup can’t hope to achieve this level of trust quickly. And, without getting a critical mass of users, the value proposition to new users is a lot tougher.
The company to build this system would need:
…then it may as well not be anywhere. The goal of this system would be to revolutionize contact management the same way that email and hypertext revolutionized written communication.
Just as email works in any email client, and web pages work can be viewed by any browser, the APIs provided by this system would have to be completely open. Any application must be allowed to interact with them, both to change data and fetch it.
In order for it to work, and really have the effect that I’m talking about, there must be absolutely no lock-in, no up-sell, and reasonably liberal rate limits.
How does something like this make money? That’s an open question, and a big one, probably part of the reason why I’m still pushing bits in a day job and not out getting VC to build this thing. I also happen to really like what I do at that day job.
Maybe it would have to be something built under the Apache foundation or some other OSS group, and sponsored by donations of capital and resources from some major players in the online social arenas. Maybe there’s some clever way in which smaller users and early adopters get the API for free, but then charge everyone else.
OpenID is a great start, but what we really need is an open profile and open contact list system, and OpenID doesn’t provide that.
Google’s Open Social is an interesting product, but the more I read about it, the more I think it’s not quite low-level enough to really deliver on what I’d like to see here. While it promises to expose social data to third-party applications in an API that could be consistent across social websites, it doesn’t fully address the issue of being able to manage contact data in a distributed way.
As I said above, the company to do this will need:
Yahoo has all four of these, but that whole China escapade has damaged Yahoo’s reputation in the eyes of many users. Of any company on the web, however, Yahoo has perhaps the most to gain from such a system and a lot of resources and domain knowledge to throw at the problem.
Even if they only share user information when presented with a subpoena, that means that using this system exposes my information to governmental intrusion, which is deeply problematic. In order to be truly trustworthy, a stronger commitment to protecting privacy needs to be in place than just words on a corporate press release. The officers of the company to provide this service should enter into a binding agreement that they will not knowingly expose user information, even in the face of governmental pressure.
Like I said, I don’t have all the answers to this product. But I know that, as a user, I’d be absolutely delighted to see something like this take hold.
I’m a little behind in posting this, since it’s been live and in the media for a while now. Go check it out: http://buzz.yahoo.com
It took a lot of work to pull this thing together. Monday night was the craziest month of my web development career, culminating in this tweet posted as I finished pushing the last packages out into production. I managed to ignore the page that came in 5 hours later. I wonder if it was some kind of subconscious self-preservation that made me leave my phone on vibrate when I finally went to bed. Sorry bout that, Trang.
And despite the rather disappointing leak, it seems to have been pretty well received.
We worked really hard on this project, and while it’s definitely still a “beta”, the feature set is enough to be interesting, in my opinion. All in all, this has been a highly rewarding product to be a part of, and I’m extremely proud to have been a part of the team that built it.
I’m in the process of moving onto a different team, and I have mixed feelings about the transition. On the one hand, the team I’m joining is doing some truly ambitious and interesting things, and it’s more in the direction that I want to go. I’ve been starting to get tired of building one web site after another, and this will be an opportunity to do something much more in the “web application” direction. On the other hand, the Orion team is one of the best groups of developers I’ve ever worked with, and it’s sad to leave them.
But that’s the joy of working at a company like Yahoo—if you get bored of doing one thing, then there’s always something else, and no matter which team you’re on, there are lots of top-notch developers among your coworkers.
So, it started out as just a few font changes. I’d been growing less and less pleased with Trebuchet MS, and had found a few cases where it broke the militant line-height rules I’d set for this site. In general, I felt that it was too crowded, and the gray hash textured background was (in my opinion) a contrived and artless approach.
I’m not a very understanding end-user when it comes to this site. The Foo Hack design team got fed up with my complaints, and decided to give in and make some changes.
Feature creep is a bitch when you have a deadline. For personal projects, there is no deadline, and so a bit of feature creep is fun, within reason. That’s a big part of the reason why I have so many back-burner projects that are almost finished; just as I near the finish line, I decide that it needs a little more, or another project gets more interesting for a while. So, on to what changed.
I was a computer geek first, but when my dad replaced our DOS home computer with a Windows 3.1 box, I fell in love with fonts. I can waste an afternoon surfing the web downloading fonts that I’ll probably rarely (if ever) use. Web design is about 90% typography, and it is an art that blends strict perfectionism and fluid acceptance in a beautiful way.
For the main body text, I wanted something subtle and easy on the eyes. Something that would let the writing speak in the voice that I intend, without getting in the way. For the “small” items of diminished importance (asides and things in <small> tags) I wanted a serif font with a very thin ex measurement, so that the text could “feel” smaller without altering the line-height. For the headings, I wanted a rounder, more open font that could evoke a sense of boldness without being too heavy. Subheadings and meta items (the date, “comment” link, etc.) should use the same font, but with less letter space and a subtler color.
For the body text, I decided to go with Helvetica, the classier old-world cousin of Arial. I was initially pleased with the results.
Cliché though it may be, Times New Roman was the perfect choice for the diminished items. It’s very light, yet still fairly readable, and when italicized, it practically disappears. I was already using it for that reason, but since I don’t love TNR all that much, I was willing to entertain other possibilities.
A problem came up when I noticed that the line-height was getting messed up whenever a TNR inline element would wrap to the next line. I figured out the problem.
Let’s say you have font A and font B. You create a block-level element using font A. Then, you have an inline element using font B, which wraps to the next line.
If A and B don’t put their letters on exactly the same point in the line-block, then the line-height will adjust up or down, as the next line is set by the position of the letters in B instead of the position of letters in A. Since the B element isn’t a block-level element, it won’t create a whole new block, and will have the effect of adding or removing a few pixels from the overall height of the A element.
To correct this problem, and still have 2 fonts sharing the same block-level element, you need to find two fonts that have exactly the same vertical letter placement on the line block.
I’m not exaggerating when I say that I created a test page and exhaustively tested every combination of serif and sans-serif fonts on my Mac. It took about a week. There was only one combination that worked.
Arial and Times New Roman.
Ah, compromise, my old nemesis. You strike again. Not willing to give up the line-height strictness, I gave in and decided to use Arial instead of Helvetica. On the bright side, Windows users were mostly going to see Arial anyway, and they’re both tried and true web-safe fonts. And, at typical screen resolutions, it’s hard to tell the difference anyway. But still, it’s a bit of a sore point.
Gotham, the masterpiece homage to urban signage by Hoefler & Frere-Jones, is quite possibly the best font ever devised. It’s open and confident even when it’s not bold, balanced and practical. I went with this as the headings, and it also was the best candidate for the smaller informational bits. (After all, it seemed fitting to put navigational links in a font designed for signage.
The downside of such a perfect font is that it’s not free. It would be fantastic if Apple or Microsoft were to license Gotham for their respective operating systems, but I don’t see that happening any time soon. Those that don’t want to install Gotham will see Century Gothic (if it’s installed, most likely because they got it bundled with Microsoft Office), or something in the Lucida family.
I brought down the contrast a bit, and did away with the background graphic. Some color was sprinkled around to help create a meaningful mental model of each page. I got the idea for the categories above the titles from Rands in Repose, one of my favorite regular reads.
The default set of post meta info that Wordpress provides is far more than necessary. I got rid of everything that didn’t directly benefit the goals of conversing with the world through this blog.
It seems awkward that my comments should have a blue left border, and reader comments are unadorned. Simply using random colors wouldn’t do. So, I wrote a function that will hash a string into a color value. A simpler hash would have sufficed, but I wanted more control over the range of colors that it selects, and rand() was a good fit. So, your comment will always have the same color (unless you use a different email address, of course) and all the colors will be in a particular mid-range of luminosity that is bold, but not overpowering.
I’ve already run into a few situations on this blog where I felt that threaded comments would have been helpful. However, the threaded forum-style comments would be complex and counterproductive. I grabbed a standard threaded-comment plugin, and tweaked it to replace the interface with a few hyperlinks. Since conversations are more naturally many-to-many, I’d like to implement something along the lines of what Dunstan Orchard used to have on his blog, but doing that the right way means a bit more investment, and this project was dangerously close to being back-burnered forever.
Very soon now, the project that has consumed my days for the last 6 months will go into public beta. It’s too early to pop the champagne, and that’s not what this post is about. However, the newness of this project is worth pointing out for the following reason: We’re already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs.
That’s right. We haven’t released anything, and there’s already enough bloat to justify spending at least 2 weeks cleaning it up.
Of course, due to the pressures of market, and the morale effects of delaying the release, we’re buttoning up what we have now as much as possible, and planning to work on the refactor once the beta is out there. It may actually help a lot, since the gigantic firehose of traffic that Yahoo! points at sites has a way of helping even the most egotistical of developers realize that their clever baby needs some serious surgery.
This post is going somewhere, I swear, but I can already tell it’s going to take me a while to get there. Go get some coffee and a snack and come back. I’ll wait.
In my opinion, the core of good software development methodology comes down to two basic questions that should be asked daily:
I call this the Go Fast method. The goal is to efficiently create software with minimum long-term cost and maximum long-term benefit, without driving away your best talent. (Have I got the attention of the business and product people in the audience? Good. Please stay for the rest of the show.)
I know that someone out there is bound to point out the obvious parallels between my “go fast” methodology and Agile, and maybe they’re right. To the extent that Agile accepts these premises as important, it is correct. But I’m claiming that everything else stands or falls based on how it affects these questions. Stand-ups, planning meetings, org charts, IDEs, frameworks; they’re all only as good as their effect on the principles of speed and stability in development.
If you’re developing software properly, you should be creating software that does something as soon as possible, and keep it up as much of the time as you can. Until there’s something to look at, you’re just spinning your wheels. Nowhere is this more true than on the web. Photoshop comps don’t break in different browsers. Product presentation decks don’t “feel klunky” when pieces load out of order. Grand visions promise to solve all your problems and make you biscuits for breakfast, but they don’t deliver.
You make software for a living? So make software, already. You wanna build a website? What’s stopping you? You don’t have designs yet? Build the parts that you do have. Always be releasing, even if your actual release date is months away.
Build the core, and grow the other pieces up around it. Every minute spent waiting, instead of actually building something, is worse than wasted. When you live in the code day by day, it becomes a part of you. You learn how the pieces work, and the knowledge is automatic. When you get too far from it, it becomes foreign, like driving a car for the first time.
The agilistas call this “shippable code.” In my opinion, that’s a horrible term. The vast majority of code that I’ve seen released into the wild was not even close to what I’d call “shippable,” but it got shipped just the same. Most of the time, the prototypes leading up to the release were significantly worse. Some agilistas are quick to point out that “shippable” doesn’t always really mean “that you would actually ship it.” Sorry, but drivable means that you can drive it, and drinkable means that you can drink it, and flyable means that you can fly it, so if “shippable” doesn’t mean “you can ship it,” then they picked a bad term.
I suspect the ambiguity is intentional on some level. I think the claim that “Agile makes software faster” is a myth. Agile methodologies, used in moderation by a well-balanced team of dedicated and talented individuals, do tend to result in higher quality software, greater management visibility, and a happier bunch of people than any alternative, but that means that the overall development time can be significantly longer. And if you’re not on a well-balanced team of dedicated and talented individuals, then no methodology is going to save that sinking ship, so dust off the CV and bail out ASAP. But I digress, and besides, you already know that I think Agile sucks.
I prefer to call it a running prototype, or an integration environment, or anything else that’s more, well, accurate. “Shippable code” tells overeager managers that they can just press “stop” at any point and ship what they’ve got. Nice dream land, hot shot, but it doesn’t actually work quite like that. Nevertheless, a good integration development environment, where everyone’s code gets placed and which is subjected to regular updates and testing—that’s important, almost as important as CVS and a decent build script, and for a lot of the same reasons. If the team’s code isn’t conversing regularly, then the team may as well not be, either. The best way to prevent blockages is to prevent assumptions; the best way to prevent assumptions is to check them every single day. So: Does it work today?
Completion feels good. Getting something that works feels good. That’s why we got into this business. Because, despite all the pain and frustration and work that goes into developing software, The High here is about as good as it gets.
You don’t get that waiting. You don’t get that talking. You get that when you are immersed in building something.
From a team dynamic perspective, I can’t even find words to express how it makes people gel when we each do our little pieces, and plug them in, and see a site come up with data and styles and behavior and images and everything. It’s a pretty marvelous event that makes everyone even more eager to get to work. If you’re a web developer, don’t wait for your designer or back-end engineer to give you what you need. Make an unstyled page with dummy data, if you have to, but make something on day 1 if possible. Then make a list of what you need. When everyone can see their effect on the product, it’s amazing how fast they tend to deliver.
Going fast and pushing to milestones also forces everyone on the team to prioritize. There may be plenty of features that might be nice to implement, but if speed is a priority, then it forces you all to work on the things that really need to be implemented. By raising red flags whenever something doesn’t work, the whole project stays in sync most of the time, and assumptions never get too far out of whack. But, of course, that’s not the whole story.
Software development is a lot like building Frankenstien’s monster. You start out with a pile of useless ugly pieces, and try to turn it into something beautiful. Along the way, it’s a monstrosity, and tends to get uglier as you tack new bits onto it. Then it kills people.
In Mary Shelly’s classic, everyone Victor loves is killed by the eloquent and sensitive creature, who was turned evil by his creator’s neglect and hatred. If software could walk and talk, how many developers would still be alive today?
Don’t build Frankenstein’s monster, or at least, don’t figure you’re “mostly done” once it walks and talks, as the strict waterfall process proposes. Going fast is important, and necessary. You can’t fix pure ideas very easily, because it’s hard to see what’s wrong with them. Write the code. But know that it’s going to have to be rewritten, possibly several times. Plan for it, and keep it in the back of your mind.
The hazard of moving quickly is that we tend to go with the easy choice, rather than the good choice. Adding one more method onto the class I’ve already got is easier than creating a whole new class; nevermind that it doesn’t really “fit” with what this type of object is supposed to be doing. Do that once, and it’s probably a lot simpler and clearer than any alternative. Do that 50 times, and you end up with a theological problem, or something else equally pungent.
This will happen. Every time. If you don’t have some downright embarrassing shit-tastic WTF-worthy code lurking in your project—I mean, the kind of thing where you see a bug report go to your coworker, and you say to him, Oh, I’ll take that one, I know that feature
, but really, you’re saying Please don’t look at that code, or at least, if you do, please don’t judge me
—then you probably weren’t going fast enough. The remarkable thing isn’t that this team has produced a product that “already” needs a cleanup; the remarkable thing is that we all seem to recognize that we need to clean things up, before it’s blown up in any serious way yet. That’s why I’m thankful to be on the team that I am. I’ve heard it said that the difference between a good programmer and a bad programmer is that a good programmer feels pain when he looks at ugly code he’s written, while a bad programmer thinks everything he’s done is great.
I used to be proud of every bit of code I’d written, like a two-year-old who just learned how to use the potty. Then I grew up a little bit, and realized that my shit stinks, and hoped no one would ever see it. I’ve grown up a little more, and realized that everyone shits, so there’s nothing to be ashamed of. I try to work towards a point where my actually shit doesn’t stink at all, but I’m not all that hopeful about that. The fecality of this line of metaphor, while a bit disgusting, is meant to highlight an important point.
You want to create code that is nice to be around? Do your best to build quality in from the start. The more you learn, the better you’ll be at it, especially if you manage to code yourself into some truly awful maintenance nightmares, where you can’t walk away and can’t blame anyone else. But no matter how good you get, it’s still gonna stink most of the time. Everyone writes bad code. So fix it.
Wikipedia says:
Refactoring neither fixes bugs nor adds new functionality. Rather it improves the understandability of the code or changes its internal structure and design, and removes dead code, to make it easier for human maintenance in the future.
To relate to the twin principles of software development, “Go fast” and “Does it work?”, refactoring is not important early on. Surely, doing things as close as possible to The Right Way is usually a big help. But The Right Way almost always flies in the face of the “Go Fast” maxim.
I’ve been privy to a few refactoring projects, I think I can safely say that almost every single one was an utter disaster. Without naming names, here’s how a few turned out:
It works now, what’s the matter with it?To the best of my knowledge, of three teams where I saw this pattern, (a) one company no longer exists, (b) another company lost all their best talent, and (c) in the third case, the team dissolved and everyone moved onto other projects (it was at Yahoo!, so it’s not like they were going to go under.) For failing before they’d even gotten started, these refactoring projects are QUADRUPLE UBER KAMEHAMEHA FAIL!!!
I strongly doubt that any of these cases were rare. In fact, I think they’re the norm.
It’s easy to cite bad management, and in a few of the above cases, management was to blame. They put a high premium on getting new features, without recognizing the hidden costs down the road of dealing with the increasing instability. However, the fact that I’ve come to realize is that most developers don’t really understand maintainability, and thus, tend to completely miss the point of what the purpose of refactoring should be. As a result, their attempts at refactoring end up being worse than failures, because they are interpreted as success.
I did get to see one significant refactoring project that I’d consider a success. It was done in an under-the-radar sort of way, simply because it was the only way to add new features to a product that had grown frightfully unwieldy. Also, I’ve seen and done a lot of small scale refactorings that were executed well and increased the quality of the product as a result. (These, as the Wikipedia article mentions, are usually referred to as simply “cleaning up” the code base.)
Refactoring can only be justified in the “Go Fast” methodology on the condition that it makes a more stable product, thus enabling developers to more efficiently create working software. Elegance is not an end in itself. The end goal is a working product, and code is the enemy. The vast majority of software development is about modifying an existing product. The short-term memory of a human is limited and fleeting. The less information that must be held in mind to understand a given section of code, the more quickly code can be modified, the lower the likelihood of introducing bugs with any given change. So, there is, ultimately, only one purpose that Refactoring should serve:
Be more obvious.
Victor was so consumed by his desire to create something new that he ended up rushing the job and not thinking ahead. When his creature opened its eyes, he feared it, and fled, and tried to hide from his sin. Shunned from the world, the creature asked Victor to make him a companion. When his request was denied, he proceeded to destroy everything Victor loved.
The moral of the story: Don’t ignore the ugliness you create. It will find you.
Refactoring should make the code simpler. Complexity is the demon that we fight in writing code. It is the enemy. It opens the doors to bugs; it increases the difficulty in fixing problems and adding features; it raises the learning curve for new developers. Sadly, a lot of refactoring seems to make code more complex rather than less.
In my opinion, there are a few guidelines that work well when changing around existing code. These guidelines apply equally well when creating new code, but since it’s harder to see the whole picture in that scenario, mistakes and shortcuts are more forgivable.
Refactoring isn’t about using the coolest object oriented tricks that you just learned. It’s not about making the code “more abstract.” Abstraction is a necessary evil, in my opinion, not a feature. Like Victor, it’s easy to get wrapped up in testing out new discoveries. Most of us got into this job because we like playing with new puzzles, and that’s a great thing. But one of the most satisfying puzzles is to reduce the complexity of an implementation without reducing the problems that it can solve.
I’m a big fan of “comment-driven development”. That is, write out the pseudo-code in comment form first, stubbing out the functions that need to be implemented. It helps to separate the design and implementation phases, since it can be hard to do both simultaneously. As implementation kinks are getting worked out, the design might change slightly, but it helps to have the comments there as a guide to what I was originally thinking. Once a module moves into a more stable place, the comments are a liability. At best, they’re clutter; at worse, misleading.
I like using those comments to clean up the code as well. The process goes like this:
I’m not so certain on the common refactoring claim that it should not change any existing functionality. Sometimes, this is of vital importance. However, there are plenty of times where the feature set of a bit of code simply needs to be trimmed or revised. For public-facing APIs and products, either the functionality should not be changed, or interfaces should be publicly deprecated long before they are actually removed.
For internal code, or APIs that are only used by a small number of consumers, my favorite approach is to just change it, and then see what breaks. This should never be done in a way that affects customers, of course, but in a development environment, it’s quite appropriate to just change the back-end, and then update the front-end once it’s fixed. Defensive coding is a good practice, but overly defensive coding, where errors are not surfaced at all, can hide problems that may turn up later in unexpected ways.
Don’t bother refactoring something you just wrote. You’ll only make it worse. You have to wait until you don’t remember how it works before you look at it again, or else you won’t be able to appreciate the difficult bits. If you don’t have time for that, then make someone else do it, and tell them to be as harsh as possible.
As they say, No matter what the problem is, it’s always a people problem.
The first and most important step in any refactoring effort is to detach our egos from our code. This is even more of an issue if something you wrote is being refactored by someone else, or worse, if you are refactoring someone else’s code.
I’ve worked hard to develop a pretty good understanding of software development. I’ve found that the technology side is easy—on the human side, I still have a lot to learn, and probably always will. Like a lot of geeks, I didn’t really get into the whole “social” thing in a serious way until I was almost an adult, and even today, I tend to focus on The Project and keep my distance from the rest of the world. So, I suppose I don’t know the best way to handle this part of the problem.
TBL’s classic recommendation to be loose in what you accept, and strict in what you send
seems to apply well to this case. On a healthy team, everyone is committed to a successful product, and leaves their egos at home when they come to work. Code is passed around and everyone seeks as much input as possible. On a dysfunctional team, everyone’s sense of self-worth is tied up in the appraisal of their work by others—which has the twin effect of making them overly sensitive to criticism, and overly critical of their teammates.
It’s natural to put up resistance to the prospect of your code being modified. After all, you wrote it that way for a reason. Sit down with the other developer, and try to hash out the best approach. If arbitrary decisions have to be made, explain that the best approach is to make them consistently. Don’t focus on the problems in their code, but rather on the need for consistency and abstraction throughout the product.
If someone won’t listen to reason, frankly, the only solution I’ve ever found is to either leave or wait for them to. Hopefully they’re a contractor or something, so you can just wait, and do your best to isolate and minimize the damage as much as possible in the meantime. If not, then the issue should be escalated, if only because it’s polite to tell your manager that their ship is in danger of sinking.
I read Refactoring about 5 years ago, but I’ve been meaning to pick it up again, since I’ve gotten a fair amount of real-world development experience since then, and have seen my share of dismal failures. But this isn’t supposed to be a book review, so I won’t review it in any detail. It’s a good read, though, and I highly recommend it, even though it is mostly in Java, and seemed (to me at least) to simply highlight a lot of design problems inherent in the Java language that lead inevitably to bloated over-sized code. But that’s a post for another day.