Foo Hack » Code Ecosystems http://foohack.com Isaac Schlueter on Web Development Tue, 03 Nov 2015 06:34:16 +0000 http://wordpress.org/?v=2.6.3 en CSS vs Tables: You’re Doing It Wrong http://foohack.com/2009/02/css-vs-tables-youre-doing-it-wrong/ http://foohack.com/2009/02/css-vs-tables-youre-doing-it-wrong/#comments Tue, 03 Feb 2009 07:40:22 +0000 Isaac http://foohack.com/?p=108 ...Read More]]> Definitions first.

Table

A specified set of columns and rows with cells that automatically and fairly smartly expand to fit their contents; cells can span rows or columns; the table as a whole can be treated as one block, and cells can contain tables. A table is a metaphor for visually laying out 2-dimensional content.

CSS

cascading style sheets; not floating divs, not any specific markup, but just the concept of the visual display of markup specified by a series of rules that are kept separate from the markup they operate on, generally in an external linked file or a style tag in the head of the document.

Talking Past Each Other

CSS is (for various reasons cited elsewhere ad nauseam), in most cases, the more optimal technological approach. That’s not to say that the existing state of the CSS language is perfect. In fact, even the most vocal CSS advocates regularly assert that there are deep problems with the existing spec and browser support.

Tables are (for various reasons cited elsewhere ad nauseam), in most cases, a more optimal metaphor than any of its rivals. There’s nothing inherently wonderful about the tags themselves. In fact, even the most vocal table advocates regularly assert that the markup is ugly.

Therein lies the crux of the problem. CSS is a great technological approach, and tables are a great metaphor. The question then usually becomes, Should I mark up my content using <div> tags or <table> tags?, which really means, Is the separation of style from content more or less important than using the optimal layout metaphor? The debate goes a little like this:

tables

This layout metaphor is better, so you should use these tags. All you idealists are unrealistic! (Also, if you think markup matters, you believe in fairies. It so doesn’t!)

CSS

Metaphor shmetaphor! You don’t care about the sanctity of data or code quality. Think of the maintenance! (Also, if you can’t do it with divs, you’re stupid. It’s so easy!)

The Right Answer, it would seem, is to use the table metaphor with the CSS technology. In the ideal world, you’d always mark up your content using the tags that would communicate your meaning most clearly to your intended user agents. Then you’d tell user agents how to treat those tags visually. To describe this visual display, you should be able to use the table metaphor, or the float metaphor, or absolute and relative positioning, or anything else, but that should not be done by your markup.

Getting Real

In the real world, it’s not so nice. The CSS specification has a set of display values designed to mimic the table tags, but they aren’t well supported, and anyway, it’s an imperfect imitation of a hack that was not really intended to be used the way we all ended up using it.

In other words, CSS is not an ideal example of what CSS aims at, and the table/tr/td tags are not an ideal implementation of the table/grid metaphor. CSS tables, though they are clever and in some cases quite useful, take it from bad to worse.

The state of the CSS art is not at a point where you can realistically expect to make any significant stylistic changes to your pages without altering the markup. If you need to move the navigation bar from the right to the left, or re-skin the page with dropshadows instead of rounded corners, or convert your gradients from 2-tone to 3-tone, or make your boxes vertically centered instead of top-aligned, you’re probably going to have to change your markup, at least a little.

In theory, it’s possible. I know a handful of my colleagues will vehemently disagree and point at countless approaches to enable virtually any kind of reskinning using only CSS changes. (I’ve used Leslie Sommer’s CSS Mojo technique to great success in the past; we used it for pretty much everything on Yahoo! Buzz.) But let me tell you, from years of experience building production sites with CSS, most of the time, in reality it just doesn’t work that way.

And why should it? What kind of crazy lunatic writes all their HTML by hand in text files, anyhow? Clearly you have some kind of back-end engine spitting it out from templates, so you just change the template, and voilà!, the markup is changed everywhere.

A Very Blue Bike Shed

Working at Yahoo, I’ve met some webdevs with truly incredible CSS ability, who crank out live code under real deadlines to exacting standards. They use CSS, and not tables, and they Get Shit Done. There are a lot of them, more than I can list here, but Nate Koechley, Matt Sweeney, Hedger Wang, and Leslie Sommer all deserve a special mention. I came to Yahoo knowing CSS pretty well, but I became an expert largely as a result of working in such a highly skilled community of developers.

Also due to my time at Yahoo, I’ve seen some absolutely crazy debates about markup and coding standards on the devel-frontend mailing list. I mean, you think this little flare-up in the blogosphere is anything? You got no idea, buddy. Seriously. And these are coworkers who (for the most part) really like and respect one another.

You wanna know a secret? It makes no difference.

Language is as language does. If everyone uses the <table> tag for markup, guess what happens: <table> is now a semantically meaningless (or at least, semantically vague) tag, and any user agents that might care about navigating tabular data have to deal with that fact. It becomes just another <div>, for all intents and purposes, but a little less flexible, and with a default styling that makes it handy for layout.

The passion for the rants comes from the fact that it is meaningless! Of course it’s a fertile ground for bikeshedism!

That’s right: It doesn’t matter.

Use whatever tags work for you. I don’t care. You’ll still be a good person.

I can tell you from experience that a deep knowledge of CSS is essential for serious front-end development. (Most so-called DHTML is actually just manipulating CSS with Javascript, and the Javascript is pretty light.) CSS can do some things that table tags can’t, so you ought to learn it. The reverse is also true, so you should know how to use tables, too.

Write code that you can maintain, that is flexible enough to let you change it without sacrificing the stability of your product. Test it in the user agents that you expect to support. Stop debating the color of this bikeshed.

If there’s ever a good reason to go back and change your tables to divs, or vice versa, you can do that. Hell, plan for that, because whether you use CSS for layout or not, you’re going to need to touch your markup sometimes. It is much much harder to build a product people want than it is to build a product that works in Browser X.

Belorussian translation provided by fatcow

]]>
http://foohack.com/2009/02/css-vs-tables-youre-doing-it-wrong/feed/
Going Fast, Frankenstein, and Refactoring http://foohack.com/2008/01/going-fast-frankenstein-and-refactoring/ http://foohack.com/2008/01/going-fast-frankenstein-and-refactoring/#comments Mon, 21 Jan 2008 17:00:48 +0000 Isaac http://foohack.com/2008/01/going-fast-frankenstein-and-refactoring/ already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs. That's right. We haven't released anything, and there's already enough bloat to justify spending at least 2 weeks cleaning it up. ...Read More]]> Very soon now, the project that has consumed my days for the last 6 months will go into public beta. It’s too early to pop the champagne, and that’s not what this post is about. However, the newness of this project is worth pointing out for the following reason: We’re already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs.

That’s right. We haven’t released anything, and there’s already enough bloat to justify spending at least 2 weeks cleaning it up.

Of course, due to the pressures of market, and the morale effects of delaying the release, we’re buttoning up what we have now as much as possible, and planning to work on the refactor once the beta is out there. It may actually help a lot, since the gigantic firehose of traffic that Yahoo! points at sites has a way of helping even the most egotistical of developers realize that their clever baby needs some serious surgery.

This post is going somewhere, I swear, but I can already tell it’s going to take me a while to get there. Go get some coffee and a snack and come back. I’ll wait.

Build Fast

In my opinion, the core of good software development methodology comes down to two basic questions that should be asked daily:

  1. What portion of your time is spent focused on the task of creating software?
    (That includes planning, discussion, and thinking as well as coding, provided that these non-coding activities are focused and aimed at creating software.)
  2. Does it work today?
    (If you don’t know, then you are full of Fail. Get on that right away.)

I call this the Go Fast method. The goal is to efficiently create software with minimum long-term cost and maximum long-term benefit, without driving away your best talent. (Have I got the attention of the business and product people in the audience? Good. Please stay for the rest of the show.)

I know that someone out there is bound to point out the obvious parallels between my “go fast” methodology and Agile, and maybe they’re right. To the extent that Agile accepts these premises as important, it is correct. But I’m claiming that everything else stands or falls based on how it affects these questions. Stand-ups, planning meetings, org charts, IDEs, frameworks; they’re all only as good as their effect on the principles of speed and stability in development.

If you’re developing software properly, you should be creating software that does something as soon as possible, and keep it up as much of the time as you can. Until there’s something to look at, you’re just spinning your wheels. Nowhere is this more true than on the web. Photoshop comps don’t break in different browsers. Product presentation decks don’t “feel klunky” when pieces load out of order. Grand visions promise to solve all your problems and make you biscuits for breakfast, but they don’t deliver.

You make software for a living? So make software, already. You wanna build a website? What’s stopping you? You don’t have designs yet? Build the parts that you do have. Always be releasing, even if your actual release date is months away.

Build the core, and grow the other pieces up around it. Every minute spent waiting, instead of actually building something, is worse than wasted. When you live in the code day by day, it becomes a part of you. You learn how the pieces work, and the knowledge is automatic. When you get too far from it, it becomes foreign, like driving a car for the first time.

The agilistas call this “shippable code.” In my opinion, that’s a horrible term. The vast majority of code that I’ve seen released into the wild was not even close to what I’d call “shippable,” but it got shipped just the same. Most of the time, the prototypes leading up to the release were significantly worse. Some agilistas are quick to point out that “shippable” doesn’t always really mean “that you would actually ship it.” Sorry, but drivable means that you can drive it, and drinkable means that you can drink it, and flyable means that you can fly it, so if “shippable” doesn’t mean “you can ship it,” then they picked a bad term.

I suspect the ambiguity is intentional on some level. I think the claim that “Agile makes software faster” is a myth. Agile methodologies, used in moderation by a well-balanced team of dedicated and talented individuals, do tend to result in higher quality software, greater management visibility, and a happier bunch of people than any alternative, but that means that the overall development time can be significantly longer. And if you’re not on a well-balanced team of dedicated and talented individuals, then no methodology is going to save that sinking ship, so dust off the CV and bail out ASAP. But I digress, and besides, you already know that I think Agile sucks.

I prefer to call it a running prototype, or an integration environment, or anything else that’s more, well, accurate. “Shippable code” tells overeager managers that they can just press “stop” at any point and ship what they’ve got. Nice dream land, hot shot, but it doesn’t actually work quite like that. Nevertheless, a good integration development environment, where everyone’s code gets placed and which is subjected to regular updates and testing—that’s important, almost as important as CVS and a decent build script, and for a lot of the same reasons. If the team’s code isn’t conversing regularly, then the team may as well not be, either. The best way to prevent blockages is to prevent assumptions; the best way to prevent assumptions is to check them every single day. So: Does it work today?

Completion feels good. Getting something that works feels good. That’s why we got into this business. Because, despite all the pain and frustration and work that goes into developing software, The High here is about as good as it gets.

You don’t get that waiting. You don’t get that talking. You get that when you are immersed in building something.

From a team dynamic perspective, I can’t even find words to express how it makes people gel when we each do our little pieces, and plug them in, and see a site come up with data and styles and behavior and images and everything. It’s a pretty marvelous event that makes everyone even more eager to get to work. If you’re a web developer, don’t wait for your designer or back-end engineer to give you what you need. Make an unstyled page with dummy data, if you have to, but make something on day 1 if possible. Then make a list of what you need. When everyone can see their effect on the product, it’s amazing how fast they tend to deliver.

Going fast and pushing to milestones also forces everyone on the team to prioritize. There may be plenty of features that might be nice to implement, but if speed is a priority, then it forces you all to work on the things that really need to be implemented. By raising red flags whenever something doesn’t work, the whole project stays in sync most of the time, and assumptions never get too far out of whack. But, of course, that’s not the whole story.

Gene Wilder as Victor FrankensteinSoftware development is a lot like building Frankenstien’s monster. You start out with a pile of useless ugly pieces, and try to turn it into something beautiful. Along the way, it’s a monstrosity, and tends to get uglier as you tack new bits onto it. Then it kills people.

In Mary Shelly’s classic, everyone Victor loves is killed by the eloquent and sensitive creature, who was turned evil by his creator’s neglect and hatred. If software could walk and talk, how many developers would still be alive today?

Slow Down

Don’t build Frankenstein’s monster, or at least, don’t figure you’re “mostly done” once it walks and talks, as the strict waterfall process proposes. Going fast is important, and necessary. You can’t fix pure ideas very easily, because it’s hard to see what’s wrong with them. Write the code. But know that it’s going to have to be rewritten, possibly several times. Plan for it, and keep it in the back of your mind.

The hazard of moving quickly is that we tend to go with the easy choice, rather than the good choice. Adding one more method onto the class I’ve already got is easier than creating a whole new class; nevermind that it doesn’t really “fit” with what this type of object is supposed to be doing. Do that once, and it’s probably a lot simpler and clearer than any alternative. Do that 50 times, and you end up with a theological problem, or something else equally pungent.

This will happen. Every time. If you don’t have some downright embarrassing shit-tastic WTF-worthy code lurking in your project—I mean, the kind of thing where you see a bug report go to your coworker, and you say to him, Oh, I’ll take that one, I know that feature, but really, you’re saying Please don’t look at that code, or at least, if you do, please don’t judge me—then you probably weren’t going fast enough. The remarkable thing isn’t that this team has produced a product that “already” needs a cleanup; the remarkable thing is that we all seem to recognize that we need to clean things up, before it’s blown up in any serious way yet. That’s why I’m thankful to be on the team that I am. I’ve heard it said that the difference between a good programmer and a bad programmer is that a good programmer feels pain when he looks at ugly code he’s written, while a bad programmer thinks everything he’s done is great.

I used to be proud of every bit of code I’d written, like a two-year-old who just learned how to use the potty. Then I grew up a little bit, and realized that my shit stinks, and hoped no one would ever see it. I’ve grown up a little more, and realized that everyone shits, so there’s nothing to be ashamed of. I try to work towards a point where my actually shit doesn’t stink at all, but I’m not all that hopeful about that. The fecality of this line of metaphor, while a bit disgusting, is meant to highlight an important point.

You want to create code that is nice to be around? Do your best to build quality in from the start. The more you learn, the better you’ll be at it, especially if you manage to code yourself into some truly awful maintenance nightmares, where you can’t walk away and can’t blame anyone else. But no matter how good you get, it’s still gonna stink most of the time. Everyone writes bad code. So fix it.

Refactoring

Wikipedia says:

Refactoring neither fixes bugs nor adds new functionality. Rather it improves the understandability of the code or changes its internal structure and design, and removes dead code, to make it easier for human maintenance in the future.

To relate to the twin principles of software development, “Go fast” and “Does it work?”, refactoring is not important early on. Surely, doing things as close as possible to The Right Way is usually a big help. But The Right Way almost always flies in the face of the “Go Fast” maxim.

I’ve been privy to a few refactoring projects, I think I can safely say that almost every single one was an utter disaster. Without naming names, here’s how a few turned out:

  1. Lead engineer spent 3 months internally redesigning a feature with the intent of making it more extensible in the future. Released. Numerous bugs and problems. When I left the company, it was still unstable. FAIL
  2. The product has grown into an amalgam of junk, and no one knows how to maintain all the disparate pieces. Can’t add features easily. Needs to be re-architected. [[numerous planning meetings]] OK, management says we have 2 months, so let’s just add another layer on it, change how it looks, and call it a day. Result was even more disorganized and difficult to extend. FAIL
  3. #2, again, with new people driving it, who insisted that the last ones were all wrong. Same result. DOUBLE FAIL
  4. Repeat, again. TRIPLE SUPER DUPER FAIL
  5. Last, but oh, so certainly not least, there were all the refactorings that didn’t even make it to a real execution. The times when the development team was so fed up that they pushed to change things, and were told repeatedly that they could get to that just as soon as the next project got done—there was always a “next project.” It works now, what’s the matter with it? To the best of my knowledge, of three teams where I saw this pattern, (a) one company no longer exists, (b) another company lost all their best talent, and (c) in the third case, the team dissolved and everyone moved onto other projects (it was at Yahoo!, so it’s not like they were going to go under.) For failing before they’d even gotten started, these refactoring projects are QUADRUPLE UBER KAMEHAMEHA FAIL!!!

I strongly doubt that any of these cases were rare. In fact, I think they’re the norm.

It’s easy to cite bad management, and in a few of the above cases, management was to blame. They put a high premium on getting new features, without recognizing the hidden costs down the road of dealing with the increasing instability. However, the fact that I’ve come to realize is that most developers don’t really understand maintainability, and thus, tend to completely miss the point of what the purpose of refactoring should be. As a result, their attempts at refactoring end up being worse than failures, because they are interpreted as success.

I did get to see one significant refactoring project that I’d consider a success. It was done in an under-the-radar sort of way, simply because it was the only way to add new features to a product that had grown frightfully unwieldy. Also, I’ve seen and done a lot of small scale refactorings that were executed well and increased the quality of the product as a result. (These, as the Wikipedia article mentions, are usually referred to as simply “cleaning up” the code base.)

Refactoring can only be justified in the “Go Fast” methodology on the condition that it makes a more stable product, thus enabling developers to more efficiently create working software. Elegance is not an end in itself. The end goal is a working product, and code is the enemy. The vast majority of software development is about modifying an existing product. The short-term memory of a human is limited and fleeting. The less information that must be held in mind to understand a given section of code, the more quickly code can be modified, the lower the likelihood of introducing bugs with any given change. So, there is, ultimately, only one purpose that Refactoring should serve:

Be more obvious.

Love Your Monster

Inside cover of the original 1918 Frankenstein by Mary Shelly.Victor was so consumed by his desire to create something new that he ended up rushing the job and not thinking ahead. When his creature opened its eyes, he feared it, and fled, and tried to hide from his sin. Shunned from the world, the creature asked Victor to make him a companion. When his request was denied, he proceeded to destroy everything Victor loved.

The moral of the story: Don’t ignore the ugliness you create. It will find you.

Refactoring should make the code simpler. Complexity is the demon that we fight in writing code. It is the enemy. It opens the doors to bugs; it increases the difficulty in fixing problems and adding features; it raises the learning curve for new developers. Sadly, a lot of refactoring seems to make code more complex rather than less.

In my opinion, there are a few guidelines that work well when changing around existing code. These guidelines apply equally well when creating new code, but since it’s harder to see the whole picture in that scenario, mistakes and shortcuts are more forgivable.

  • There should be as few layers as possible, and no fewer.
  • There should be as few objects as possible, and no fewer.
  • Each object should be as small as possible, and no smaller.
  • Each object should know as little as it needs to, and no less.
  • Each piece should have a job, and should stick to it.
  • Comments should be mostly unnecessary.

Refactoring isn’t about using the coolest object oriented tricks that you just learned. It’s not about making the code “more abstract.” Abstraction is a necessary evil, in my opinion, not a feature. Like Victor, it’s easy to get wrapped up in testing out new discoveries. Most of us got into this job because we like playing with new puzzles, and that’s a great thing. But one of the most satisfying puzzles is to reduce the complexity of an implementation without reducing the problems that it can solve.

I’m a big fan of “comment-driven development”. That is, write out the pseudo-code in comment form first, stubbing out the functions that need to be implemented. It helps to separate the design and implementation phases, since it can be hard to do both simultaneously. As implementation kinks are getting worked out, the design might change slightly, but it helps to have the comments there as a guide to what I was originally thinking. Once a module moves into a more stable place, the comments are a liability. At best, they’re clutter; at worse, misleading.

I like using those comments to clean up the code as well. The process goes like this:

  1. Read through the code, starting at the entry point, and opening each file as it’s referenced.
  2. If a section of code has a shit-ton of comments, then it needs work. If it’s clean, just make sure it makes sense.
  3. Make sure the comments match what the code is doing.
  4. Remove any incorrect comments (comments that are lying.)
  5. If the code isn’t understandable without the comments, then dig in and clean it up. Rename methods, sort things better, etc.
  6. Remove any and all implementation-related comments. Those are dangerous.
  7. Repeat until the file is almost comment free, and understandable.

I’m not so certain on the common refactoring claim that it should not change any existing functionality. Sometimes, this is of vital importance. However, there are plenty of times where the feature set of a bit of code simply needs to be trimmed or revised. For public-facing APIs and products, either the functionality should not be changed, or interfaces should be publicly deprecated long before they are actually removed.

For internal code, or APIs that are only used by a small number of consumers, my favorite approach is to just change it, and then see what breaks. This should never be done in a way that affects customers, of course, but in a development environment, it’s quite appropriate to just change the back-end, and then update the front-end once it’s fixed. Defensive coding is a good practice, but overly defensive coding, where errors are not surfaced at all, can hide problems that may turn up later in unexpected ways.

Don’t bother refactoring something you just wrote. You’ll only make it worse. You have to wait until you don’t remember how it works before you look at it again, or else you won’t be able to appreciate the difficult bits. If you don’t have time for that, then make someone else do it, and tell them to be as harsh as possible.

Ego

As they say, No matter what the problem is, it’s always a people problem. The first and most important step in any refactoring effort is to detach our egos from our code. This is even more of an issue if something you wrote is being refactored by someone else, or worse, if you are refactoring someone else’s code.

I’ve worked hard to develop a pretty good understanding of software development. I’ve found that the technology side is easy—on the human side, I still have a lot to learn, and probably always will. Like a lot of geeks, I didn’t really get into the whole “social” thing in a serious way until I was almost an adult, and even today, I tend to focus on The Project and keep my distance from the rest of the world. So, I suppose I don’t know the best way to handle this part of the problem.

TBL’s classic recommendation to be loose in what you accept, and strict in what you send seems to apply well to this case. On a healthy team, everyone is committed to a successful product, and leaves their egos at home when they come to work. Code is passed around and everyone seeks as much input as possible. On a dysfunctional team, everyone’s sense of self-worth is tied up in the appraisal of their work by others—which has the twin effect of making them overly sensitive to criticism, and overly critical of their teammates.

It’s natural to put up resistance to the prospect of your code being modified. After all, you wrote it that way for a reason. Sit down with the other developer, and try to hash out the best approach. If arbitrary decisions have to be made, explain that the best approach is to make them consistently. Don’t focus on the problems in their code, but rather on the need for consistency and abstraction throughout the product.

If someone won’t listen to reason, frankly, the only solution I’ve ever found is to either leave or wait for them to. Hopefully they’re a contractor or something, so you can just wait, and do your best to isolate and minimize the damage as much as possible in the meantime. If not, then the issue should be escalated, if only because it’s polite to tell your manager that their ship is in danger of sinking.

Oh, right, that book…

Refactoring, by Martin FowlerI read Refactoring about 5 years ago, but I’ve been meaning to pick it up again, since I’ve gotten a fair amount of real-world development experience since then, and have seen my share of dismal failures. But this isn’t supposed to be a book review, so I won’t review it in any detail. It’s a good read, though, and I highly recommend it, even though it is mostly in Java, and seemed (to me at least) to simply highlight a lot of design problems inherent in the Java language that lead inevitably to bloated over-sized code. But that’s a post for another day.

]]>
http://foohack.com/2008/01/going-fast-frankenstein-and-refactoring/feed/
Surgical Team or Motley Crew of Adventurers? http://foohack.com/2007/12/surgical-team-or-motley-crew-of-adventurers/ http://foohack.com/2007/12/surgical-team-or-motley-crew-of-adventurers/#comments Mon, 03 Dec 2007 19:10:17 +0000 Isaac http://foohack.com/2007/12/surgical-team-or-motley-crew-of-adventurers/ geekiest thing every written, I catalog some different character classes and explain their corollary roles in a web development team. I suspect that other types of programming teams have similar roles. ...Read More]]> I’ve read The Mythical Man-Month lately (at Geoff’s suggestion, among others) and it’s great. I definitely recommend it highly to anyone in the software development field.

In particular, I was struck by the idea of Mills and Brooks that software development could be done more effectively by a “surgical team.” That is, the experienced lead developer writes all or most of the code, and is supported by a team of people to make him more effective. While this is perhaps an effective way to capitalize on the significant difference between excellence and mediocrity in coders, it seems to me that most of the jobs on the team are today either (a) made obsolete by technological advancements, or (b) best done by specialists who float between teams. For example, Yahoo! has some of the best Javascript and CSS “language lawyers” in the world—but does every team really need one? The need for a full-time toolsmith has been all but completely surpassed by the customizability of workstations and the availability of many high-quality tools for coding and such.

The nature of software development has changed somewhat as well. We aren’t writing assembly code any more; not only do we have high-level languages, in my current project, there are no fewer than 9 different languages in use, 10 if you count regular expressions, which we all use quite frequently. [1] We aren’t all creating the same kind of stuff. Thus, there is a strong need for all of us to be experts, surgeons if you will, in our respective areas. In addition, the non-coding jobs require their own sort of specialized focus.

It occurred to me that this division of labor, where everyone depends on everyone else but everyone has a very different job, is more like a Dungeons and Dragons party than a surgical team. Each member plays a vital but remarkably different role, each owns what they do, and each is the team’s expert in what they do. As a webdev, I get consulted about front-end concerns, and am quick to ask a back-end engineer about things that touch their domain. UI questions are decided by the interaction designer; questions of product features are handled by the product manager, etc. I suspect that this is not unique to software for the web; a similar metaphor exists (or at least, could exist) for application software.

The best D&D parties are a bit formulaic, but that’s simply because a certain distribution of talents tends to be most adaptable to the most different kinds of problems.

Fighter

fighter Not the flashiest or most prestigious role, Fighters have the lowest barrier to entry. They accumulate experience quickly, and don’t have a lot of class bonuses. (Until 3rd Edition made them into action heroes, that is.) But their simplicity belies a focus and strength that is crucial. The fighter is the guy in the party that protects the more “thinky” types from anything that might get in their way. There can be honor and satisfaction in bashing orcs one at a time.

On a web development team, there are two archetypes that sometimes satisfy this role. The first is the diligent code monkey who might not be the most ambitious or inventive, but is good at getting things done. They are perfectly happy to churn through buglists, knocking down whatever they can. If you show them how to do something, they’ll make it work however it has to through sheer force of will—not always fast or elegant, but functional. When depended upon properly, code monkeys are a great and valuable asset.

Also in the fighter category are the producers who manage the content of the site, use the admin tools, troll for spammers, and do all the other busywork of web site maintenance. When the programmers have moved on to the next problem, they’re still working. Don’t forget them. Whenever possible, make their lives easier. They are the ones that keep your product shiny, so you can continue to be proud to have the URL sitting there on your resume.

Ranger

Rangers are good at surviving on their own. They shun convention and human society. ranger Often a bit wild themselves, they feel more comfortable in the company of nature. Their chaotic bent can sometimes be a liability in an adventuring party, but their wide-ranging knowledge makes them hard to discount. They are fiercely independent and difficult (if not impossible) to tame.

In a programming team, this is the quintessential hacker. Most mashups are created by a single ranger acting alone. They are great at creating new things, and no browser quirk or network irregularity will stand in their way. On the other hand, they don’t always follow convention, and have a tendency to create overly clever code that is unintelligible to anyone who follows in their wake. Thus, they have to be watched closely lest they venture off.

I’ve learned a lot of great tricks and techniques from the Ranger programmers I know. I’m always a bit envious of their free approach to things, but I guess I’m just too timid to abandon convention completely. Make sure that you understand their code before you try to adapt it to your own ends—wild animals turn vicious when handled improperly.

Paladin

paladin The paladin is the character class with the most stringent requirements. They must conform to the strictest of moral codes and demanding attribute requirements. As such, they’re rare, and respected. The paladin’s job is twofold. As a warrior subclass, he protects the party members from outside threats. However, he’s also a healer, and a leader. Charismatic and compassionate, he holds the party together by negotiating internal conflicts. Unshakably positive, he nonetheless keeps everyone going with firm conviction in the mission.

The best project managers aspire to paladin-like grace in their roles. They defend the team from the pressures of management and the monotony of GANT and PERT charts, and make sure that everyone is able to work at their top potential. When there’s a problem, they are approachable and always seek a win-win situation. By being consistently honest and righteous, they earn the respect and admiration of those both below and above them on the management totem pole.

When someone leads like a paladin, you want to get your tasks done and meet your deadlines. Not because you’re afraid of the consequences, but simply because their drive and attitude are contagious.

There are some managers that are decidedly un-paladin-like. Few things are as hazardous on a project as leadership that does not inspire confidence and commitment. Attitude is everything in a leader.

Rogue

rogue The rogue is a highly adaptable character. They can pick locks and disarm traps, making them essential in any dungeon-faring party. Surviving in the streets and cities on his wit and stealth, a rogue comes to know the different factions and groups, and learns to navigate the urban waters. He knows how to make a buck, how to make a friend, and how to sell, and where to buy.

The product manager takes on this role for a programming team. They are often the ones who first envision the product and pitch it to management. Often the only member of a programming team with a business background [2], they settle questions of business goals and product requirements. They make the powerpoint presentations that eventually drive the development of a real software product.

The product manager, like a rogue, also sings the song of the party, talking them up when necessary and interacting with foreign powers [3]. If a product requires buy-in from another team, whether it’s a standards group in the same company or a potential outside partner, these are the people who make the calls and settle the deal. Their day is usually full of meetings.

I’ve known some developers who look down on product managers as being less technical. This is a big mistake. Their job is every bit as intellectually rigorous as ours, due to the huge amount of information that they need to be able to call forth on demand. Since they deal with features and requirements that have not even been fully envisioned, and might not exist in any form whatsoever in the market, they are even more in the world of pure thought-stuff than programmers. Make friends with your product manager, and you’ll find that they often do actually know their stuff. And after all, some rogues will stab you in the back if you cross them ;)

Mage

The mage is the general purpose wizard. They are resourceful and possess a broad understanding of all types of magic. While not specialized in their studies, they are deeply theoretical, research oriented, and possess incredible focus and discipline. Their power comes from being able to summon a wide variety of spells, but they tend to be weak and vulnerable in the chaos of melee, since they must prepare spells ahead of time.

Many experienced back-end engineers fit into this role. They tend to prefer structured approaches and strongly typed languages, and produce exceptionally high-quality code. wizard They often have advanced degrees in computer science and are knowledgeable in the intricacies of design patterns and memory management. They excel in areas that are somewhat regular and predictable, where the environment can be controlled, and the assumptions and edge-cases checked accurately.

Dedicated programming wizards are obviously important on any programming team. However, as much as “laymen” tend to see all software as a form of magic, the truth is that mages are actually pretty rare in the development world. They are the core of the “20%” of developers who truly care about what they create and diligently work to improve their abilities. Their craft is important to them, and they approach each problem with care and precision. It is much more than just a way to pay the bills.

While dedication is important, it comes at a price. Many back-end engineers come to get out of touch with the way that “normal” people think and interact with software. Most are pretty aware of this fact; I once knew an engineer who complained that bugzilla’s user interface was obviously designed by a database expert.

Specialist Wizard (illusionist, abjurer, conjurer, etc.)

specialist_wizard These are the wizards that have selected a certain area of magic in which to specialize, at the expense of reduced ability in some other area. They have the same theoretical focus as a mage, but instead of having a broad understanding of every type of magic, they work to gain a much deeper understanding of a single area. When it comes to solving that sort of problem, they are the best—however, in other areas, they may be lacking.

There are many “specialist wizards” in the programming world. Here at Yahoo!, there are teams that focus 100% of their time on developing PHP, or MySQL, or Apache. (Rasmus comes to mind.) The “Paranoids” team sits in the cubes adjacent to ours, and they work to make sure that all the software at Yahoo! is as bulletproof as possible. The exceptional performance team does research to help us webdevs make our pages load and run faster.

When you have a hammer, the world can begins to look like a nail. A MySQL specialist will be tempted to suggest a database schema to solve any problem, even perhaps if the problem would be better solved by some other method. You may be asking for trouble if you ask them to do something outside their specialization. However, if you have a certain task that you’re certain to need, they can be invaluable. Database optimization is a highly developed science, and it’s not always easy or simple. Good programmers are rarely security experts (or even security knowledgeable!)

If you aren’t likely to have a lot of problems in their specialty, it’s often better to consult with a specialist only if and when they’re necessary. That way a single specialist can service multiple different teams.

Sorcerer

Sorcerers are similar to mages in the types of spells that they can cast, but instead of preparing spells from a spellbook, they simply select from the spells that they know, calling them when needed. As such, they tend to be much more spontaneous. However, they do not cultivate the discipline or depth of understanding of the wizards, and so they are limited in the number of spells that they can learn. The sorcerer excels in areas that are uncharted, and their spontaneous approach to magic allows them to quickly adapt to any situation, no matter how unpredictable.

The sorcerer is the perfect corollary to the webdev, aka the front-end engineer. Web browsers are a constantly moving target, with each behaving a little differently and new versions being released all the time. sorceror While programming patterns and the “right” ways of doing things can be useful, they are often too bulky for Javascript that has to be delivered, parsed, and executed on demand. Essentially, the webdev creates software in an environment where the rules often make no sense; where the code has to work on different platforms with conflicting requirements; where speed and features are both must-haves.

I’ve only recently heard of any colleges that make any serious attempt to teach courses on web development. From what I have heard, the things they teach are pretty bad—at best out of date, at worst outright harmful. Most professional web developers that I’ve met didn’t learn their craft in school. Like the sorcerer, they simply picked up their craft as they could, and learned techniques by studying “in the wild”; that is, by viewing source and reading websites and trying things out.

I’ve said before, it takes a certain kind of crazy to want to write software in the browser. You have to love the chaos and enjoy pulling out tricks and coming up with new hacks to work around impossibly ridiculous requirements. At the same time, unlike the ranger’s free-wheeling wildness, the team’s sorcerer must have enough discipline to tame the chaos and create software with whatever degree of stability he can, in order to help create a functional and stable site.

It can be a challenge at times to find stability and see the bigger picture and not write code for a single case. Yes, yes, that hack made it work in Safari. Lovely. But it’s also going to be a nightmare to maintain, or you just ruined the accessibility. Trade-offs are always present, and as far as I can tell, only experience and a lot of mistakes can teach you how to choose wisely.

Cleric

Diligent and committed to the ethos of their higher power, the cleric summons heavenly forces to do his bidding. By making himself an instrument of the divine, he can accomplish great things. A cleric’s power comes from their will and common sense (in D&D, this is the Wisdom score), as opposed to the cleverness and wit that Wizards rely on. The cleric has the power to vanquish undead through the force of their convictions, and their healing spells make them a must-have for any party.

cleric In a programming party, this is the role of the QA tech. These noble souls must possess tenacity and diligence far beyond what is normal or sane. They must have the conviction and tenacity to insist that every test case be faithfully executed, even if it’s “impossible” that this-or-that change would have caused a regression. When they see something in the product that doesn’t look right, they must get to the bottom of it, carefully catalog the event, and enter it into the bug queue. They help banish the skeletons by exposing them to the light, no matter where they may try to hide.

Many a programmer, close to a deadline, has remarked with exasperation, Jeez, %$!@# QA keeps making me revisit this thing, I fixed that already!, only to find, when they go through the steps in the bug report, that yes, it is indeed broken.

While they may seem difficult or inefficient or outright stubborn at times, they’re vital. Their wisdom is a balance to the programmers’ wit, and they keep teams from burning up. Budget plenty of time for QA. Start it early. Get someone who knows what they’re doing, and use a proper bug tracking system. Your customers won’t thank you for it, but they’ll certainly come to hate you if you don’t.

Druid

Free from any affiliations or alignment, the druid transcends the simple human existence to become one with the natural world. Unlike the clerics who request their power from a deity, the druid opens themselves up to be nature’s instrument. It is an understatement to say that they know about the wild world; they understand the ways of the animals and plants intuitively and instinctively. Always seeking the way of balance and harmony and simplicity, they do not try to change or restrict the chaos of nature, nor do they rebel against the order that sometimes comes out of that chaos.

druid Designers are called by many names in our industry. UEDs, Visual Designers, User Interface Architects, HCI Specialists, Interaction Designers, IAs. (Indeed, these roles are somewhat different, as the task of designing the user interface has many facets.) Whatever they’re called, their job is to understand the expectations and habits of people they’ve never met, and shape the product into something that will fit the user’s mental model. At the same time, they must understand how those models change in the face of new problems and new solutions, and be able to adapt their software properly.

Like the druid, the UED must be free from any biases that may impede their ability to find the best solution for the user, even if that sometimes means that they may take issue with business objectives. (If the business goals and the user goals cannot both be met, then the product is in real trouble, anyhow!)

Part of the UED’s task is to map out the competition’s sites, to determine where this new product fits into the current ecosystem and how the target market’s expectations may have already been set. They must sometimes balance the “ideal” way of doing something with the “common” way of doing it, simply because users have been trained in a certain way.

More Similarities

A lot of the same characteristics that make up a good party are also true of a good programming team.

  1. A team should have enough members to get the job done, and no more.
  2. Each member should know his role, and enjoy doing it
  3. Balance is key. Each part of the task should be accounted for, and no one area should have more resources than is needed
  4. Druids make so-so healers; UEDs are usually pretty good at QA. A really good sorcerer might be able to handle the magic support if it’s not too intensive; most webdevs can usually throw some kind of back-end support together. But, if a certain area of the project is significant, then it’s worth having someone who knows it best.
  5. Multiclassing can make a character more diverse, but they won’t increase in levels as quickly.
  6. Treasure should be distributed fairly.
  7. If you have more than one mage (or sorcerer, or cleric, or druid, or whatever), they should either divide responsibilities or one should be in charge. Same for the roles on a programming team.
  8. It’s hard for a fighter to know how good a wizard is, or vice versa. It takes one to know one. Make sure your team helps interview new hires.
  9. Raising levels and carefully assigning skill points will lead to a more powerful character. We min-max in real life, which is why we do it in RPGs.

Like most of the essays in The Mythical Man-Month, the fundamental principles of the surgical team are valid, even if the specific roles are mostly out of date. Within each class on the adventuring party, you can sometimes see a combination of surgeon and co-pilot; sometimes the co-pilot is unnecessary, or there are two surgeons who divide up the work.

The basic idea of thinking about the programming organization in a creative way is also worthwhile, even if only as a mental exercise. I’m sure that there are other metaphors that would work equally well.

In general, I think that the motley crew of adventurers is a lot more fun than the surgical team.

Disclaimer

All images copyright © 2007 Wizards of the Coast. Please don’t sue me, WotC! We love you!

I’ve taken some liberties with the images used. The Rogue is actually a Bard. The Specialist Wizard is some kind of psionicist. The Ranger is actually a Scout. I picked more for style than accuracy.

Footnotes

[1] MySQL, PHP, Javascript, HTML, CSS, Bash, Perl, XML, yinst, regular expressions. Actually, 11 if you count Apache config directives.
[2] Unless the webdev is a business school drop-out, that is.
[3] Some teams, of course, have a dedicated business development person, or "BizDev" who handles outside relationships so that the product manager can focus on product development. I realize that I’m blurring the distinction, but I don’t think it’s a big deal.

]]>
http://foohack.com/2007/12/surgical-team-or-motley-crew-of-adventurers/feed/
Agile Scrum Sucks (but so do the alternatives) http://foohack.com/2007/11/agile-scrum-sucks-but-so-do-the-alternatives/ http://foohack.com/2007/11/agile-scrum-sucks-but-so-do-the-alternatives/#comments Mon, 12 Nov 2007 17:00:38 +0000 Isaac http://foohack.com/2007/11/agile-scrum-sucks-but-so-do-the-alternatives/ ...Read More]]> Software development is not like manufacturing.

In manufacturing, you build to the same specifications over and over again.
In software development, you create a new thing each time, and the specifications typically change along the way.

In manufacturing, 100 factories can generally produce 100 times as many things as 1 factory. Adding a new factory means that more things will be produced.
20 programmers are far less productive than 5. Adding another programmer often means that the project will be delayed even further.

In manufacturing, you can gradually tune your methods based on past experience, and become more efficient over time.
In software development, past problems generally have little if anything to do with future problems. The best you can hope for is to find and learn very broad abstract efficiency boosters like “best practices.”

Manufacturing and software development are similar in what I think of as the “expert effect.” Generally speaking, for a given task and a given group of people, one of them will be best at that task. If that expert is given control of that task, then they will tend to do it well; if they are forced to take input from everyone else in the group who isn’t as good as them, then the task will tend to suffer, if only from the time required to evaluate the input and decide not to use it. Management is one such task, perhaps the most important one.

Most people I know seem to accept these premises. That is, unless you’re an Agile Scrum zealot. Then you may think that:

  1. Specifications change, but that’s OK, because it’s “iterative design”.
  2. You can predict how long something will take, and your predictions will get better over time.
  3. Tasks can be broken down based on the time required, and can be assigned to any “qualified” person.
  4. Programming experts will be happier if they do project management, and are qualified to do so.

This is all 100% bullshit. In the real world:

  1. Specifications change because it’s virtually impossible to design something that’s never been done if you can’t see it as you go along. The specs stop changing when the team puts their collective foot down to management and the customer, and that’s when “research” stops and “development” begins. The time at which this occurs is flexible, but if you’re targeting a fixed release date, it’d better be soon! If you don’t build in enough time for research, then the product will not be satisfactory.
  2. Your estimates suck. And they’ll keep sucking, no matter what you do. You’re probably around 10-15% accurate, and if you work your ass off, you can get up to a 20-25% accuracy rate. Estimates that aren’t even remotely accurate are a waste of everyone’s time.
  3. I’m not the only one, I’m sure, who finds that at least 60% of my planned tasks for any given sprint don’t get accomplished, either because a) they depend on something else that wasn’t evident at the planning meeting, b) they just weren’t as high priority as something that came up since the planning meeting, or c) my time estimates suck, and so I budgeted way less time than required for a few of my tasks.
  4. No one likes project planning meetings. But, since project managers typically aren’t programmers dealing with the software each day, they need our input. If you’re a product manager, trust me, your team will absolutely love you if you can get these chores down a half hour or less. Streamline every part of the meeting that requires their attention. Focus their time on what you actually need them to weigh in on. Have estimates prepared, and ask them to critique them. Wrapping it in silly games won’t help—that just makes it take longer. Programmers hate project management. So make them do as little of it as possible. Trust me, if they wanted to be project managers, they wouldn’t have degrees in computer science. Let them get back to work. There’s no worse blocker than having to stop coding for a meeting. Like torture or invasive surgery, it’d better be necessary, otherwise it’s all kinds of wrong. (The “5 levels of estimates” is the Dread Cthulhu of time-wasters.)
  5. Prototypes don’t pay the bills. If you create a bunch of dummy-functional UI screens, you haven’t created software, you’ve created a demo. A prototype is an important first step, a valuable communication tool. But you can’t sell a demo, which means you can’t create demos forever. Once the “research” phase is over, demos are a liability, because they tend to obscure missing functionality.

There are a few things that scrum gets right:

  1. When it comes to estimating, often the crowd is smarter than the individual. “Planning poker,” though overly goofy and time-consuming, does a pretty good job. But it takes forever, and while I’m waiting to show my card, I’m only thinking, I’d really rather be coding right now. I’m not a project manager (that’s part of the problem with Agile Scrum) but I know there simply must be a better way.
  2. A daily meeting is crucial if there’s more than one person on the team. It’s still best if everyone can physically see the people that they’re working with from where they sit, but that’s not always possible. Even in that case, forcing everyone to stand up for a few minutes each morning and tell everyone what they’re up to is a good way to make sure that the team doesn’t get sidetracked or splintered. It keeps programmers from “going dark” and coming back a week later with something that doesn’t mesh with the rest of the product.

It seems that some Scrum Masters are die-hard zealots, and others follow a more “pick and choose” approach. The Agile methodology explicitly encourages project managers to use what works, and customize the process to meet their needs.

In other words, Agile Scrum makes a lot of ridiculous assumptions, and suggests a set of practices—some harmless, some obvious, some useful, and others specifically counter-productive—and then says, customize to your team/product/environment. If your project fails or misses a deadline, well, you didn’t do it right, or the team wasn’t committed, or the management was unreasonable. Anything at all to avoid placing blame on the methodology.

Anyone else see the similarities to a faith healer?

Ultimately, a successful project that uses Agile Scrum is successful for the same reason that any other project is successful:

  1. The management managed, and got out of the workers’ way.
  2. The workers were capable of doing what they were called on to do, and happy doing it.
  3. The product vision was something the customer wanted
  4. No one went dark. (That goes for upper management, as well!)

On my current team at Yahoo!, we use Agile Scrum, at least in name. Ostensibly, Yahoo! officially endorses the use of Agile Scrum. In my experience, successful use of agile amounts to this:

  • Every week or two, the team gets together for an hour to talk about where the project is at.
  • Every day, the team spends 15 minutes in the morning with each person telling what they’re up to.
  • Throughout the day, everyone is in constant communication via IM, email, and in person.
  • Put your tasks in Bugzilla, and mark them done when they’re done.
  • Most important: Management’s vision of the product is something that the team is happy building, and the team is selected for their expertise in the kinds of problems that the product raises.

No one likes to create crap. No one likes to be surprised with new requirements in the 11th hour. No one likes to be held responsible for something they’re no good at.

Everyone loves to be a part of a quality product. Everyone likes to have visibility into their own future. Everyone likes to be praised and respected for doing what they love.

By taking credit for the success of projects that use it, Agile Scrum glosses over these fundamental principles and tries to turn the team into a bunch of part-time project managers. Anything that deviates from the plan is derided as “cowboy coding” or “waterfall process.” Granted, these two paradigms do tend to standardize bad practices, and they do little to encourage communication, but they also admit important aspects of the software development world:

  1. Actual coding, typing characters into an editor, is best done alone without distractions. You think with your own brain only. All coding is, at some level, “cowboy coding.”
  2. If piece A depends on piece B, then it makes no sense to start working on A until B is done. It makes even less sense to QA a half-working early pre-alpha dev version of the product, or worse yet, file bugs against it. Of course it’s broken. Don’t distract your developers telling them what they already know.

Chances are wildly against getting any software product out on time or under budget. My heart goes out to project managers and executives who are tasked with controlling the chaos. I understand why many of them might grasp at ideologies that promise to solve the problems, but Scrum rarely helps the odds.

If you’re doing software development right, you’re probably doing Agile wrong.

]]>
http://foohack.com/2007/11/agile-scrum-sucks-but-so-do-the-alternatives/feed/