Great Hackers

Want to start a startup? Get funded byY Combinator.

July 2004
(This essay is derivedfrom a talk at Oscon 2004.)
A few months ago I finished a newbook,and in reviews I keepnoticing words like "provocative'' and "controversial.'' To saynothing of "idiotic.''
I didn't mean to make the bookcontroversial. I was trying to makeit efficient. I didn't want to waste people's time telling themthings they already knew. It's more efficient just to give themthe diffs. But I suppose that's bound to yield an alarming book.
Edisons
There'sno controversy about which idea is most controversial:the suggestion that variation in wealth might not be as big aproblem as we think.
I didn't say in the book that variation inwealth was in itself agood thing. I said in some situations it might be a sign of goodthings. A throbbing headache is not a good thing, but it can bea sign of a good thing-- for example, that you're recoveringconsciousness after being hit on the head.
Variation in wealthcan be a sign of variation in productivity.(In a society of one, they're identical.) And thatis almost certainly a good thing: if your society has no variationin productivity, it's probably not because everyone is ThomasEdison. It's probably because you have no Thomas Edisons.
In alow-tech society you don't see much variation in productivity.If you have a tribe of nomads collecting sticks for a fire, howmuch more productive is the best stick gatherer going to be thanthe worst? A factor of two? Whereas when you hand people a complextoollike a computer, the variation in what they can do withit is enormous.
That's not a new idea. Fred Brooks wrote aboutit in 1974, andthe study he quoted was published in 1968. But I think heunderestimated the variation between programmers. He wrote aboutproductivity in linesof code: the best programmers can solve a given problem in a tenththe time. But what if the problem isn't given? In programming, asin many fields, the hard part isn't solving problems, but decidingwhat problems to solve. Imagination is hard to measure, butin practice it dominates the kind of productivity that's measuredin lines of code.
Productivity varies in any field, but there arefew in which itvaries so much. The variation between programmersis so great that it becomes a difference in kind. I don'tthink this is something intrinsic to programming, though. In everyfield,technology magnifies differences in productivity. I think what'shappening in programming is just that we have a lot of technologicalleverage. But in every field the lever is getting longer, so thevariation we see is something that more and more fields will seeas time goes on. And the success of companies, and countries, willdepend increasingly on how they deal with it.
If variation inproductivity increases with technology, then thecontribution of the most productive individuals will not only bedisproportionately large, but will actually grow with time. Whenyou reach the point where 90% of a group's output is created by 1%of its members, you lose big if something (whether Viking raids,or central planning) drags their productivity down to the average.
Ifwe want to get the most out of them, we need to understand theseespecially productive people. What motivates them? What do theyneed to do their jobs? How do you recognize them? How do youget them to come and work for you? And then of course there's thequestion, how do you become one?
More than Money
Iknow a handful of super-hackers, so I sat down and thought aboutwhat they have in common. Their defining quality is probably thatthey really love to program. Ordinary programmers write code to paythe bills. Great hackers think of it as something they do for fun,and which they're delighted to find people will pay them for.
Greatprogrammers are sometimes said to be indifferent to money.This isn't quite true. It is true that all they really care aboutis doing interesting work. But if you make enough money, you getto work on whatever you want, and for that reason hackers areattracted by the idea of making really large amounts of money.But as long as they still have to show up for work every day, theycare more about what they do there than how much they get paid forit.
Economically, this is a fact of the greatest importance,becauseit means you don't have to pay great hackers anything like whatthey're worth. A great programmer might be ten or a hundred timesas productive as an ordinary one, but he'll consider himself luckyto get paid three times as much. As I'll explain later, this ispartly because great hackers don't know how good they are. Butit's also because money is not the main thing they want.
What dohackers want? Like all craftsmen, hackers like good tools.In fact, that's an understatement. Good hackers find it unbearableto use bad tools. They'll simply refuse to work on projects withthe wrong infrastructure.
At a startup I once worked for, one ofthe things pinned up on ourbulletin board was an ad from IBM. It was a picture of an AS400,and the headline read, I think, "hackers despiseit.'' [1]
When you decide what infrastructure to use for aproject, you'renot just making a technical decision. You're also making a socialdecision, and this may be the more important of the two. Forexample, if your company wants to write some software, it mightseem a prudent choice to write it in Java. But when you choose alanguage, you're also choosing a community. The programmers you'llbe able to hire to work on a Java project won't be assmart as theones you could get to work on a project written in Python.And the quality of your hackers probably matters more than thelanguage you choose. Though, frankly, the fact that good hackersprefer Python to Java should tell you something about the relativemerits of those languages.
Business types prefer the most popularlanguages because they viewlanguages as standards. They don't want to bet the company onBetamax. The thing about languages, though, is that they're notjust standards. If you have to move bits over a network, by allmeans use TCP/IP. But a programming language isn't just a format.A programming language is a medium of expression.
I've read thatJava has just overtaken Cobol as the most popularlanguage. As a standard, you couldn't wish for more. But as amedium of expression, you could do a lot better. Of all the greatprogrammers I can think of, I know of only one who would voluntarilyprogram in Java. And of all the great programmers I can think ofwho don't work for Sun, on Java, I know of zero.
Great hackersalso generally insist on using open source software.Not just because it's better, but because it gives them more control.Good hackers insist on control. This is part of what makes themgood hackers: when something's broken, they need to fix it. Youwant them to feel this way about the software they're writing foryou. You shouldn't be surprised when they feel the same way aboutthe operating system.
A couple years ago a venture capitalistfriend told me about a newstartup he was involved with. It sounded promising. But the nexttime I talked to him, he said they'd decided to build their softwareon Windows NT, and had just hired a very experienced NT developerto be their chief technical officer. When I heard this, I thought,these guys are doomed. One, the CTO couldn't be a first ratehacker, because to become an eminent NT developer he would havehad to use NT voluntarily, multiple times, and I couldn't imaginea great hacker doing that; and two, even if he was good, he'd havea hard time hiring anyone good to work for him if the project hadto be built on NT. [2]
The Final Frontier
Aftersoftware, the most important tool to a hacker is probablyhis office. Big companies think the function of office space is toexpressrank. But hackers use their offices for more than that: theyuse their office as a place to think in. And if you're a technologycompany, their thoughts are your product. So making hackers workin a noisy, distracting environment is like having a paint factorywhere the air is full of soot.
The cartoon strip Dilbert has alot to say about cubicles, and withgood reason. All the hackers I know despise them. The mere prospectof being interrupted is enough to prevent hackers from working onhard problems. If you want to get real work done in an office withcubicles, you have two options: work at home, or come in early orlate or on a weekend, when no one else is there. Don't companiesrealize this is a sign that something is broken? An officeenvironment is supposed to be something that helpsyou work, not something you work despite.
Companies like Ciscoare proud that everyone there has a cubicle,even the CEO. But they're not so advanced as they think; obviouslythey still view office space as a badge of rank. Note too thatCisco is famous for doing very little product development in house.They get new technology by buying the startups that created it-- wherepresumably the hackers did have somewhere quiet to work.
One bigcompany that understands what hackers need is Microsoft.I once saw a recruiting ad for Microsoft with a big picture of adoor. Work for us, the premise was, and we'll give you a place towork where you can actually get work done. And you know, Microsoftis remarkable among big companies in that they are able to developsoftware in house. Not well, perhaps, but well enough.
Ifcompanies want hackers to be productive, they should look atwhat they do at home. At home, hackers can arrange things themselvesso they can get the most done. And when they work at home, hackersdon't work in noisy, open spaces; they work in rooms with doors. Theywork in cosy, neighborhoody places with people around and somewhereto walk when they need to mull something over, instead of in glassboxes set in acres of parking lots. They have a sofa they can takea nap on when they feel tired, instead of sitting in a coma attheir desk, pretending to work. There's no crew of people withvacuum cleaners that roars through every evening during the primehacking hours. There are no meetings or, God forbid, corporateretreats or team-building exercises. And when you look at whatthey're doing on that computer, you'll find it reinforces what Isaid earlier about tools. They may have to use Java and Windowsat work, but at home, where they can choose for themselves, you'remore likely to find them using Perl and Linux.
Indeed, thesestatistics about Cobol or Java being the most popularlanguage can be misleading. What we ought to look at, if we wantto know what tools are best, is what hackers choose when they canchoose freely-- that is, in projects of their own. When you askthat question, you find that open source operating systems alreadyhave a dominant market share, and the number one language is probablyPerl.
Interesting
Along with good tools, hackerswant interesting projects. Whatmakes a project interesting? Well, obviously overtly sexyapplications like stealth planes or special effects software wouldbe interesting to work on. But any application can be interestingif it poses novel technical challenges. So it's hard to predictwhich problems hackers will like, because some becomeinteresting only when the people working on them discover a newkind of solution. Before ITA(who wrote the software inside Orbitz),the people working on airline fare searches probably thought itwas one of the most boring applications imaginable. But ITA madeit interesting byredefining the problemin a more ambitious way.
I think the same thing happened atGoogle. When Google was founded,the conventional wisdom among the so-called portals was that searchwas boring and unimportant. But the guys at Google didn't thinksearch was boring, and that's why they do it so well.
This is anarea where managers can make a difference. Like a parentsaying to a child, I bet you can't clean up your whole room inten minutes, a good manager can sometimes redefine a problem as amore interesting one. Steve Jobs seems to be particularly good atthis, in part simply by having high standards. There were a lotof small, inexpensive computers before the Mac. He redefined theproblem as: make one that's beautiful. And that probably drovethe developers harder than any carrot or stick could.
Theycertainly delivered. When the Mac first appeared, you didn'teven have to turn it on to know it would be good; you could tellfrom the case. A few weeks ago I was walking along the street inCambridge, and in someone's trash I saw what appeared to be a Maccarrying case. I looked inside, and there was a Mac SE. I carriedit home and plugged it in, and it booted. The happy Macintoshface, and then the finder. My God, it was so simple. It was justlike ... Google.
Hackers like to work for people with highstandards. But it's notenough just to be exacting. You have to insist on the right things.Which usually means that you have to be a hacker yourself. I'veseen occasional articles about how to manage programmers. Reallythere should be two articles: one about what to do ifyou are yourself a programmer, and one about what to do if you're not.And thesecond could probably be condensed into two words: give up.
Theproblem is not so much the day to day management. Really goodhackers are practically self-managing. The problem is, if you'renot a hacker, you can't tell who the good hackers are. A similarproblem explains why American cars are so ugly. I call it thedesign paradox. You might think that you could make yourproductsbeautiful just by hiring a great designer to design them. But ifyou yourself don't have goodtaste,how are you going to recognizea good designer? By definition you can't tell from his portfolio.And you can't go by the awards he's won or the jobs he's had,because in design, as in most fields, those tend to be driven byfashion and schmoozing, with actual ability a distant third.There's no way around it: you can't manage a process intended toproduce beautiful things without knowing what beautiful is. Americancars are ugly because American car companies are run by people withbad taste.
Many people in this country think of taste assomething elusive,or even frivolous. It is neither. To drive design, a manager mustbe the most demanding user of a company's products. And if youhave really good taste, you can, as Steve Jobs does, make satisfyingyou the kind of problem that good people like to work on.
NastyLittle Problems
It's pretty easy to say what kinds ofproblems are not interesting:those where instead of solving a few big, clear, problems, you haveto solve a lot of nasty little ones. One of the worst kinds ofprojects is writing an interface to a piece of software that'sfull of bugs. Another is when you have to customizesomething for an individual client's complex and ill-defined needs.To hackers these kinds of projects are the death of a thousandcuts.
The distinguishing feature of nasty little problems is thatyoudon't learn anything from them. Writing a compiler is interestingbecause it teaches you what a compiler is. But writing an interfaceto a buggy piece of software doesn't teach you anything, because thebugs are random. [3] So it's not just fastidiousness that makes goodhackers avoid nasty little problems. It's more a question ofself-preservation. Working on nasty little problems makes youstupid. Good hackers avoid it for the same reason models avoidcheeseburgers.
Of course some problems inherently have thischaracter. And becauseof supply and demand, they pay especially well. So a company thatfound a way to get great hackers to work on tedious problems wouldbe very successful. How would you do it?
One place this happensis in startups. At our startup we hadRobert Morris working as a system administrator. That's like having theRolling Stones play at a bar mitzvah. You can't hire that kind oftalent. But people will do any amount of drudgery for companiesof which they're the founders. [4]
Bigger companies solve theproblem by partitioning the company.They get smart people to work for them by establishing a separateR&D department where employees don't have to work directly oncustomers' nasty little problems. [5] In this model, the researchdepartment functions like a mine. They produce new ideas; maybethe rest of the company will be able to use them.
You may nothave to go to this extreme.Bottom-up programmingsuggests another way to partition the company: have the smart peoplework as toolmakers. If your company makes software to do x, haveone group that builds tools for writing software of that type, andanother that uses these tools to write the applications. This wayyou might be able to get smart people to write 99% of your code,but still keep them almost as insulated from users as they wouldbe in a traditional research department. The toolmakers would haveusers, but they'd only be the company's own developers. [6]
IfMicrosoft used this approach, their software wouldn't be so fullof security holes, because the less smart people writing the actualapplications wouldn't be doing low-level stuff like allocatingmemory. Instead of writing Word directly in C, they'd be pluggingtogether big Lego blocks of Word-language. (Duplo, I believe, isthe technical term.)
Clumping
Along withinteresting problems, what good hackers like is othergood hackers. Great hackers tend to clump together-- sometimesspectacularly so, as at Xerox Parc. So you won't attract goodhackers in linear proportion to how good an environment you createfor them. The tendency to clump means it's more like the squareof the environment. So it's winner take all. At any given time,there are only about ten or twenty places where hackers most want towork, and if you aren't one of them, you won't just have fewergreat hackers, you'll have zero.
Having great hackers is not, byitself, enough to make a companysuccessful. It works well for Google and ITA, which are two ofthe hot spots right now, but it didn't help Thinking Machines orXerox. Sun had a good run for a while, but their business modelis a down elevator. In that situation, even the best hackers can'tsave you.
I think, though, that all other things being equal, acompany thatcan attract great hackers will have a huge advantage. There arepeople who would disagree with this. When we were making the roundsof venture capital firms in the 1990s, several told us that softwarecompanies didn't win by writing great software, but through brand,and dominating channels, and doing the right deals.
They reallyseemed to believe this, and I think I know why. Ithink what a lot of VCs are looking for, at least unconsciously,is the next Microsoft. And of course if Microsoft is your model,you shouldn't be looking for companies that hope to win by writinggreat software. But VCs are mistaken to look for the next Microsoft,because no startup can be the next Microsoft unless some othercompany is prepared to bend over at just the right moment and bethe next IBM.
It's a mistake to use Microsoft as a model, becausetheir wholeculture derives from that one lucky break. Microsoft is a bad datapoint. If you throw them out, you find that good products do tendto win in the market. What VCs should be looking for is the nextApple, or the next Google.
I think Bill Gates knows this. Whatworries him about Google isnot the power of their brand, but the fact that they havebetter hackers. [7]
Recognition
So who are the great hackers? How do you knowwhen you meet one?That turns out to be very hard. Even hackers can't tell. I'mpretty sure now that my friend Trevor Blackwell is a great hacker.You may have read on Slashdot how he made hisown Segway. Theremarkable thing about this project was that he wrote all thesoftware in one day (in Python, incidentally).
For Trevor, that'spar for the course. But when I first met him, I thought he was acomplete idiot. He was standing in Robert Morris's office babblingat him about something or other, and I remember standing behindhim making frantic gestures at Robert to shoo this nut out of hisoffice so we could go to lunch. Robert says he misjudged Trevorat first too. Apparently when Robert first met him, Trevor hadjust begun a new scheme that involved writing down everything aboutevery aspect of his life on a stack of index cards, which he carriedwith him everywhere. He'd also just arrived from Canada, and hada strong Canadian accent and a mullet.
The problem is compoundedby the fact that hackers, despite theirreputation for social obliviousness, sometimes put a good deal ofeffort into seeming smart. When I was in grad school I used tohang around the MIT AI Lab occasionally. It was kind of intimidatingat first. Everyone there spoke so fast. But after a while Ilearned the trick of speaking fast. You don't have to think anyfaster; just use twice as many words to say everything.
Withthis amount of noise in the signal, it's hard to tell goodhackers when you meet them. I can't tell, even now. You alsocan't tell from their resumes. It seems like the only way to judgea hacker is to work with him on something.
And this is the reasonthat high-tech areasonly happen around universities. The active ingredienthere is not so much the professors as the students. Startups grow uparound universities because universities bring together promising youngpeople and make them work on the same projects. Thesmart ones learn who the other smart ones are, and togetherthey cook up new projects of their own.
Because you can't tell agreat hacker except by working with him,hackers themselves can't tell how good they are. This is true toa degree in most fields. I've found that people whoare great at something are not so much convinced of their owngreatness as mystified at why everyone else seems so incompetent.
Butit's particularly hard for hackers to know how good they are,because it's hard to compare their work. This is easier in mostother fields. In the hundred meters, you know in 10 seconds who'sfastest. Even in math there seems to be a general consensus aboutwhich problems are hard to solve, and what constitutes a goodsolution. But hacking is like writing. Who can say which of twonovels is better? Certainly not the authors.
With hackers, atleast, other hackers can tell. That's because,unlike novelists, hackers collaborate on projects. When you getto hit a few difficult problems over the net at someone, you learnpretty quickly how hard they hit them back. But hackers can'twatch themselves at work. So if you ask a great hacker how goodhe is, he's almost certain to reply, I don't know. He's not justbeing modest. He really doesn't know.
And none of us know,except about people we've actually workedwith. Which puts us in a weird situation: we don't know who ourheroes should be. The hackers who become famous tend to becomefamous by random accidents of PR. Occasionally I need to give anexample of a great hacker, and I never know who to use. The firstnames that come to mind always tend to be people I know personally,but it seems lame to use them. So, I think, maybe I should sayRichard Stallman, or Linus Torvalds, or Alan Kay, or someone famouslike that. But I have no idea if these guys are great hackers.I've never worked with them on anything.
If there is a MichaelJordan of hacking, no one knows, includinghim.
Cultivation
Finally, the question the hackershave all been wondering about:how do you become a great hacker? I don't know if it's possibleto make yourself into one. But it's certainly possible to do thingsthat make you stupid, and if you can make yourself stupid, youcan probably make yourself smart too.
The key to being a goodhacker may be to work on what you like.When I think about the great hackers I know, one thing they havein common is the extremedifficultyof making them workon anything theydon't want to. I don't know if this is cause or effect; it may beboth.
To do something well you have tolove it.So to the extent youcan preserve hacking as something you love, you're likely to do itwell. Try to keep the sense of wonder you had about programming atage 14. If you're worried that your current job is rotting yourbrain, it probably is.
The best hackers tend to be smart, ofcourse, but that's true ina lot of fields. Is there some quality that's unique to hackers?I asked some friends, and the number one thing they mentioned wascuriosity.I'd always supposed that all smart people were curious--that curiosity was simply the first derivative of knowledge. Butapparently hackers are particularly curious, especially about howthings work. That makes sense, because programs are in effectgiant descriptions of how things work.
Several friends mentionedhackers' ability to concentrate-- theirability, as one put it, to "tune out everything outside their ownheads.'' I've certainly noticed this. And I've heard severalhackers say that after drinking even half a beer they can't program atall. So maybe hacking does require some special ability to focus.Perhaps great hackers can load a large amount of context into theirhead, so that when they look at a line of code, they see not justthat line but the whole program around it. John McPheewrote that Bill Bradley's success as a basketball player was duepartly to his extraordinary peripheral vision. "Perfect'' eyesightmeans about 47 degrees of vertical peripheral vision. Bill Bradleyhad 70; he could see the basket when he was looking at the floor.Maybe great hackers have some similar inborn ability. (I cheat byusing a verydenselanguage,which shrinks the court.)
This could explain the disconnect overcubicles. Maybe the peoplein charge of facilities, not having any concentration to shatter,have no idea that working in a cubicle feels to a hacker like havingone's brain in a blender. (Whereas Bill, if the rumors of autismare true, knows all too well.)
One difference I've noticedbetween great hackers and smart peoplein general is that hackers are morepolitically incorrect.To theextent there is a secret handshake among good hackers, it's when theyknow one another well enough to express opinions that would getthem stoned to death by the general public. And I can see whypolitical incorrectness would be a useful quality in programming.Programs are very complex and, at least in the hands of goodprogrammers, very fluid. In such situations it's helpful to havea habit of questioning assumptions.
Can you cultivate thesequalities? I don't know. But you can atleast not repress them. So here is my best shot at a recipe. Ifit is possible to make yourself into a great hacker, the way to doit may be to make the following deal with yourself: you never haveto work on boring projects (unless your family will starve otherwise),and in return, you'll never allow yourself to do a half-assed job.All the great hackers I know seem to have made that deal, thoughperhaps none of them had any choice in the matter.
Notes
[1] In fairness, I have to say that IBM makes decent hardware. Iwrote this on an IBM laptop.
[2] They did turn out to be doomed.They shut down a few monthslater.
[3] I think this is what people mean when they talkabout the "meaning of life." On the face of it, this seems anodd idea. Life isn't an expression; how could it have meaning?But it can have a quality that feels a lot like meaning. In a projectlike a compiler, you have to solve a lot of problems, but the problemsall fall into a pattern, as in a signal. Whereas when the problemsyou have to solve are random, they seem like noise.
[4] Einstein atone point worked designing refrigerators. (He had equity.)
[5]It's hard to say exactly what constitutes research in thecomputer world, but as a first approximation, it's software thatdoesn't have users.
I don't think it's publication that makes thebest hackers want to workin research departments. I think it's mainly not having to have athree hour meeting with a product manager about problems integratingthe Korean version of Word 13.27 with the talking paperclip.
[6]Something similar has been happening for a long time in theconstruction industry. When you had a house built a couple hundredyears ago, the local builders built everything in it. But increasinglywhat builders do is assemble components designed and manufacturedby someone else. This has, like the arrival of desktop publishing,given people the freedom to experiment in disastrous ways, but itis certainly more efficient.
[7] Google is much more dangerous toMicrosoft than Netscape was.Probably more dangerous than any other company has ever been. Notleast because they're determined to fight. On their job listingpage, they say that one of their "core values'' is "Don't be evil.''From a company selling soybean oil or mining equipment, such astatement would merely be eccentric. But I think all of us in thecomputer world recognize who that is a declaration of war on.
Thanksto Jessica Livingston, Robert Morris, and Sarah Harlinfor reading earlier versions of this talk.