Mea Culpa

Reflecting upon my previous post, I am wondering why LISP triumphalists like Paul Graham annoy me so much? Perhaps it is because I used to be one myself, in spirit if not in syntax. And also because I now see them as a major symptom of what ails programming.

It is an easy trap to fall into. Programming requires a certain kind of analytical intelligence. Being more intelligent in that way increases your programming ability exponentially. It is emotionally satisfying to think of yourself as a different species from the average programmer. Programming becomes a demonstration of your superior intellect. Surely such powers shouldn’t be wasted on mundane chores, but instead applied to timeless works of brilliant inspiration, to be admired by the common programmer only through protective eyewear. What a load of self-indulgent adolescent crap. In programming as in the rest of life, attitude trumps intelligence. I had to learn that the hard way.

My road to perdition was the seemingly noble cause of doing computer science research in the real world. It is in the crucible of practical problems that great new ideas can form. Unfortunately the result was a strange brew of triumph and disaster. I built some systems that were uniquely powerful and far ahead of their time. Yet they were also wildly idiosyncratic, and deeply flawed in some practical aspects. I inflicted a legacy of pain upon the programmers who had to work with my misbegotten creations for decades afterwards. I too was afflicted. I spent those decades debugging the Frankenstein. In the OS crash debugger. In hex. Because wasn’t it oh so clever and efficient to build a radically new kind of database with extreme reliability requirements as a highly multithreaded kernel extension to the OS?

What may have saved my soul is that I did not sell the company and walk away from my mess. I stuck with it for 20 years, and eventually cleaned up the mess as best I could with a massive migration to a new framework. That experience taught me a lot about what really matters in programming. It is not about solving puzzles and being the brightest kid in the class. It is about realizing that the complexity of software dwarfs even the most brilliant human; that cleverness cannot win. The only weapons we have are simplicity and convention. Tattoo that on your forehead in reverse so that you always see it reflected in the screen. What is truly decisive on the battlefield are attitudes: hard work, responsibility, and paying attention to reality instead of the voiceover in your head.

Programming is an embarrassment compared to other fields of engineering and design. Our mainstream culture is one of adolescent self-indulgence. It is like something from Gulliver’s Travels, with the curly-bracketeers vs. the indentationites vs. the parenthesesophiles. The only thing that everyone seems to agree upon is how stupid all the other programmers are. Try googling “stupid programmers”. We have met the enemy, and he is us.

Programming will not grow up until our culture grows up. We can only patiently and persistently do our part to elevate the level of discourse, and share what wisdom we have gained.

P.S. I deleted a bunch of the cursing, but I think I’ll let the rest stay as a testament to my point.

125 Replies to “Mea Culpa”

  1. For a beginning programmer such as myself this post is spot on with many of my feelings and my approach of learning to code. Thank you very much for the validation from somebody with so much experience in the field and trenches 🙂

  2. Why do you think it is that the programmers you describe are all about fancy “highly productive” languages, but don’t want a structured editor?

    1. That is a very good question Jake. By “structured editor” do you generally mean IDE’s like Eclipse, or the stricter meaning of directly editing S-exps or AST’s rather than ASCII? True structured editors go back to InterLisp, which for various reasons was displaced by the MIT-centric text-based LISP environments. Some people still do structure editing of LISP in Emacs.

      But on the matter of Eclipse-like IDE’s, I agree with your observation. People who like mathematically-flavored programming languages don’t like the taste of Eclipse. They also scoff at debuggers. I don’t have any ready-made theories to explain that.

      1. This always boggled my mind. I love all kinds of languages, but there’s a reason we have GUIs now, they ease a lot of the cognitive load. Also, hunting around with print statements starts being very hit or miss in a complex system, yet people don’t use the debugger.

        1. Many people overvalue a debugger. Frequent use of a debugger is a practice smell IMHO. I find that primitive print statements and a debugger are both tools in my toolbox but I rarely have to fall back to either if I force myself to write simple and clean code. The best way I’ve found to achieve that is with TDD. My 2c.

      2. I love mathematically-flavored languages (e.g: Haskell) and really wish it had better debuggers… I like debuggers very much.

      3. I was talking more about directly editing the tree rather than using a fancy version of notepad. I’m sure there are people that do like both lisp and structured editors, perhaps even a higher percentage than people who don’t like lisp, but apparently not enough to make it mainstream. Heck, even smalltalk isn’t really a structured editor in this sense. They stopped just short of that.

        I wonder if it’s about ego. If you can say “Look at this really complicated thing I made with this difficult to understand language using only duct tape and chewing gum for tools!” then you win. GUIs are for users.

        1. paredit mode is quite popular within the lisp community. Anyway, lisps aren’t “difficult to understand” languages…

  3. But why is computer science/software engineering this way? Surely there cannot be that much difference in types of people that go into say, structural engineering or electrical engineering or even biology and the people that go into computer science. Is it just that software developers are allowed to get away with the behavior and others are not?

    1. My theory for this is that there’s so much demand for software to be written that it’d be crazy expensive to try and write all software using high quality engineers with sufficient time allocation to make quality software. People aren’t willing to spend enough on high quality software to justify it. In many cases people would rather buy a $30 app that might crash here and there and be a little less intuitive vs a $100 app that is of high quality. There are examples on the other extreme to this that prove the point like the software that runs on the shuttle/space station where they are willing to spend tons extra to ensure the quality of the software.

      Also, the majority of software doesn’t kill someone if it crashes unlike a structural engineer’s products.

      It could also be that software engineering hasn’t been around enough to become as formalized as other sciences.

      1. The perfect is the enemy of the good.

        The embarrassment, as far as I can tell, is not that software should be nearly flawless (presumably what you mean by ‘high quality’, given the NASA reference), it’s that folks think that being clever alone will deliver ‘high quality’ applications.

        If you think of software in a Zero Defect, Six Sigma, or TQM sense, you’re going to be perpetually disappointed, and/or find yourself going down the very dark, very deep ‘provable code’ rabbit hole. Great for NASA, not so great for the billions of other places that software development is necessary.

        The real problem, though, is that programming is fun! 🙂 When you enjoy doing software development, it’s hard to not inject a little cleverness into your code; it’s almost an expression of joy. Sure, it nearly always comes back to bite, but it’s hard to avoid it up front. You need to recognize the ‘cleverness’, note it, and come back later to clean it up. If you keep from doing it in the first place, you’ll get bored, and nothing is worse for creativity than feeling bored with what you’re doing.

        The other side, unfortunately, is that folks who don’t think of programming as creative work, or fun (and therefore are less likely to inject cleverness) inevitably are also less ‘productive’ (that is, they may generate more LOC in the same time, but it does less).

        I’m a selfish bastard; I hope that software development never becomes unfun in my lifetime. 🙂

        1. I’ll second the fun comment. The other aspect of why Software Development isn’t as refined and defined as some other branches of engineering is pretty simple. There’s always another way to program something, but a wheel is always round.

          As analytical as you need to be in order to program well, you always have the opportunity to try something new, do it a little differently, do it faster, cleaner, prettier.

          Please note how subjective all of those words are, and you instantly see why we really enjoy our work. And how ten programmers will design a program ten different ways.

          1. And 1,000 users will cuss you because you have so very little real world experience using said crap.

    2. Software development is WAY different from the other engineering fields. Just think about it, before someone makes a CPU, they spend a vast amount of time in *DESIGN*. When a company makes a circuit board, CPU, etc, making one prototype is VERY expensive. This FORCES those forms of engineering into doing detailed design. They work out huge number of issues in the design phase, then implementation is just a mechanical process.

      With software development, design is a bad word. Software folks just want to code, code, code, we will fix issues later, we will work it out as we go. A big part of the problem is that we don’t have the tools yet to do the detailed design, where the electrical engineer does have the tools. We also like the plumbing too much. To many programmers think their screw or nut (collection, or sort algorithm) is better then the other dudes, so we reinvent the wheel, too much. Maybe not at the collection/sort algorithm level, but details that should be considered plumbing we keep reinventing rather then using what is already there.

  4. Software is hard. Get used to it.

    Some problems cannot be solved without intelligence.
    Other problems cannot be solved without iron discipline.

    None of the big projects can be completed without the right attitude.

    1. Hmm; I like this comment a LOT.

      A lot of software failures come from using only one approach, where the other was necessary.

  5. I think that seeking simplicity is something that comes from mastery. First people need to prove themselves, and the measure of the beginner is cleverness. You’re an outlier.

    But when that phase passes, the real mastery implies going ahead. And it is then that you apply what for me is real/mature cleverness, because the right combination of simplicity, having an intuition for unnecessary complexity, balance, beauty, and pragmatism, is in general not easy at all, and not reachable with less experience.

    There’s a zen tale that depicts the evolution of a student where he perceives a mountain with several levels of sophistication. When the student becomes an old zen master, the mountain it is just a mountain.

    I love that tale, you can find that path in many human activities.

    1. There is a difference between clever code and obfuscated code. Clever code is often the simpler and shorter code but not necessarily obvious, where as brute force code can be easy to understand, but actually be more complex and longer in length. Its like the difference between a full equation and it simplified form.

  6. Underengineering is as much or even more of a problem than overengineering is. Your case is pretty extreme (and it does sounds pretty bad), but for every such case there are a thousand cases where coders never thought a little bit further than was directly required of them, and wrote these horrible systems where the complexity creeps in because things weren’t thought through in the first place and anything new has to be implemented as some workaround or another.

    1. I can’t say I’ve seen much of that. What I have observed is that disasters tend to happen in the specifications, especially when programmers are not involved in them.

    2. Jonathan says he’s not seen many underengineered systems. I see them all the time, and therefore I think Eelco’s comment is spot on.

      I’m talking about systems where programmers always take the “easiest route” and hack more spaghetti everywhere, where the codebase is 20x bigger than you would expect from the problem domain, where all new features or bug fixes take ages.

      I also know what Jonathan is talking about, I’ve been involved with a system which was so overengineered that we just debugged the unnecessary and overcomplicated frameworks all the time. And those monsters never became stable or predictable, eventually they were just thrown away.

      There has to be a balance between these two forces. Just saying that “clever” is always wrong just gives justification to all those guys who write unmaintainable, Big Balls of Mud.

      1. I’d say it’s unfair to call a system that does only what it is required to do at any given time ‘under engineered’. If that system morphs into a giant ball of mud as new requirements come along then the blame lies with those who extended the system. Now, if it was a ball of mud after it’s early iteration where it did what it was supposed to and no more that’s another story.

        1. Building a system that ‘only does what is required’ sounds naive. In practice, no matter how hard you tried nailing all the requirements down upfront, you’re going to have to change some assumptions, because the requirements weren’t understood properly by all parties, or what looked good on paper doesn’t work all that well in practice, etc. Not being prepared to change things around and simply blaming ‘the business for messing up again’ as is common practice doesn’t cut it imho.

          Anyway, the kinds of underengineering I’m talking about are pretty simple. E.g. people start with a ‘created date’ for entity x, then add a date for ‘uploaded’, then add a field ‘status’ for ‘enqueued for reviewing’, ‘accepted’, ‘reviewed’, then add columns with dates for those statusses, and all the while when they found out they need to just support that next little workflow requirement, they are creating a big pile of dung-spaghetti that no only is inflexible, but also overly complicated to maintain and at some point hardly anyone understands what is going on. Proper engineering would’ve meant someone recognizing that, hey, in generic form, we have an entity that can be in different states, and each of these states have a transition date, etc; let’s put that in a separate table at the very least. Just doing that little upfront thinking would’ve saved a lot of time slapping stuff on later, and led to a better quality of the software, plain and simple. And this is just one example out of many I see on a pretty regular basis.

        2. James: I agree with you. What I’m afraid of with this “clever, bad” talk and stating that Paul Graham et.al. are promoting overengineered, rocket science systems is that this can turn into an excuse to make a mess. I can just see how the 9-to-5 developers who do not have any passion for this profession can use it as a counterargument if you e.g. try to tell them that duplicating code all over the place is bad and we should refactor the design.

    3. If you ever want to know how well a house is built, grab the newel post at the bottom of the stairs and give it a good shake. If you cannot move it, you know the carpenter who built that may be as much as 100 or 200 years ago had the experience and was given time to do it right.

      Project management makes all the difference in the success, maintainability and extensibility of a project. A good project manager defines the processes in place to ensure high-quality, communicates effectively information to developers, mentors less experienced engineers and defines what the project actually is. It is my experience that software engineers are not intellectually lazy but do what they do when they do with the information at hand because that is the way to solve what they understand the problem to be at that point in time given the constraints management has imposed. We all want to leave a newel post that is stiff and not movable for the next guy. I know we have all done things knowing perfectly well we are creating an unmaintainable and un extendable boondoggle but have no choice because of the deadline looming. The simple and elegant often eludes most dedicated engineers because of this.

    4. I think that the under-engineering happens when folks forget to

      “Begin with the end in mind”

      in other words, think and envision what the end product should look like externally and internally. Think through to the end, not just the next step. It doesn’t need to be a 100% complete, spec’ed version, just a solid vision.

  7. I don’t think I agree with most of the things written in this post. I say most since I want to be polite – infact I can’t think of ANYTHING I agree with in this post.

  8. I’ve definitely been on the mental masturbation side of things and also seen my fair share of collosal fuckups. To me the least common denominator for code quality is comprehensibility.

    Comprehensibility is language-independent. You can spew a mountain of nonsense in French just as well as Spanish or Russian. The best a language or framework can do to aid you in creating comprehensible code is to help keep your application logic well organized. But if you want to do anything interesting, you’re eventually going to have to color outside the lines. Do so with intention and self-awareness.

  9. I agree with your statement “The only weapons we have are simplicity and convention”.

    Back in 1968, Edsger Dijkstra wrote a letter to the Communications of the ACM titled Go To Statement Considered Harmful. His letter is an example of a convention that most modern programming now embody.

    We need more examples of conventions and how to make things simple and help programmers learn and apply these. This the case in other engineering disciplines. Isn’t it a good idea to do the same in software engineering?

      1. What do you mean by “complete”? Many languages do not have “goto” at all..

        1. Many of the languages that “don’t have goto at all” have simply disguised it. It’s pretty straightforward to look at exceptions as cross-procedural goto.

          1. Well, a limited form of “goto” (including loops, even function calls) is not “goto”.

            Many languages got rid of the unchecked (harmful) power of “goto”. When you say it is impractical to get rid of it, it sounds like you imply we’ll be stuck with the problematic powerful goto forever.

          2. The fact that a given feature can be accomplished with GOTO shouldn’t be evidence against the features correctness (I think there are other arguments against exceptions’ correctness). Ex: a WHILE could be done with an if and a goto, but that doesn’t make it less correct.

            I think GOTO is a hint that we are missing degrees of freedom in traditional imperative languages. Exceptions have been introduced to try and address some of them, but I think there are still holes, notably in moderately complicated multi-condition logic flows that share bits of code for different sets of conditions. I think the problem there is we are tied to ASCII based trees (yay schematic tables!).

          3. Exceptions are clearly not ‘GOTO’. They are ‘GETOUTOFHERE’ and *MUCH* less coupling than ‘GOTO’.

    1. I wonder if the Dijkstra article is really about the actual language construct GOTO or if it’s more about not constructing a spaghetti mess using random jumps without any discipline or convention.

      You can still GOTO by returning from any arbitrary point in a function.

  10. I’d like to add that this is not just the problem with computer science, but also with a wide array of other fields like electronics. People complicate stuff too much.

    When you said, “The only weapons we have are simplicity and convention,” you were spot on.

    Einstein also said,

    Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage — to move in the opposite direction.

    The programmers we normally consider intelligent may really not be so. That’s why your post title, “Attitude triumphs intelligence” is a little misleading. If a programmer cannot solve a programming problem in a straightforward manner, employing the best solution, he is not intelligent, regardless of the amount of clever hacks he knows.

  11. A very interesting read, and one I agree with in most points. Software sucks, and as programmers, we (mostly) suck. But to be fair, I think we’re doing a lot better than any other engineering discipline did 60 years in. And imagine developing civil engineering in a world where the equations of force distribution are constant, but the steel changes every 5 years, and in 10 year the chasm across which you’ve built a bridge is now twice as wide, or is a gently sloping hillside criss-crossed with scenic roadways?

    All these things give me optimism though, because there’s people doing work like yours, or the guys at VPRI, or Intentional, and probably countless others I know nothing about – addressing the core issues, and looking for more of the things that don’t change, and fundamental shifts in tooling. There’s a reason we don’t use a blacksmith forge to make I-Beams, which is what we’re doing with our silly little text editors and text-indexers disguised as IDEs.

    In my opinion, anybody writing application software in LISP nowadays would be a fool, but anybody wanting to be all he can be in our profession should definitely understand and study it, simply by virtue of it being almost but not quite the “Maxwell’s equations” of modern procedural (as in, call-wait-result, rather than non-OO) programming.

    Admin: the proxy-blocker is understandable, but frustrating (at work). Go-go gadget Mifi!

  12. One thing I felt the indentationites and the parenthesesophiles did was to assume that their way was easier and “more natural.” I don’t think their goal was to show off how you could never get to their advanced level, but to “show off” in the sense that you could learn from them and it was the breadth of their vision that brought you there. To put it another way, they wanted to feel special by being there first.

    P.S. The voiceover in my head sounds like Morgan Freeman, so I pay attention to it often.

  13. Agreed 100%, I only wish my egotistical lecturers who taught me C/C++/Java in first and second year at Computer Science had of communicated this. Rather than putting all of our code through Binary I/O checkers, so we were either right or wrong, nothing in between.
    The reality of Software Engineering in comparison to what you might initially think can be quite different.

  14. I like to laugh at people who forget that programming languages are a tool.

    ALL programming languages are identical, until you program in them. They solely exist to make solving a problem easier than it would be otherwise.

    ____________

    I’ll gladly use COBOL if I want a language that has some really nice tools for batch processing, where the only type of bug you’ll normally see is an error in business logic, and that is insanely easy to learn.

    Right now I’m using Lisp for a project where I want to keep the noise ratio down(the extra stuff that languages use to communicate to the compiler “public static void main…”), and where I can minimize the amount of XML/scripting I would have to use. However it’s only server side, the client is in Java. I imagine the traditional lisp community would be tempted to toss some tomatoes, though fresher ones than usual 😀

  15. Software engineering is what your talking about. It is hard. It is a lot of work. Some places require a lot of it. Medical, Transport, and a few others, life critical situations.

    But for most applications, a reset or hiccup is ok. For those majority of cases there are clever programmers trying clever things, most of which don’t work out in the end. But because software is so easy to distribute and patch after deployment, those cases can be fixed enough, to work well enough, most of the time.

    But here is the point, sometimes, just sometimes, a clever programmer comes up with something that is much better! And because software is easily distributed, that cleverness propagates very rapidly. And for most cases for consumers and other interests, that is exactly what they want. We don’t really care about the odd reset or hiccup if it moves use far ahead.

    A common characteristic of the best designers in any field is a sense of creativity and wonder about what is possibile. Software development has a lot more of this youthful exploration because there is a lot more room for exploration and the risks are typically low.

    I am very glad to be working in such an immature industry. Like you wouldn’t expect a new shirt to fit a child for very long, our creations for the growing industry are temporary. Plan for that, keep your head up and look around with wonder and excitement.

  16. Each commenter should post a sample of his/hers most clever lines of code. It would be interesting to see how much their comments resemble their code 🙂

    I really like Paul Graham. He’ll always be a hacker but the articles on his site clearly show his mastery not onmly in Software Engineering but also in architecture, management and entrepreneurship. His brush of choice may be Lisp but I really doubt he imposes it to the very clever kids they choose at YCombinator.

    I think all programmers are intelligent enough. Even web devs who totally skipped Computer Science are. You just can’t look at some code and try to change it if your brain is not wired right. Too many abstract unknowns.
    But it’s really a journey. There is even a movement that promotes this way of thinking about it. In an immature industry with very high personal and financial rewards for the smart starters all the kids want to be the wiz kids.
    There is almost no responsibility at first except that it should work. You have to write some code that does one thing and since you’re really smart you can do it fast. It’s almost too easy. And since you need to look smart you do it in 3 different ways and promote the one you like most. Which is always the smartest way. The distilled piece that you almost don’t know how you wrote it. The art. That happens to be also the one that is the least reusable, changeable and maintainable.
    The zen comes from increased responsibility for creating and managing large complexity. Simplicity and convention helps keeping things maintainable and reusable.
    Yes, this will take some time…

    Before enlightenment: Chop wood, carry water. After enlightment: Chop wood, carry water

  17. Beautifully written. The part about “the complexity of software” says in words something I’ve been feeling for a long time.

  18. to summarize: complexity is the enemy, and only way of conquering it is simplicity and convention. however, true simplicity is result of intelligence, and appropriate convention as well. to make things that are simple and efficient and follow convention in appropriate matter you have to be smart, *sometimes* really smart. but your post reads as if main thing that mattered would be to have discipline and good marching orders …

    remember hercules and augian stalls ? you would take a shovel, i suppose 😉

  19. Software is 8 times out of 10 about solving a problem. Whether it’s business systems, scientific applications or other. While computer science strives to produce a more efficient algorithm or appropriate language, at the end of the day the end user could not give a toss about how the software is written, apart from the fact the software does the job it was meant to do in an error free way. But by the same token not all users are built the same, and some users do have an acute understanding about various aspects of the software. The challenging part of the software frontier is producing a more efficient development methodology encompassing all aspects of the Software Development Lifecycle which captures business requirements and translates these into an almost 100 % guranteed functioning system. This is something which is neglected i think in most software courses these days i.e the bigger picture of the part software plays in the real world.

  20. btw. speaking of paul graham, silver bullets he promotes aren’t that well engineered or thought out themselves. common lisp is a hodgepodge standard designed by commitee 😉

  21. another thing that crossed my mind: i find it really funny how “software engineers” complain about immaturity, rate of failure and some sort of (imaginary) rock star appeal of their work. they should really take a good look at real world practices in building, military, financial … you name it

  22. “Programming is an embarrassment compared to other fields of engineering and design. ”
    I can not agree with this. In the domain I work as a programmer, things are like that:
    Someone builds a plant or machine – shure, complex but possible with some minor errors. Then someone “programms” the controler logic (PLC etc.) – he/she has to understand the whole machine/plant logic. -> more complex, more errors possible. Then someone has to make the thing controlable by humans – the visualization/interface programmer – he/she has to understand the whole machine, the controler logic and the user -> much more complex, much more errors possible. Depending on the machine/plant a software-crash could easily cost life.
    So what? Most serious failures I see come from bad or unsecure designed machines/plants and not from bad written software -> conclusion:
    Other fields of engineering are an embarrassment compared to programming, given their “lower” complexity and easier tangibility (when you see a machine in front of you it’s much easier to know what it does compared to a software product)
    😉 🙂 😉

  23. The have been building and engineering buildings since the beginning of time. Not so much on the software side. Another aspect that few people realize is that software engineering, particularly in the enterprise, is more related to personalities, egos and general social behaviors.

  24. Attitude may be more important than intelligence when two people are of similar levels of intelligence, but we must face the facts that without a certain base intelligence attitude isn’t going to make a whole hell of a lot of difference.

  25. That must be why many of the greatest discoveries and insights are made by geniuses and eccentrics like Newton, Tesla, Einstein, Feynman and the like.

  26. I must agree with Nestiiii on the idea that most of the complexity is due to the requirement that the application itself should make most of the decisions a human would and nicely ask it’s user to make the ones it cannot handle.
    To the question “Is it harder to design a standard building or to design a CAD application for designing buildings” the difficulty of the first is rather constant but the second would be as difficult as the number of features it provides. Meaning just how much it makes the first easier to accomplish.
    Also Joe Shawfield has something there with psychology and software engineering. The though crossed my mind more than once. Why are we the most difficult people to work with?

  27. I always tell new programmers that persistence is more important than intelligence. People who go after a problem and keep at it will last longer than brilliant quitters.

  28. What I find surprisig is the lack of the word, “discipline”, as in a “prescribed pattern of behavior” in conjunction with simplicity and convention. I take issue with “the complexity of software dwarfs even the most brilliant human”. It is not inevitable; it comes from lack of discipline. To me, it is the same sort of behavior as driving too fast, dring too much, or failing to follow the Golden Rule.

    I do agree that the culture has to grow up. When I was leaving my job as VP of Product Development as a startup in the 90’s, one of the founders gave me this piece of advice toward staying employed, “Find another software company that needs an adult.”

  29. I find this sort of comments often are completely counter to my experience, but there are whole industries in software development built upon the sort of mistaken idea that trying to understand and think through the problem to the greatest extent possible/practical before you start coding and as you go along is a mistake, just emit the first crap you can think of and who cares about later churn.

    You have engineered too much is if you spend more time solving something than it is worth, but other cases seem to be cases of incompetent engineering, trusting data and requirements before they were sufficiently developed, etc. Good engineering produces simplicity and beauty, and bad engineering or lack of engineering does not, in any true sense of the word.

    Any great simple product was generally not without extensive pain arriving at the minimalist design that can make it work, ask Steve Jobs. Misbegotten creations are the result of insufficient or misguided engineering. A “clever” hack or throws-together solution is often anything but. It is called a hack precisely because of the shallowness of the approach, incomplete engineering that do not support the long-term requirements.

  30. Simplicity is the hardest aspect to achieve.
    The best programmers create the simplest possible code, APIs, and UIs.
    Which implies, if not a kind of humility, at least a lack of techno-arrogance.

    That being said, real-world coding requires fearless code-slinging.
    The problems are complex, their factors shift as we are solving them, and there’s never enough time.
    Which implies a bit of the Viking, a bit of the Matador.

  31. Hi.

    As a student who wants to go into back-end web development what can I do to be a good programmer? I read lots of books and practice coding in my spare time, but I almost always have to go back and re-write my code, I’m OK with syntax but my code has no meaning behind it, I basically end up writing code snippets I’ve read from books, and changing variables to suit the project. I understand what OOP means but have no idea how to implement it. Do I read more books? Practice more? Google “learn OOP” and read as much as I can?

    Ideally I’d like to rewire my brain to handle programming in a smart way, even if it takes ages to get right, I’d much prefer to spend my time learning the art of programming rather than turn into a horrible programmer.

    If you had any advice at all, even if it’s vague, I think I just need a push in the right direction – it’d be appreciated.

    Thanks for this post, it has opened my eyes and inspired me to think before I code.

    Sorry for all the questions.

    1. Take on small projects and finish them, then graduate to more challenging projects.

      Forget about “OOP” for now, it has no well-defined meaning and will likely lead you in unproductive directions at this point.

      If you are reading books, I recommend SICP, but you need to do all the exercises, not just read the text, and it is difficult. The overwhelming majority of programming books are worthless.

      1. The part about useless programming books scared me but I guess I see it myself sometimes!

        I’ll work on personal projects from now on.

        SICP looks like a great read, thanks, for your advice,

    2. James, glad you found this of some interest. I haven’t done much teaching, so I am hesitant to give any strong advice. Write code that is harder than comfortable, then read other people’s code to discover new techniques. If you are just getting started, an environment like Greenfoot might work for you: build some games, study the polished examples. Have fun! If you aren’t having fun after a few weeks then you are doing it wrong. Good luck!

      1. Thanks very much, I’m going to to checkout a few open source projects a see what I can learn from then…and I’ll make sure to have fun!

  32. The problem with trying to think of software development as engineering is that we’re dealing with languages whose expressive powers are difficult to predict let alone optimize.

    I like Peter Norvig’s analogy to engineering bridges. Other “proper” engineering disciplines have a degree of certainty about the conditions and environments their bridges will be built and the materials they will be built with. Once a bridge over a river is complete, it should stay up and operate as expected or the engineer who built it will face dire consequences. And why should they not? They have the ability to study the geology of the area, the weather patterns of the region, and can put a reasonable amount of trust into the materials they select to build their bridge with. Once it’s done, the bridge should never fall down and in most cases they never do.

    A computer programmer on the other hand has no such luxuries. They have to build a bridge that can span any river in any part of the world. Their bridges have to be capable of moving from one river to another on demand. You can’t just build a bridge anymore. You have to tell a machine how to build your bridges… and you can’t even rely on having to tell the same kind of machine!

    There aren’t enough certainties in software development to be able to contrast it to the same standards that govern engineering disciplines. As much as programmers want to call themselves, “engineers,” we’re two different breeds. We have to stop thinking about it this way and find a new language and standard to govern our practice by.

  33. I think software complexity is mis-attributed to “cleverness” when it is really just lack of modularity.

    If you have modularity — it doesn’t really matter that much if any given module is “too clever” – you can just rewrite it with the proper API semantics.

    I think systems which are too complex to understand and work with simply lack modularity – and do not have parts that can be comprehended separately and fully.

    Modularity is the key to software success — “cleverness” at worst hampers our ability to understand a small implementation of a small interface (if you have modularity, all of your interfaces are rather small :-). In that case, feel free to re-write that implementation.

  34. Agreed, simplicity is what we should strive for. Some problems really are complex, and require complex solutions. However, not everything needs to be ultra-complex.

    If Rube Goldberg would look at your solution and say, “isn’t that a bit much?”, you’ve failed.

  35. Very well put.

    It reminds me of a funny quote (unsure of attribution):
    It takes twice as much intelligence to debug code as it does to write it. If you write a piece of code as clever as you can, you have no possibility of debugging it.

  36. There seems to be a pervasive belief in this thread that software development is Engineering.

    That belief is – well – an insult to Engineering.

    The field of software development is, what, 60 years old? Look at how old the field of civil construction became before it became Engineering, e.g. the two-tone cathedrals built in the 1700’s which changed architecture in mid-construction when the architect died and was replaced by another, or the experiments with gargoyles to counter-balance roof beams, etc.

    Software development attracts primadonna’s because it is still a young field with lots of room for discovery.

    Engineering is:

    – Engineering is a legal obligation. Upon graduation, Engineers are given a seal which they affix to their designs and sign, thus, accepting responsibility for their work in Law. Engineering schools require students to take a course in Tort Law.

    – Engineering is about specialization, not generalization. Engineers create a design for a specific problem / environment. OOP, UML, et al, are about generalization (aka, abstraction), hence, are heading in a direction AWAY from Engineering. (Generalization does appear in Engineering in the form of Research, but the two are well separated).

    – Engineering is about designs that use standardized methods and materials. Software development and languages (and UML) are heading AWAY from Engineering by trying to incorporate EVERY software methodology known to man, instead of culling what we know and using the simplest/best building blocks (e.g. IMO, the first thing that should be culled from Software is the notion of Call-Return, boy, did we ever take a wrong turn at Albuquerque with that one!).

    – It is my observation that most fields of real Engineering express their designs using standardized, concrete diagrams. I am of the opinion that Software will not become Engineering until we develop a way to communicate real software architectures using solid diagrams (UML ain’t even close, thanks in part to it’s kitchen sink, let’s-include-everything mentality).

    Jonathan, is your frustration a result of thinking that the primadonnas should stop prancing about and just get on with Engineering software?

    I would argue that the tools and conditions are not there yet for this.

    Q: What would we (the software industry) do if the government enacted laws akin to those applied to Engineering? E.g. the head engineer of a project must sign (literally) off on the design and accept responsibility for the work of all of his/her underlings and become personally liable, in Law, for the correct operation of the work, and be personally liable in Tort Law for any harm brought to a person by malfunction of the work? Would we use the techniques we already use? Would we cull? Would we quit the profession and not accept this responsibility? Would Microsoft apply for an exemption to the laws?

    From my perspective, we should be looking for ways (diagrammatic, yet) to allow Engineers to Engineer. I’m not particularly interested in making software development easier for non-engineers and hobbyists.

    1. There seems to be a pervasive belief in this thread that software development is Engineering. That belief is – well – an insult to Engineering.

      I could not agree more with your post. Software development is so very young and is far from being a engineering field, and we have a LONG way to go, but I think it is vital that we march down the long road, fore I think someday Software MUST become a engineering field if we are to achieve all that we can.

    2. Paul, you make some very good points. I agree that programming is in a Medieval state. I also agree that we need to mature as an engineering discipline. We have not yet discovered materials like structural steel, nor developed the theory of structural analysis that would allow us to use it. But I disagree that we should be primarily evolving into an Engineering discipline. I believe that software is a fundamentally new and different medium, and that we will need to develop an entirely new kind of discipline to work in it. I believe that programming has more to learn from the fields of Design than of Engineering. But that is another post…

      1. Jonathan,

        In many great feets of works, such as the Burj Khalifa in Dubai, there is both great design and great engineering. They are very different, but both equally important when doing large projects. On small projects, say building that shed in the back yard to hold my lawn mower and yard tools, neither is all that important when the goal is to simply “get her done”.

        It is my impression this is what happens when building something like the Burj Khalifa, the architects work the civil engineers to create the blue prints, then they hire construction workers to implement. It has been working for thousands of years with every other type of thing humans build, why would it be any different with software? What makes software so special?

        I think the real problem with software is the cost of building a program (hit that “Compile” button), or the lack there of. I don’t know the cost of build the Burj Khalifa, but I know it was a heck of a lot more then what it takes to build a program. Mind you, I am NOT saying write, create or design software, simply to build it. Because the cost to build is zero, the attitude is, “Oh, the design is wrong, we will change it and build it again, it is only software, it doesn’t cost anything, just press the button.”

        Once you have your requirements, design the system via modeling. I am sure somewhere, but somewhere in the world, there are one or more models of Burj Khalifa. Model both the UI and the backend, seperately. Then review the models with all the parties with an interested to make sure the models are correct; make sure that all parties concerned are in agreement with the fact the model meets the requirements. This is the engineering part, no code has been written, just like no actual ground has been broken at this point in building the Burj Khalifa.

        Once the design/model is correct, then hand it over to the junior level folks to implement. Just like how inspectors come in and inspect the work of those building the Burj Khalifa, code reviews are needed to make sure the implementation follows the design. No so much to count every line of code, but to make sure the general blue print/design is followed.

        Can the software be built different from the plans? Of course, every building is constructed a little different from the actual blue prints; there are always issues that need to be worked out in implementation. But in the end, the general archeture of the system should hold true to the design, if the design was good to begin with. Folks make sure the design of something like the Burj Khalifa is correct, we as software creators need to do the same.

        If we as software creators ever want to pull of feats like the Burj Khalifa, we will have to follow the same tried and true process: engineering! The quicker we get there, the faster we can do some amazing things!

    3. I agree that software development is not engineering, but I don’t believe we should or even can move towards making it an engineering discipline. Software is on a continuum between engineering/analytics on one side and art/creativity on the other. Lean too much towards one side, and you won’t have a viable product.

      The analogy I like to use is that true engineering is like Chess. It is a game of complete information. You can, at any given time, research all the variables, and make the absolute best decision at the time (similar to the analogies above of building a bridge and knowing all the physical specs of the land you are building it on). Software is more like Poker or Scrabble… games of incomplete information. You have to take the knowledge you have now, and use intuition, creativity, and past experience to make reasonable guesses about the best direction to move in.

      You won’t always get it right, and as long as have the right attitude and discipline, you will be able to change your direction. Constantly making corrections as you gain more information is a hallmark of great software creation.

      Great software requires you to deeply care about your craft, yourself, your customer, and your team members.

      1. I think we’re conflating two different things into the term “software development”.

        It’s the most obvious in the example of game design. In game design there is very clearly an art of choosing the game mechanic, the effects, etc, but once the art is done and those choices have been made I think we should strive to make the process of designing a software machine to do that work a well defined and understood engineering process. This holds true for non-game applications in making choices about UI/featuresets, etc.

        I don’t see how it’s fundamentally different from designing a mechanical machine to do a certain task. You could argue that there is art in that just as easily. A mechanical engineer could come up with a nifty hack to rotate a cog or something, but mechanical engineers don’t seem to have this conversation as much as we do.

  37. Nice post. I am a 15+ year veteran and I think we could use more of this kind of thinking.
    Programming ‘wisdom’ is too often ignored, and it devalues programming as a profession. More respect for the sort of issues you touch on would make this a more suiting long term career. As it is companies just want to hire the guys right out of school because they think smarts = wisdom.
    As in all other disciplines, it is the combination of the two that adds value, neither alone.
    Good post!

  38. Once upon a time, way back in the “Dark Ages” of programming (sometime between the instantiation of PL/I as a language and the creation of the tiny RPG punch cards), the terms “hacker” nad “hack” did not have pejorative meanings. In those by-gone days, calling someone a “hacker” was a compliment. A “hack” was often termed an “elegant” solution.

    Yes, some hacks were a bit less than obvious to the absolute novice but they were neither overly clever nor obfuscated. (For example, the fastest way to exchange the contents of 2 registers, Reg1 and Reg2, is the following sequence of Assembler statements: Reg1 XOR Reg2, Reg2 XOR Reg1, Reg1 XOR Reg2.) Invariably, they relied on simplicity of operation and straight forward logic. These were, truly, elegant solutions.

    The primary points of consideration for a piece of code to be considered “elegant” was whether or not it could be readily understood and easily maintained. I learned this from a mentor early in my career when he advised:

    If you are the only one who can understand and maintain your code, then you must realize that you can never have a vacation, get a promotion, or leave the project; so, code so that anyone can understand and maintain your code so that you are not forever chained to that one piece of code in that one project.

  39. Interesting post.

    I think the clincher is the passage from “I inflicted a legacy of pain upon the programmers who had to work with my misbegotten creations for decades afterwards … What may have saved my soul is that I did not sell the company and walk away from my mess. I stuck with it for 20 years, and eventually cleaned up the mess as best I could with a massive migration to a new framework.”

    What you committed to was, in effect, communication.

    Galileo famously wrote his works in the vernacular.

    The trouble with Lisps and the like is that they are to the “blub” languages of everyday coders as the scholarly Latin of Europe was to old Italian. There are always many great new languages, but the trick is to use the vernacular as a starting point. The key is not to write too many sentences even the basic structure of which can’t be comprehended by passers by.

    Programmers are far from the only ones to be afflicted by this: any field that requires the expression sufficiently precise specifications or sufficiently abstract structures: law, mathematics, philosophy, literary theory, tax accounting – will be.

    Where we have the advantage is that we’re the ones whose REPL turns around the fastest. There are a number of legal drafters in my family, and the bugs and exploits in their code take decades to shake out. As for the theorists and philosophers: they’ll only see the bugs if someone tells them they’re there, or takes their funding away.

  40. Thanks, Jonathan, for this awesome essay. You’re going to change a lot of minds… or if not change, at least give them uncomfortable pause.

    What people don’t seem to notice is that many of those “triumphalists” (as you so accurately call them) haven’t got the kind of experience you do. They build small things, trifling things, or they build and sell them, and don’t have to manage them afterwards. (Paul*cough*Graham.) That’s part of how they can justify their extremism.

    I was having a conversation with my one of my very best “brain” friends the other day. He was talking about how much he’d learned about himself at his new job, where he was being pushed as an engineeer, and had to interview job candidates every week. He said that, if he’d interviewed his year-ago self today, he would have vetoed his hiring. I agreed!

    The smarter a programmer is, the more he (or she) wants to work on *interesting problems*. But the problem is, one of the highest goals of programming is to eliminate interesting problems. And most people don’t actually want to kill the goose that laid the interesting egg. Their enjoyment is at odds with doing their job well.

Comments are closed.