Open source is stifling progress

My previous post lamented the Great Software Stagnation. We could blame technology lock-in effects (the QWERTY syndrome). We could also blame civilization-wide decadence: the Great Stagnation that was alluded to. But a big part of the blame is something completely unique to software: the open source movement.

Open source is the ideology that all software should be free. This belief is unprecedented in the history of technology. It seems to be related to the fact that software is a form of abstract information. A lot of people seem to think music and movies should also be free, but not too many musicians or movie-makers agree. It is bizarre that open source has been promoted largely by software creators themselves.

Not much user-facing software is open source. But it has almost taken over software development tools. Building things for ourselves and other programmers tickles our nerd sensibilities. Nerd cred is a form of social status we have a shot at. Open source has certainly enabled a lot of software startups to get rich quickly building (closed source) software. But it also killed the market for development of better software tools. There used to be a cottage industry of small software tool vendors offering compilers, libraries, editors, even UI widgets. You can’t compete with free. You can’t even eat ramen on free. You get what you incent, and open source de-incentivizes progress in software tools.

Open source strongly favors maintenance and incremental improvement on a stable base. It also encourages cloning and porting. So we get an endless supply of slightly-different programming languages encouraging people to port everything over. It’s a hobby programming club that reproduces the same old crap with a slightly different style. Inventing really new ideas and building really new things is *hard*, with many trials and errors, and requires a small dedicated cohesive team. Invention can’t be crowdsourced, and it can’t be done on nights and weekends. So the only progress we get is the table scraps of the MegaTechCorps.

Open source and Unix and the internet boom are all wrapped up together. They took over around the same time, 1996, and I blame them for the lack of much progress since.

There are signs that open source is changing. Nadia Eghbal has shown a spotlight on the dark side of open source. There are attempts to convert it to a more sustainable charity model. However I believe the more fundamental change is the imposition of Codes of Conduct, which are trying to change the social norms of open source. Will it still function if it is no longer a private club for autism spectrum guys? Open source as we know it is over, for better or worse.


11 Replies to “Open source is stifling progress”

  1. Open source has its place, and that place is any software that purports to be secure, or to provide the security for some system of which it is a part.

  2. The important issue is “who owns the software you write?”.

    At a USENIX conference long ago, a programmer who had put his life into developing a next generation operating system had that effort abruptly terminated when another company bought the first company and squashed it into oblivion.

    Corporate take-over do NOT improve progress. They obliterate it.

  3. Open Source and “free” software can be really confusing. For example see Richard Stallman’s points at:

    I think the analogy to music and movie is getting closer to the real issue: copyrights. When and what can be copied? When $ gets involved, the time gets extended to very long time. In the U.S.A. 70 years after the author’s death. What about copyrights owned by corporations? When do they “die”?

    Software has another subtly: the source code vs the compiled executable code. Here a automobile analogy is often used. Only selling or sharing the executable, but not the source code, is equivalent to selling a car with the hood locked so that a mechanic cannot repair it. If a mechanic were to completely copy the engine under the hood, then sure they would be in violation of “copyright” or “patent” laws. But replacing broken components with the same or better components–that is OK. Repairing a defect in software should also be possible, without the permission of the manufacturer. Unfortunately code is very easy to copy vs copying a physical device.

    I think the solution to the the “source code” issue is something related to the intent of copyright and patent laws: they exist to create a “temporary” monopoly for the creators so that they will be rewarded for releasing their works into the public domain. Otherwise, software or recipes for how things are made could be lost.

  4. Open source may have had an impact on technical innovation but it have also enabled the development of thousands of projects that have improved healthcare, education, agricultural and industrial production, coordination and organization of all kinds of social structures and processes and many others. And of course a wide distribution of knowledge that would have been unthinkable just a few years ago. All of which has improved the live of millions of people, all of which would not have be possible if an Oracle license had been the price of admission. So I do think a wider perspective is in order.

  5. I agree there is stagnation going on. And I agree that free distribution of tools/infrastructure stifles innovation.

    I’m not convinced any of this is about open-source, Unix or the 1996 watershed. Not to say you’re wrong; I think you already know that the argument is in need of further baby/bathwater separation. I’d be interested to read an extended/updated version of it.

    Personally I think the issue is scale. All this time, scales have been changing and the system dynamics changing with it, in very non-linear ways.

    If you’re a cottage-industry player wanting to sell an innovative tool, having less-good but established competitors that are free is just one of the obstacles. They’re less good, after all — or so I’m positing — so if you can reach that point, by definition you still do have an advantage. I believe you can compete with free, if you’re substantially better. Of course it’s also getting more difficult to achieve that, for a given size of organisation.

    But back in the posited scenario, a less obvious obstacle is: who do you sell to, and how? Your customers have a certain size; they, too, used to fit in cottages but now rarely do. A friend who built a successful business selling tools told me the following. You can’t price at less than $1000 because then the purchasing decision isn’t escalated to someone high enough to drive the adoption of your tool within the organisation. When organisations were smaller, this effect did not occur… an N+1-layer pyramid produces literally a step-change over an N-layer one.

    Scale is also killing open source, for the record. Beyond a certain codebase size or rate of churn, you need to be a mega-corp to contribute to a nominally open-source project. To me it seems perverse to blame this failure on the ideals of open-source. Perhaps a better viewpoint is that open-source rhetoric hasn’t caught up with the observation that it only works at relatively small scales. Also, much software rhetoric still unthinkingly assumes that progress implies bigger scale.

  6. Thanks Stephen, those are points worth considering.

    I really don’t understand what is going on, but I have the feeling that open source is like a religion which has lost its fervor, or perhaps like late-stage Soviet Communism. Many people derive power from it and strongly espouse it, but it increasingly rings hollow and its inherent contradictions are surfacing.

  7. I agree that open-source, while in many ways the “solution” that Freed Brooks mentioned in NSB (“buy vs. build”, except for free..), is currently stifling innovation, for pretty much the same reason.

    Why spend time, effort and ultimately money on improving productivity when you can just get stuff for free?

    And in many ways, the current incarnation of both open-source and free software is at odds with, or at best doesn’t fulfil the original idea of free software: that users should be in control of the software they use, largely due to to the point that Steven made:

    “Open Source should be more about being able to tinker with well-made apps in useful ways, rather than downloading and compiling gargantuan and incomprehensible tarballs of C/C++ code.”

    For free software to be meaningful, rather than just a way for megacorps to exploit free labor, it has to become drastically smaller and more comprehensible, which is only possible if we drastically improve our ability to abstract from meaningless detail and then be very concrete in the things that matter.

  8. The reason musicians don’t share the idea music is free, is that they have no saying on the consumption of the good they produced. Open source programmers do have at least some form of control given that open source isn’t “consumed” the same as music. It is used by other programmers who need the support of the authors. That point may seem small but it actually is what makes open source (or free software) possible.

  9. I dont think open source is really as much of a problem as its tendency to gather cheerleaders who feed on themselves and enslave others while not making change easy. Its the new enterprise.

  10. “Open source is the ideology that all software should be free.”

    No, it isn’t. Open Source was specifically created as a rejection of the Stallman-esque perspective that all software should be free.

    This is no secret. In the late 90’s when ESR was giving talks promoting it, he listed the criteria for when open source made business sense. He gave examples of companies who came to him to ask how to open-source their software, and he looked at their situation and concluded that they should not.

    “This belief is unprecedented in the history of technology.”

    Where is your evidence for this claim? While the exact definition of “free” or “open” has changed with the medium, the idea that ideas should be shared is central to most advancements in many fields. For almost all of history, advancements in science or technology meant a new physical form, which could not be kept secret from competitors! The internal combustion engine was not a trade secret.

    “However I believe the more fundamental change is the imposition of Codes of Conduct, which are trying to change the social norms of open source.”

    Change, or preserve? I see only a Shirky-esque attempt to codify the norms of a group which is pushed to rapidly grow beyond its ability to teach and assimilate new members. We’re still trying to figure out how to recover from Endless September.

    “Will it still function if it is no longer a private club for autism spectrum guys?”

    I can’t tell if that’s meant derisively, but even with the most charitable interpretation, you could ask exactly the same about the software industry as a whole. You aren’t seriously suggesting that back in the 70’s and 80’s, the capitalist nerds were somehow more well-adjusted than the share-and-share-alike nerds, are you?

    “Open source as we know it is over, for better or worse.”

    You have a unique conception of what you “know” to be Open Source. Perhaps you should begin by explaining what you think it is.

Leave a Reply