Alarming Development

Software is eating the world. But progress in software technology itself largely stalled around 1996. Here’s what we had then, in chronological order:

LISP, Algol, Basic, APL, Unix, C, SQL, Oracle, Smalltalk, Windows, C++, LabView, HyperCard, Mathematica, Haskell, WWW, Python, Mosaic, Java, JavaScript, Ruby, Flash, Postgress.

Since 1996 we’ve gotten:

IntelliJ, Eclipse, ASP, Spring, Rails, Scala, AWS, Clojure, Heroku, V8, Go, Rust, React, Docker, Kubernetes, Wasm.

All of these latter technologies are useful incremental improvements on top of the foundational technologies that came before. For example Rails was a great improvement in web application productivity, achieved by gluing together a bunch of existing technologies in a nicely structured way. But it didn’t invent anything fundamentally new. Likewise V8 made new applications possible by speeding up JavaScript, extending techniques invented in Smalltalk and Java. Yes, there is localized progress – for example ownership types were invented in 98 and popularized in Rust. But Since 1996 almost everything has been cleverly repackaging and re-engineering prior inventions. Or adding leaky layers to partially paper over problems below. Nothing is obsoleted, and the teetering stack grows ever higher. Yes, there has been progress, but it is localized and tame. We seem to have lost the nerve to upset the status quo. (Except Machine Learning, which has made real progress, but is also arguably an entirely different kind of software. I am talking here about human programming. )

Those of us who worked in the 70’s-90’s surfed endless waves of revolutionary changes. It felt like that was the nature of software, to be continually disrupted by new platforms and paradigms. And then it stopped. It’s as if we hit a wall in 1996. What the hell happened in 1996? I think what happened was the internet boom. Suddenly, for the first time ever, programmers could get rich quick. The smart ambitious people flooded into Silicon Valley. But you can’t do research at a startup (I have the scars from trying). New technology takes a long time and is very risky. The sound business plan is to lever up with VC money, throw it at elite programmers who can wrangle the crappy current tech, then cash out. There is no room for technology invention in startups.

Today only megacorps like Google/Facebook/Amazon/Microsoft have the money and time horizons to create new technology. But they only seem to be interested in solving their own problems in the least disruptive way possible.

Don’t look to Computer Science for help. First of all, most of our software technology was built in companies (or corporate labs) outside of academic Computer Science. Secondly, Computer Science strongly disincentivizes risky long-range research. That’s not how you get tenure.

The risk-aversion and hyper-professionalization of Computer Science is part of a larger worrisome trend throughout Science and indeed all of Western Civilization that is the subject of much recent discussion (see The Great Stagnation, Progress Studies, It’s Time to Build). Ironically, a number of highly successful software entrepreneurs are involved in this movement, and are quite proud of the progress wrought from commercialization of the internet, yet seem oblivious to the stagnation and rot within software itself.

But maybe I’m imagining things. Maybe the reason progress stopped in 1996 is that we invented everything. Maybe there are no more radical breakthroughs possible, and all that’s left is to tinker around the edges. This is as good as it gets: a 50 year old OS, 30 year old text editors, and 25 year old languages. Bullshit. No technology has ever been permanent. We’ve just lost the will to improve.

[The discussion continues at Open source is stifling progress.]

[Jan 5: Deleted then reinstated by popular demand. The argument could certainly be improved. I should at least note that there has been substantial technical progress in scaling web performance to meet the needs of the MegaTechCorps and Unicorns. The problem is that much else has languished, or actually worsened. For example small-scale business applications as were built with Visual Basic, and small-scale creative work as with Flash, and small-scale personal applications as with HyperCard. This trend is alarmingly similar to the increasing inequality throughout our society.]

My pandemic project has been to get down to solving the hard research problems needed to make Subtext real. I started with the Update Problem, which is at the heart of the imperative vs. functional programming dilemma. My last run at the problem was in Two-way Dataflow. I now have a new approach and prototyped enough of it to believe it works. It has restricted “hygienic” forms of writing through aliased pointers and triggering callback cascades that are safe from many of the usual pitfalls in imperative programming (and equally their emulation in monads and effect handlers). The big win is having updatable views, which is cleaner and more compositional than the zoo of state management frameworks engendered by reactive programming architectures.

Unfortunately I’ve concluded that I can’t publish these ideas in their current state. Neither practitioners or academics will consider such radical ideas without proof that they work in practice on realistic cases. Small contrived examples don’t cut it. TodoMVC doesn’t cut it. They’re right: extraordinary claims require extraordinary proof. I need to build out a fully working programming system creating credible applications. Research, like everything else, requires proof of work. And I don’t have graduate students to do it.

I’m just not ready to do that work now, because there is another fundamental problem that needs to be solved first: the Edit Problem. This is really a cluster of problems related to making structured editing beneficial enough to displace text editing. Actually this is a more important problem: not everyone has to update data, but everyone has to edit code.

I guess the point of this post is to help me work through the research grieving process. I am very disappointed to find myself yet again believing that I have a new solution to an important problem yet unable to communicate it to anyone. Add it to the stack and move on.

[update] Here lies Subtext 10

A colleague asks what he should tell his students when they say:

What’s the point of formalizing all of this into such a long dense 10-page paper? why can’t we just put up our code and demos on github, maybe write a blog post on it, make some youtube demos and give talks, etc.?

Tell them they don’t want to end up like me!

Continue reading “Publish or Vanish”

I submitted this abstract to PPIG’20 but have just withdrawn it. I’ve been working on this idea for the last 2 years. But lockdown has given me time to think about my research goals and it no longer seems like the right thing to be doing right now. It just isn’t going to make a dent. Leaving the abstract here as a marker.

---

Continue reading “Query by Manipulation”

Subtext 10 is well underway, but version 9 deserves some mention. I didn’t record a demo video as in the past, because I have concluded that medium fails to communicate in sufficient detail. In the future I am going to try the forms discussed in my last post. Unfortunately Subtext 9 has fallen between the cracks, and all it gets is this lousy blog post.

Continue reading “Subtext 9”