Informality sans Mysticism

Gregor Kiczales (of Aspect Oriented Programming fame) gave a keynote talk at OOPSLA’07 titled Effectiveness sans Formality (podcast and slides). Roly Perara has posted a critique, which reminded me that I wanted to discuss it too.

Gregor’s talk triggered a bipolar reaction in the audience: some people said it was the deepest thing they had heard in ages, and others said they didn’t understand a word of it. What impressed me most was his determination to dig down to the true nature of programming, whatever the obstacles. I may quibble with his technique, but I have the highest respect for his intention, and can only hope to emulate him.

I believe that Gregor is raising the central problem of programming. I count Gregor as a fellow anti-formalist. By formalism I mean the belief that we should treat software as a mathematical theorem to be proved, or as a machine to be engineered. Dijkstra is perhaps most associated with the former idea, and Parnas with the latter. Of course every program contains a theorem, and can likewise be seen as a machine, but to view them as such is to miss their most salient features.

Gregor observes that “things are not formal at the top”. By this he means that there are no formal requirements for most software, nor can there be. Most software exists in a social context that has no formalized rules, and which continuously evolves, sometimes in reaction to the features of the software itself. The formalist enterprise presumes a fixed precise specification of requirements. But humans, and their social systems, are not formalized. What are the formal requirements for a Word Processor? How can you derive a design for paragraph style sheets from the nature of writing itself? The hard part of programming is not implementing requirements, but designing the requirements. Formalism begs the most important questions.

Gregor also says that “things are not formal at the bottom”. He quotes Brian Cantwell Smith about the inherent “slop” of the real world as compared to formalizations of it. The point here is that formalization is subjective and provisional. Gregor seems to think highly of Smith, but I found On the Origin of Objects maddening. Instead try William Kent’s Data and Reality, written 30 years ago, a clear and still-relevant discussion of the inherent subjectivity of data modeling. Kent says: “This book projects a philosophy that life and reality are at bottom amorphous, disordered, contradictory, inconsistent, non-rational, and non-objective.”

I believe that it is actually worse – software design is not just subjective – it is chaotic. Designs change abruptly and non-locally in response to tiny fluctuations in requirements, or even the programmer’s emerging understanding of the design itself. The crucial fact is that software is in general a work in progress, never finished. As such, designs never reach equilibrium. The ideals of mathematics and engineering are to optimize a design for simplicity and power. Such optimization does not make sense for a moving target, and wastes energy in design turbulence. I call this “premature design optimization”, and I believe that it is incited by the naïve formalist ideology rooted in our languages and tools.

Anti-formalism has a long intellectual history – whenever there is a movement to reduce some complex domain to be “just” some formal system, a reaction against it emerges. Modern anti-formalists often invoke emergence. The latest fashionable metaphors in software are biology and urban planning. The problem is that all these anti-formalisms are better at explaining what they are not than at offering a constructive alternative. You get a lot of metaphors and hand-waving bordering on mysticism.

This philosophical argument has been raging for centuries, but I believe that software, for the first time ever, gives us a chance to make lasting progress. That is because software is the first human-built artifact with sufficient power and flexibility to expose the problem. Biological, mental, and social systems also challenge the ideology of formalism, but we can’t build them to experiment with alternative approaches. Software for the first time lets us do Experimental Philosophy.

Metaphors and mysticism don’t build systems. This is the problem Gregor frames in his title: Effectiveness sans Formality. Effectiveness means having some way to make things work, even though we reject formalism as the account of how they should work. I paraphrase it as Informality sans Mysticism: the search for a constructive way to build informal systems. That is the central problem of programming.

Gregor gives us some examples of the sorts of informal techniques he has in mind. Read the slides and listen to the podcast. A common theme is the coexistence of multiple views, none of which is authoritative. Roly also talks about pluralism, points of view, and “a paradigm where software components are intrinsically, persistently and bidirectionally connected”. The multiple-view idea is a perennial theme of anti-formalism, to such an extent that I see it as the identifying characteristic of the movement. It is our secret handshake.

My spin is that we don’t need to invent arcane methods to build informal systems. They are already right in front of our noses, in our everyday practice. Informal systems are simply “broken” formal systems. The ideology of formalism makes value judgments about designs, endorsing such things as simplicity, power, efficiency, analyzeability. These properties are optimal – when you have a fixed final design. I claim however that when we look at designs-in-progress, these properties are in fact quite sub-optimal. When you optimize for flexibility and adaptability you want different sorts of properties to take precedence. You want to support half-baked, half-working systems. The problem is that our languages and tools are all imbued with the ideology of formalism, and often take the idealistic stance that the language should only let you state largely settled designs. Static type checking is one example of this tendency: you can’t even compile a program that doesn’t have a large part of its act already figured out (flame retardant: I love types – it is just impractically idealistic to treat a program that doesn’t type check as meaningless and unexecutable).

To uncover clues to building informal systems all we need do is observe where the practice of programming diverges from its ideology. My favorite example is copy and paste. We all know that copy and paste is immoral, illegal, and fattening. Yet we still do it. All of the time. This fact is telling us something. It says that while we should ideally refactor all our copying and pasting into some form of parametric abstraction, that just isn’t always pragmatic. My idea is to “decriminalize copy and paste” by reifying it in the IDE into a first-class language mechanism. First class copy and paste can semi-automatically propagate changes between copies, and let us mine and refactor abstractions incrementally and opportunistically. Such a construct can be seen as a “broken abstraction”, or a “proto-abstraction”, one that is optimized for flexibility rather than a fragile elegance. I have been talking about this idea for a few years now, and I really should demonstrate it, but something else always comes up.

I should admit that I pitched the copy and paste idea to Gregor, and he didn’t totally buy it. So maybe I am misreading him and we don’t actually share a common philosophy. The multi-view conspiracy could be merely my own fevered imagining. Be that as it may, I think that Gregor’s talk was important, and I look forward to seeing where he goes with it.