The Summer of Code

The time has come to stop writing papers and start releasing code.

This last paper is a turning point. I now know enough to start building something more solid than a demo. The fundamental problem of mutable state and effects has been blocking me for years, but now I finally have an approach that fits with my other ideas. I had a sort of monadic approach in Subtext 1 but it just wasn’t good enough.

The other breakthrough for me is syntax. The paper seemed to present a textual programming language, but that was an artifice. I have come up with a syntax that is IDE-equivalent, isomorphic with IDE operations. Yet it looks like code from a distance. I think I have found where syntax belongs: in the IDE, and only in the IDE. The keyboard is a very efficient input device, so having a way to type in syntax is effective. Likewise we have highly developed mental skills for reading and understanding grammatical text. I am using text as a UI mechanism, not as a representation. The internal representation is still in terms of semantic data structures as in Subtext, and more specifically, as first-class differences between structures. Text is ephemeral, not the source. The biggest difference is revealed in names – they can be ambiguous, and it is even possible to refer to anonymous structures. Humans are really good at disambiguating by context. Computers aren’t, so the IDE will ask you to clarify ambiguous names when you enter them. There are no source files, and no compiler, only an IDE on top of a persistent data model, yet you can still read and write code textually. As I said in the conclusion of the paper, it is important to satisfy both verbal and visual thinkers.

The other win with this treatment of syntax is that it lets me decouple the IDE from the interpreter. My next step will be an interpreter with a textual REPL, as in the paper. I think of it like the console inside many games: an internal hacker’s interface. I will still need to build a fancy IDE to do cool demos. But I won’t need an IDE to play with the interpreter, and I can get it out faster.

I hope a working prototype that other people can play with will open up new avenues for communicating and collaborating. That is what people have been telling me here for years, but now I am finally ready.

14 Replies to “The Summer of Code”

  1. My basic reaction is, “well, duh”. BUT! I have to step back and realize this isn’t obvious to everyone else. In particular, your average heads-down senior programmer who bangs out solutions for companies. Or your average researcher funded by IBM, Microsoft or Sun, who devotes too much energy trying to solve problems with poor programming models rather than trying to think of good programming models and problems that can make good even better.

    Text is just a picture. CJ Date and William Kent have been saying this forever.

    Actually, text makes versioning components harder, in spite of having reasonable defaults. Text should only be a decorator. A good example of this is the C# programming language, where it’s parser is too ingrained with the way the programmer is expected to work.

  2. “Well, duh” too. 🙂

    Nobody expects a textual representation of a database. Sure, you can dump it to text, but people don’t do that so that they can work on it.

    So, it’s a mystery to me why, to this day, we still require computer languages to have a defining syntax instead of a syntatical projection.

  3. If there is a two-way transform from the internal form to text, you can call whichever you want the “source”. 🙂

    Having program source in text has other potential benefits. Text can be scanned and manipulated by generic text-oriented tools. Otherwise you up having to implement all the equivalents. Hint: you will not imagine all the possibilities, and will not have time to implement all that you know about. Arguably this is what killed IBM’s VisualAge for Java (an otherwise interesting product – of which at least some notions got recycled in Eclipse.)

    In any case, interested in what you come up with….

  4. Er, Daniel – in fact if you dig around, you will find folk who have indeed dumped databases to text, ran a Perl (usually) script over the text, and re-loaded the database. You can do some transforms on text that would be prohibitive to attempt with database operations.

    This is by no means the usual case, but tremendously useful when needed.

  5. Preston,

    While I see your point, I think “database operations” is a little vague. Where I work, we love SQL CLR integration services. It is quite the opposite from what you describe: we do some transforms on text that would be prohibitive in other languages. The only time we munge data dumps is in ETL procedures. Also, the concurrency abilities of a database are generally faster than writing the code in most languages.

    There is a finer detail here, though. By *dumping* a data structure into plain text, you need some way to then *blit* that text into a new data structure. Jonathan is not proposing a dump at all. Instead, he is talking in terms of an isomprhic projection from one data structure to another, effectively saying they are equivalent. By doing so, the data structures themselves will never get out of sync and that in itself is a huge boost to maintenance programmers.

    Hope that is clear. I do like the fact you push back on what others are saying, as you are noting a common objection I see from practitioners. More than likely, Jonathan will face similar objections.

  6. Hooray! That was my big question about your last paper — will you stick with the Subtext programming model. Two things blew me away in your presentation I saw at OOPSLA. One was a truly live programming environment, even more live than REPLs and image-based programming environments. The other was the copy-paste-and-fiddle-with-it way of programming which is, I think, the way 99.9% people really think about programming and therefore the method that programming environments should support.

  7. @Preston:

    You say “Text can be scanned and manipulated by generic text-oriented tools.”. But to do anything sophisticated with text, you need to parse the text and do semantic analysis. It’s easier to do that on a semantically rich model. For example, programmers can easily implement sophisticated Java transformations and refactorings as IntelliJ plugins, building on IntelliJ’s internal model of the program. There are many languages for which this should be much easier than Java (Haskell, for example) but for which there is hardly any tool support. Programmers are stuck working in terms of text, not in terms of program transformations as Java programmers are now used to.

  8. Nat,

    Is Haskell’s purity the reason why you say it would be easier?

    F#, another ML language family descendent, doesn’t have anywhere near the same tool support as Ocaml. camlp4 or camlp5 simply aren’t supported by Microsoft, and my thought here was that F# allows side-effects, which complicates the sort of transformations one might do when building DSLs. However, I don’t really know the answer!

  9. This sounds like a great plan. I documented a similar design of having text based IDE representation over an internal representation in my Evolvable languages paper. If you’re interested it’s here …

    In terms of the persistant data model. If you happen to develop your interpreter in Java, then please consider Argot 1.3 for the persistance layer. Infact I’d be happy to help write the persistance layer code for you. As I’ve just implemented meta data versioning I’m keen to apply it to a real world problems.

  10. Are you planning to make the syntax customizable as well? Will your core just be a few simple constructs a la Lisp, with transformations built on top? I’m still trying to figure out how all this differs from what Charles Simonyi’s been working on for the past 15 odd years (and amassing piles of patents in the process).

    [I plan to post a review of Intentional Software soon – Jonathan]

  11. Hi Jonathan,

    Have you thought about using the Oslo framework? From reading your articles I have a sense that there is a lot of correlation between your idea and the conceptual thinking of the Oslo framework. However, I could be wrong and miss-understood what you communicating here.

  12. hey, gosh, maybe you’d become famous for having solved the 2 top issues in CS /ever/: (1) spaces vs. tabs, (2) which lines the curly braces go on. seriously, if the ide could just always show me my style, and always show other people their own style, and then store it all as a core structure in SVN (or whatever) so that the history diffs work well, then that might be nice news to sell it all 😉

Comments are closed.