At this juncture …

I have spoken before of the need to rebrand Subtext. It is stereotyped as a Visual Programming Language, and as such will never command respect. Using a non-textual code rep violates everyone’s expectations for how programming is done, and even how we write about it. The really fundamental problem is that I have been trying to solve problems that people don’t know they have, or won’t admit they have. No one is willing to admit they aren’t smart enough to program with current languages.

So I am going to focus on a problem people know they have: building interactive systems. Like web apps and GUI apps. That problem is MVC, which I refer to as Callback Hell. Many people have tried to solve this problem using declarative languages. I think I have a new take.

I propose to call my new language Juncture. I even have a logo. And I am working on my powerpoint slides. 🙂 Juncture will have a textual rep, so I can explain it without freaking people out. But not textual source – you can’t use Emacs. Subtext is still there, inside the Trojan Horse.

Sorry I don’t have more details to share yet, but I wanted to be open about what I’m thinking, and invite discussion.

40 Replies to “At this juncture …”

  1. I like the name and the logo and the new problem domain. It sounds like Juncture will be very practical, and relevant. MVC has been somewhat abused of late, especially in the area of web-based applications. Here’s one man’s opinion (from Slashdot): http://www.pointy-stick.com/blog/2008/11/30/removing-model-view-controller-straitjacket/

    An observation: Last evening I was reading Dijkstra’s ‘On the Cruelty of Really Teaching Computer Science’, again because of a Slashdot article: http://news.slashdot.org/article.pl?sid=08/12/02/1410254. It occurred to me that he might have been interested in your project because one is able to program at the semantic level, thereby opening the way to easier proofs of correctness.

    Can’t wait for the slides.

    Cheers,

    Peter

  2. It is really a shame that stubbornness has kept Subtext from gaining traction. I hope the new venture goes well and I’ll be keeping my eye on it and waiting for a chance to contribute (hint hint, open things up early this time!)

  3. Hi!
    I recently found out about Subtext and it resonates with some thoughts I had, by providing a concrete way to approach these issues.

    My domain to use such a tool would be in designing networks of communicating asynchronous entities (processes) seen as Finite State Machines. My hope is that it will be easier to design and implement communication protocols in this environment.

    MVC is a better domain if the goal is to attract community, so please don’t reconside for my sake 🙂 What I wanted to point out is that there are other interesting areas and that I hope you will keep the generic parts, well, generic enough 😉

    best regards,
    Vlad

  4. Hi Jonathan,
    two questions:
    1) Is junction a new name for what we called Subtext2 until now? I am bit confused. What you released last time (schematic tables) was not Subtext2, or?
    2) What do you mean by Callback Hell? Googling it gave me no clear results.
    Thanks,
    Peter

  5. Anyway, this Junction-thing is like when parents wrap medicines into chocolate or candies to make their kid to eat it. Yout treat programmers as kids, that need medicines but don’t want to swallow it. Is this really necessary?
    If they don’t admit having conceptual problems with their tools, then why would they use such a “strange” tool?
    Only time will tell. But this “wrapping” will add unnecessary complexity and will blur — or even hide — the real story.

  6. I find it pretty amazing how many developers today think a Controller object is a good idea. One of the most widely read programming books, The Pragmatic Programmer, even suggests MVC is a good way to partition control in an application. Yet, there are so many things wrong with MVC.

    The most obvious problem is people are gullible and believe marketing material. Call a framework like Struts “object-oriented”, spread the word wide enough and people will believe it, and one person tells two people, even though it violates several OO design tenets. The fundamental problem is people associate meaning to words. They hear “object-oriented”, and naturally assume consistency with an ideally “object-oriented” program, even if they have no idea what “object-oriented” is. People reason by analogy, often superficially, and naturally assume consistency. Most of us are South Sea islanders building cargo cult airports.

    I constantly refer to myself as a bad programmer, because I’m relentlessly self-critical. However, people only hear me refer to myself as a “bad programmer”, and immediately dismiss me as unworthy of conversation. They can’t see past superficial descriptions. Then again, neither could the Trojans see past the Achaeans elaborate ruse.

    @Peter C Marks
    I dislike articles like the one you pointed to above (“Removing the MVC Straightjacket”). It is poorly written, and falls back on giving no real insight at all: It tells me how Django works, criticizes Django, but doesn’t explain why Django’s framework has a “guide wire ” of ill-defined architectural constraints. The suggestion that “Ajax” roughs things up for MVC frameworks is also ill-defined scientifically as to why. Really, all the author is trying to state can be re-stated succinctly: “there is a myth in our profession that programmers share the same psychology, yet here is a counterexample, known as MVC, de-bunking the myth.”

    My architectural constraints explicitly state “No Controller Objects”. This rule is Marc Balcer and Steve Mellor’s “second rule of control partitioning” (Pg. 236, Executable UML). Controller objects hardwire sequences of collaborations into the top-level facade of a program. Furthermore, Models should be able to render their own View. Although this sounds awkward to many programmers, the best-selling software program in the world follows this architectural constraint fairly well. In the old days of COM, it was called the “In-Place-Activation” feature of Object-Linking-annd-Embedding (OLE).

    Recently, on USENET’s comp.object lisp, Nick Keighley asked me if this was a well-known design principle, and I said flatly it wasn’t well-known. It’s a shame, because understanding this principle leads to better designs. Yet, this poor knowledge of design is fundamentally linked with how we teach programming in our educational institutions. Rigorous models of program execution are considered “advanced topics”, and most programmers don’t even use Petri Nets (of any kind [Finite State Machines, Statecharts, etc.]) to minimize system-level defects, and even fewer know what “Run-To-Completion semantics” means and even fewer are familiar with how incorrect messaging models cannot only increase maintenance costs but also corrupt object state by dispatching on a state machine and leaving the program’s execution semantics open to interleaving state race conditions.

  7. @Dercsar Peter

    “Callback hell” is programmer slang occasionally used on USENET to describe poorly partitioned control among subsystems.

  8. I found your site a while back when looking for information on alternative programming techniques. It was so surprising to find that someone had ideas so similar to mine and that had already done a lot of the leg/thought work for me 🙂 I really liked the decoupling of logic and execution flow of subtext, and while I think the interface wasn’t as smooth as it could be, it really seems like it would be a big leap for programming as a whole.

    On the topic of textual input, I wonder again if we sharing the same thought. I don’t think people are tied to text interfaces so much. It’s the keyboard they prefer, and it has been historically tied to console and simple text input, but I don’t think that has to be the case. As an example that’s not too much of a stretch, imagine a traditional unix shell with a graphical front end. You type ‘ls’ space and it immediately shows a window that will interpret the following text input on the fly, building a graphical table that represents the command flags you are passing to ls, similar to filling in a dynamically generated form depending on the options you want to send to the command. Not sure that makes sense, but it looks great in my minds eye. And that’s just the _transition_ period! Once it becomes the standard we can start to shed the legacy commands and really begin to flesh out the new interface. Perhaps even a new keyboard would be in order that isn’t bogged down with legacy keys. I mean really, when was the last time you used the pause/break key in something other than a legacy unix app, or similar?

    Well, it sounds exciting to me anyway 🙂

  9. @Dercsar Peter

    Also, probably the most mainstream example of callback hell is ASP.NET Classic.

    However, there are a lot of non-scientific reasons as to why the ASP.NET “Postback Pattern” was a bad idea. For instance, search engine optimization is hindered by the fact that all links are converted into POST requests.

    On the scientific side, you could say it violates Command/Query Separation (EiaC – everything is a command, there is no co-algebraic reasoning) and exhibits poor control partitioning for this very reason. Some developers criticize ASP.NET Classic because it uses a “batch processing model”, too, which is essentially a way to say that hypermedia systems should have “alternate hard and soft layers” – not just bifurcating state persistence levels into “bookmarks” (hard state, canonically represented by URI) and “navigation history” (soft state, canonically represented by back/forward buttons and POST fences as boundaries).

    Some authors, like the one Peter C Marks linked to above (”Removing the MVC Straightjacket”) mistakenly believe “Ajax” automatically lifts the EiaC architectural constraint imposed by ASP.NET. Yet, if you look at Atlas, Microsoft’s solution for a suite of “Ajax controls”, the underlying execution model is still a Postback Pattern and therefore everything is still a command and there is no co-algebraic reasoning. Just because you change the scope of batch execution and divide it into subrouties (Ajax calls) does not change the fact it is still a batch execution model. This makes programmatically reasoning about large-scale data movement operations difficult, too, because you have no way to coalesce or quench messages in event sources/sinks. As a result, it isn’t clear to the program what bandwidth saving measures he can make – too much data is sent back-and-forth across the wire. The entire hypermedia document is bound to a World State.

    [Interesting comment. I may regret saying this 🙂 , but can you elaborate? I hadn’t heard of command/query separation before, but it does apply very much to Juncture. What does coalgebraic reasoning have to do with “coalesce or quench messages in event sources/sinks”? AFAIK, colagebraic reasoning is a fancy term for talking about infinite traces through a state space. – Jonathan]

  10. only to say Thanks for fighting the good fight, and continuing giving folks like us some food for thought.

  11. “The really fundamental problem is that I have been trying to solve problems that people don’t know they have, or won’t admit they have”

    Excellent!

    I’ve been thinking about Subtext recently and had a complete change of opinion. I think there’s no point in trying to convince existing programmers. They’ve spent years being training their minds to think about the problem of programming in a way that is fundamentally different from Subtext. Instead, I think you should address the needs of people who are not programmers. What I saw of Subtext at OOPSLA seemed a perfect fit to the way that “naive” programmers work. That is, it fits the way that people naturally think about problem solving with computers before their minds are warped by learning to program!

    I look forward to learning more about Juncture.

    But I think you should not limit the system to a single way of representing programs. If there is a deeper underlying representation, then can it not be portrayed and manipulated in different forms?

  12. “Controller objects hardwire sequences of collaborations into the top-level facade of a program.”

    This is a misunderstanding of MVC. In MVC, a controller implements a direct translation of a sequence of UI events (often a single event) into a method call on a Model object. If it implements hardwired collaborations between model objects, it’s not a “Controller” in the MVC sense, and those collaborations should be moved into the model classes.

  13. It’s sad to see that subtext is getting trojanized into another tool — I can’t imagine it working. emacs has lisp buried not-so-deeply and it didn’t spawn a lisp revolution.

    But, over the past two years I told so many people about subtext and couldn’t get traction either, so maybe its time to try something new. So much so good falls by the wayside, and we end up with nice solutions built on flimsy foundations. Eclipse editing Java, one curly-brace at a time.

    Hope the web toolkits do well. If it works as intended, you could boostrap it — publish Juncture using Juncture online so people didn’t have to download and install it and click and tweak.

  14. “Branding” isn’t the problem. Functionality is the problem. People will respect Subtext when you can use it to create modern software. Can I use Subtext to write an email client? A web browser? A web server? Anything non-trivial? A new logo isn’t going to matter if it still lacks the basic ability to create useful software.

  15. Nat,

    This is a topic that literally kills programmer brain cells. I’m going to do my best to be fair and not kill brain cells, and avoid getting called a loser by the peanut gallery for disagreeing with you. (I respect your work on Babble, JMocks, and Hamcrest a lot). An important thing to realize is that very few real world systems fully satisfy OO – most of them are probably real-time embedded control systems.

    My point, elaborated in fuller detail on USENET, would be that just about every programmer I come across modifies the architectural constraints to their suiting. In fact, The Pragmatic Programmer incorrectly describes MVC, according to your definition (the Smalltalk-80 definition)! Note that Thomas and Hunt advocate deriving Views from Views, and Controllers from Controllers. While recursive design is often the hallmark of a well-designed system, the utility of derivation here is questionable. In fact, Hunt and Thomas describe an anemic (domain) model: Knowledge about the domain is placed into the View. For example, their reference to keeping a running total of the score.

    Let’s place more emphasis on WHY programmers are lured into using Views on top of Views. Deriving a Controller from a Controller isn’t that bad (although the derivation should use “specification inheritance” instead of “implementation inheritance”). Analyzing the WHY in how humans behave is way more practical than asserting that “this is a misunderstanding of MVC, [the Smalltalk-80 model is this]”. For what it’s worth, I consider Smalltalk-80 MVC to also be a “busted” architectural pattern. However, my conclusions are qualified verbiage. Quoting Steven Burbeck’s paper, “Constraints on the type of objects allowed to function as models would limit the useful range of applications possible within the MVC paradigm. Necessarily, any object can be a model.” This is plain wrong. This lack of architectural constraint makes it permissible to use Views on top of Views. However, WHY would you want to? This sort of flexibility is only necessary if your toolkit violates OO design tenets. In particular, the Open-Closed Principle. A more interesting question might be, “If the View only varies the representation of the Model, then what the heck am I doing creating a contract with the View when I could be creating a contract with the Model?” Furthermore, mixing a View in as a Model makes refactoring possibilities harder to see. Worse, when the CANONICAL model changes, some of the Models are invalidated. This is a strict violation of the Open-Closed Principle, no questions. Furthermore, using the View as a Model also violates the Single Responsibility Principle. Responsibilities formulate WHY for the programmer. When a class changes, there should be a single reason why — a SINGLE policy accord was broken and the policy associated with a responsibility is therefore changed.

    This sort of loosey-goosey system is great for marketing purposes and decreases vendor support costs. Everything can be done – through a hack! What sort of hacks do these systems exhibit, other than Views on top of Views? Typically, all Models expose Properties, Methods and Events. What happens if you have a currentlySelected property in one child View but not the parent View, but then later on the requirments change and you need one in the Parent view? User interfaces, especially in B2B web applications, are way too volatile for introducing the possibility for dealing with these kinds of naming collisions.

    Burbeck, in a confusing way, does sort of hint that this is all a bad idea: “Because the input and output behavior of most applications is stylized, much of it is inherited from the generic classes — View and Controller. These two classes, together with their subclasses, provide such a rich variety of behavior that your applications will usually require little added protocol to accomplish their command input and interactive output behavior. The model cannot be stylized.” Burbeck basically gives a conflicting definition. Any object can be a Model, therefore a View can be a Model. Yet, a View can be stylized but a Model cannot be stylized. This isn’t mathematical! Where’s my law of excluded middle? YOU NEED AN ARCHITECTURAL CONSTRAINT! Recursive design only works if the corresponding hierarchy of description logics make provable sense (no lexical ambiguities). Structurally, it’d be nice to at least not violate the Dependency Inversion Principle, wouldn’t it?

    People wonder why there is so much ambiguity over the term MVC. It’s because one of the most widely distributed tutotials is inconclusive as to WHY it is designed this way.

    You should not even be asking, “Does it make sense to derive a View from a View?” More practically, if the View is a user interface, the people designing the View should NOT be programmers. They should be visual designers and interaction designers working with visual design tools, not cutting code. The programming role is one-half of the developer-designer facade proffered by the general notion of model-view separation. This facade is a great idea, and worth implementing correctly. I highly recommend others do so. The benefit is parallel execution of project tasks.

    Furthermore, on the Web, use of MVC is unjustified for other reasons I haven’t listed above. In particular, you are increasing accidental complexity if you use MVC as a universal way to coordinate n-tier applications. Primarily, it’s the following recommendation by Burbeck that makes MVC unsuitable to n-tier architecture: “Controllers must cooperate to ensure that the proper controller is interpreting keyboard and mouse input (usually according to which view contains the cursor).” This recommendation has the built-in assumption that it is the client programmers responsibility to re-jigger the Controllers every time input/output mapping maintenance occurs. No thanks. I’d rather have an object-oriented design, where I make a change in ONE PLACE. This means synthetically creating a distributed system, even if the system is on the same machine. Alan Kay actually understood this really well. He wanted the Document Object Model to have every element in it be addressable in the URI. http://video.google.com/videoplay?docid=-2950949730059754521 This is similar to how some Google applications are architected (Gmail, Google Maps), although they create a “peer model” of the DOM in a hidden iframe to manage the hard-and-soft application state. Note that really, if you are trying to solve problems that require components to be context-aware, then a distributed event bus is extraordinarily helpful. That’s also the essence of Burbeck’s use case for Controllers: interpreting input according to focus.

    The hallmark of a good system is ONE FACT, ONE PLACE, ONE TIME.

    If you are making a change in more than one place, you have semantic ambiguity. Ted Codd, William Kent, C.J. Date, Count Korzybski, and Alfred North Whitehead were all really smart people who understood this particularly well. Establish semantic invariants.

    Attempts to establish semantic invariants, such as Hierarchical MVC, “Solve the Wrong Problem”. The HMVC prposal is “not even wrong”, because it’s not searching for the right problem. Find the Right Problem. Identify WHY things occur. Apply basic OO principles like Command/Query Separation, Single Responsibility Principle, Open/Closed Principle, Dependency Inversion Principle, etc.

    I’d also note that some MVC frameworks are better than others, like QT4 is vastly superior to most but still requires implementing low-level logic directly for creating widgets from scratch, which is a non-starter for having interaction designers quickly prototype new UI concepts.

    I’d also note that word “Controller” is just a word, and nothing more. For example, there is no connection between how software engineers typically throw around the term and how the term has a very concrete meaning in the fields of Fuzzy Systems Engineering.

  16. Hi Jonathon,

    I’m *definitely* looking forward to seeing Juncture. As for subtext, what I’ve seen of the implementation -which is admittedly not very much- is not so great, but the idea is in my opinion one of the most important changes we need to move to if the state of our art is to progress beyond re-inventing LISP over and over. I hope you’re not going to leave it behind.

    Keep up the good work!

  17. Thanks for all the comments. Let me clarify a few things. When I talk about Subtext being hidden inside Juncture, I mean the essence of Subtext: the underlying semantic model, in which code is a living reactive cross-linked tree. Also the principle that all linkages are based on tree paths – URI’s if you will – rather than lexically bound names.

    Juncture generalizes the Subtext semantic model in a key dimension: a new way to handle mutation of state. I intend to start off with a seemingly normal textual rep of the code model as a gentle way to introduce the concepts. But it is really still a Subtext tree underneath. I can delay revealing the power that comes from treating code as live data with a fluid rep until people are ready for it. Schematic Tables may not make it into the first prototype.

    As to MVC, I think our discussion here does tend to make the point that there is no consensus on what it means exactly. A slogan I am using is “MVC without the C”.

    You can visit some of the inner circles of Callback Hell by looking at WPF Event handling, or the JSF Lifecycle. Without having actually looked, I would bet that the internals of those big Javascript Ajax libs are just a mass of writhing callbacks.

    It’s been real guys, but I am taking off for a week now, and I am going to try to not do any work. Later.

  18. Could you please make the visual editing component using FLEX?
    I really want to use it inside of my program in FLEX.

  19. “It is stereotyped as a Visual Programming Language, and as such will never command respect.”
    Maybe if you made it LOOK more UML like, it would command the respect of those who like UML. Fit it in as an extension of UML editing.

    People can’t handle new things – in the business world you make software for $, new things like this are useless, in the academic world you need to build on past papers, peoples past ideas, with lots of references to be accepted.

    Both these motivations, money or prestige are not going to support your work – as your work is a creative re-think on programming and no-one wants that, expect for us programmers who actually think we are Gods and can do what we like.
    Grass roots support will be the only support for this, the common will of people.

  20. Functional Reactive Programming is a relatively new (not yet mature, but quickly maturing) declarative way to define interactive systems.

    Its worth checking out, though you might need to decipher some Haskell syntax on the way.

    Here’s a GUI toolkit based on FRP:

    http://www.haskell.org/haskellwiki/Phooey

    Then there’s also an OpenGL/GLUT interactive FRP platform:

    http://www.haskell.org/haskellwiki/FieldTrip

    [Thanks for the links Eyal. I think of FRP as Higher Order Callback Hell. Though to tell the truth I really don’t fully understand all the stuff they do – I know that something that abstract can’t possibly be useful, so I lose interest rapidly. I hope Juncture will be “FRP for the rest of us”. – Jonathan]

  21. Hey Jonathan.

    FRP is about elimination of callbacks – not making them higher-order.

    In FRP, instead of specifying what steps to execute when something happens (callback), you specify what something is, in terms of other things. Those other things are allowed to be event sources.

    So lets say that we want to represent a little UI with an “increase”, and “decrease” buttons, and a number-label that is controlled by those buttons.

    We could do this the imperative/callback way:
    A. Set up the layout, buttons, and label. Attach callbacks on the buttons that write to the label.
    B. Set up the layout, buttons. Create a label that is an expression of the buttons’ events.

    B is actually expressed via something like (much less verbosely, of course):

    buttonInc = CreateButton(“Increase”, event_value=func_that_increases_by_1)
    buttonDec = CreateButton(“Decrease”, event_value=func_that_decreases_by_1)
    buttons = combine_layouts_and_events(buttonInc, buttonDec)
    label = Label(value = RunFunctionsFromEventSource(initialValue=0, buttons))
    return BuildUIFrom([buttons, label])

    All these ugly names have much nicer short names/operators in practice.

    Its not that abstract, but there are some generalizations — which I believe are inherently abstract.

    [Eyal, Thanks for the mini-tutorial. I need to study this latest Applicative Functor incarnation of FRP more deeply. There is something about FRP that bothers me, but rather than spout off cute slogans I should take the time to more fully understand it. – Jonathan]

  22. “My domain to use such a tool would be in designing networks of communicating asynchronous entities (processes) seen as Finite State Machines. My hope is that it will be easier to design and implement communication protocols in this environment.”

    Vlad, and other people interested in reactive programming, you might be interested in my thoughts and Visual Language for hierarchical communicating state machines and event-based programming:

    http://www.visualframeworksinc.com

    [Paul – how does your language compare to the synchronous reactive languages, like Esterel, Lustre, and SyncCharts? Their synchronous semantics provides similar causality guarantees to VF. – Jonathan]

  23. Personally, I have encountered plenty of situations where Subtext would have been helpful. Part of the reason I respect Subtext is probably my strong background in formal logic. I found the idea of using Subtext to handle my program logic to be a good one. However, Subtext feels too much like an academic language. How do I use Subtext to access the Windows API? And will it be as practical to do so using Subtext as it is using C#, which makes it absurdly easy? What about adding a GUI to my programs?

    Until such questions are answered, I won’t find Subtext that exciting, unless it’s merely an interesting add-on to an existing language, in which case I’ll look at it briefly and then run away, since using two languages at once means it’s tougher to find other people to work on my program. Subtext solves a very important problem, but there are also a ton of practical problems, which you did not mention.

    Of course, I’m coming at this from a business perspective. Back when I was a graduate student, I would’ve been eating this stuff up.

    I bet Subtext would be *perfect* for writing a program to work with logic. Great for writing Fitch clones 🙂

  24. Jonathan

    Thank you for your questions.

    A preamble, before answering more specifically:

    As you must know, most people seem to believe that textual languages already express every conceivable programming concept, hence, they believe that Visual Languages are “useless” and “cannot” express anything “new”.

    This myopic view comes from the inability to step back and ask oneself the questions: what repetitive design operations are not expressible in our current textual languages, and, what ideas from other engineering domains can be stolen (aka reused) to augment the software development process?

    As my colleagues and I, over the past couple of decades, hand-rolled multi-tasking kernel after multi-tasking kernel, we grew progressively dissatisfied with our “success rate”. We grew up designing hardware, then moved to software, as that became more lucrative. When we designed hardware using “state of the art” techniques, the result was something that worked. When we designed software, even well-understood software like compilers, the results were always late and buggy, and, we weren’t the only ones with this problem :-).

    At one point (see the papers), we were forced to solve a certain set of problems which forced us to consider using Harel statecharts. The experience forced us to redesign statecharts to get rid of some of the inherent problems. This grew into what I call VF today.

    VF is a Visual Language which provides programming features that are not provided by any textual language familiar to me:

    – It provides constructs for rigorously expressing “architecture”.

    – It provides features for expressing composition of systems out of small units.

    – It provides features for structuring such composition.

    – It treats every architectural unit as an asynchronous block (idea stolen from the digital hardware world).

    – It treats all communication between architectural units as one-way “events”. (This allows composability and visualization).

    – It deals with reactive, goto-full programming (hierarchical state machines).

    – The semantics are chosen to preserve the principle of “what you see is what you get” (i.e. no implicit connections such as with Harel’s parallel machines, children cannot override transitions of parent, etc.).

    – It does not try to solve problems that are already solved in textual languages. VF is a hybrid – it includes textual code snippets at its leaf levels.

    – VF breaks away from the pervasive call-return mind-rot which prevents the software world from solving concurrent problems in a reasonable manner.

    – All elements of VF (other than comments) are compilable.

    Comparisons:

    – Simulink

    Simulink is a gui for constructing mathematical simulations of continuous systems (e.g. analogue electronics). It allows one to use canned “blocks” of functionality, but, before simulating, the tool flattens all blocks and compiles the system into a wadge of mathematical code that juggles all of the signals in the system. Clearly, it must “transform” the mathematics from the analogue domain to the digital domain. I suspect that this requires chopping the concept of “time” into tiny discrete steps, then iterating the whole system once for every time step. Any paradigm that requires a system-wide clock is toast when it comes to full parallel computing.

    Simulink is not a programming language aimed at solving the Software Engineering puzzle.

    VF preserves the design hierarchy at run time. VF only uses discrete events (an event is just a unit of data with source and destination tags – much like a tiny data packet in a network). What you see on the diagrams is what is placed into memory and executed. This also means that a VF system can be changed by snapping a part out and replacing it with another (pin-compatible) part. This can be done without recompiling the whole system.

    In Simulink, many inputs can change “at the same time” (i.e. during the same time step). In VF, events define time. Two events cannot arrive at the same time at the same component.

    Labview

    Another simulator.

    Wikipedia claims that Labview uses a dataflow programming paradigm.

    VF events are at a “lower” level than dataflow. A VF component can receive exactly one event at a time, regardless of how many input pins it actually has and how many input pins are fired “close together” in time. If a designer wants a part to wait until there is data on every pin, the designer must program that behviour in.

    In my opinion, VF events are the “atoms” of concurrency.

    Esterel / Scade / Lustre

    This appears to be more, but not entirely, similar to VF.

    From Wikipedia: “Signals are the only means of communication.” This is similar to the VF concept of simplifying/culling to achieve standardization and modularity.

    From Wikipedia: “Signals are broadcast across the program, and that means any process can read or write a signal.” I take this to mean that Esterel systems are not very well “structured”. That limits scalability – at some point the system becomes unmanageable – the old “spaghetti code” problem of the pre-structured-programming days. I was at Mitel Corp. in the early ’80’s when they discovered that a system (SX2000) composed of zillions of processes without rigid “principles of structuring” was a problem to develop and maintain.

    VF allows one to construct “parts” using other parts, hierarchically. In a well-drawn VF system, every layer in the hierarchy is easy to understand (even management understands VF programs, we have discovered (some programmers consider that to be a threat) :-).

    Esterel also deals with time as discrete slices, where more than one input can fire at the same time. Wikipedia notes that “Avoiding causality violations is often difficult”, which is probably a direct result of this notion of time.

    The Lustre tutorial indicates that it is a language for simulating the mathematical state of a system.

    VF doesn’t simulate, it “just does it”. VF is a visual programming language that supports cooperative multitasking at its very core.

    UML

    VF is often conflated with UML, although they are very different animals.

    UML tries to “model” every known programming paradigm. VF culls the herd and constricts the description of architecture to using only parts, schematics, wires and events. (Other “kinds” of programming are allowed inside “leaf” parts).

    UML tries to capture requirements. VF doesn’t.

    UML has Harel StateCharts bolted on the side. VF stripped and discarded the bad parts of Harel’s notation. VF replaces the notion of parallel state machines with schematics composed of asynchronous components with explicit visualization of the event flow.

    Not every operator (e.g. “stick people”) in UML can be sensibly compiled to code. The design of VF began with the principle that everything must be compilable.

    UML, being fundamentally based on data-oriented modeling tools emits (often empty) classes. Control-flow is an after-thought in UML. Control-flow is central to VF (and data design is relegated to the textual code snippet sub-language).

    UML promotes generalization.

    VF promotes the opposite – specialization. The specific solution of a specific problem. One problem at a time. If you observe Engineers designing a bridge, you will note that they do not design a “class” of solutions, they design a very specific solution tuned to that very specific place. In my opinion, software engineering will not become Software Engineering until that kind of mindset is employed and supported by the tools.

    Concurrency

    I like to think that VF does for concurrency what “stacks” and “parameters” did for sequential programming.

    A very simple principle is applied uniformly throughout the system. Every “part” has one input queue and one output queue and a busy flag. The semantics guarantee that events are delivered atomically (regardless of to how many parts/pins the event is delivered), i.e. event delivery is a critical section of code that is automatically handled by the system (inside the tiny kernel).

    This very simple standardization may appear wasteful, at first, much as using the stack to pass parameters appeared wasteful to assembly programmers who knew that “registers” were much more efficient.

    It does, I contend, lead to a very simple-to-use concurrent language and design paradigm. These semantics lead to causality which makes it easier to decompose problems and easier to design / implement / integrate them in parallel (aka the Mongolian hordes approach).

    It is powerful enough to solve complicated embedded and non-embedded problems without encountering the usual array of concurrency problems, semaphores, priorities, preemption, et al (we’ve shipped real products with this).

    pt

  25. SyncCharts are based on Esterel and inspired by Harel (obviously, Harel’s work inspired a number of interesting offshoots!).

    This SyncChart paper describes the causality “problems” that Esterel encounters:

    http://www.i3s.unice.fr/~map/WEBSPORTS/SyncCharts/TR96-28.pdf

    Basically, it is entirely reasonable to draw a diagram in which an output pin of a part is connected back to an input pin of the same part (directly or indirectly). This is called a feedback loop.

    In Esterel and its descendants, feedback causes a problem.

    The problem occurs because of Esterel’s notion of time. Esterel’s time is a sequence of snapshots. All activated inputs and outputs are scanned in the same time slice. Feedback causes an ambiguity – how can an input cause an output that affects the very same input at the very same time?

    (This scan-all-the-inputs-at-once technique was used in industrial controllers based on ladder logic – PLC’s (programmable logic controllers). When you have lots of inputs (sensors) and lots of outputs, this technique is SLOW – milliseconds per scan, back around 1990).

    Thankfully, we were commissioned to expressly beat this slowness. The controller had to react to a physical event in less than 25usec. That’s 0.025 of the scan time of a then-current PLC – i.e. PLC’s could not react in time. It’s also 0.3 of the context-switch time of a 68000 running OS/9 – i.e. multi-tasking kernel was too slow to react to this event.)

    VF allows feedback, because it’s useful and because VF disallows recursion at the architectural level (yes, my favourite textual language is Common Lisp).

    VF semantics were carefully crafted to bludgeon the feedback problem.

    Here’s how: each part has an input queue and a busy flag. When an output event is sent, it is placed on the input queue of the receiving part(s) and the busy flag of each of the receivers is checked. If a part is marked “busy = true”, nothing else is done. If the part is not busy, “busy = false”, it is awoken and its busy flag flag is set. The part then processes its input queue, one event at a time, until the queue is empty, then the part goes idle again (busy flag reset).

    In the case of feedback, the part produces an output event, drops the event on its own input queue, checks its own busy flag, finds that it’s busy and does nothing more, happily returning to process events from its input queue. Eventually (or immediately if the queue was empty) it will find the feedback event and process it.

    In effect, input queues + busy flags cause a delay in the feedback loop, thus, feedback is not problem, and becomes a useful feature.

    I will try to explain why this design decision was made:

    Reactive systems sit idle much of the time. When a physical event occurs, like the 25 usec event above, one would like to dedicate 100% of the CPU to dealing with the physical event, instead of wasting CPU cycles scanning inputs or performing context switches (which are expensive because you must pour registers onto the stack and that takes valuable time).

    This design – a chain of parts with input queues – allows the device driver to create a flurry of activity dedicated to reacting only to the given physical event. Events are fairly cheap. Queues are fairly cheap – especially in the reactive world, because the queues are almost always empty. Pushing an event from the device driver through a chain of software modules interconnected by queues turns out to be much cheaper than doing it via context switches or input-scanning. And, it allows almost 100% of the CPU to be devoted to the task at hand.

    Practically, the only other way to solve this problem would be to do it “underneath” the O/S – i.e. as a device driver. Device drivers are tricky to write. They are wildly tricky to manage when events happen faster than the context switch time – because you cannot punt the work to a cleanly-written process and you have to use ad-hoc device driver code (i.e. raw C or assembler with no assistance from the O/S).

    In fact, VF works COMPLETELY at the device driver level (if you let it), but, it provides a way to organize and modularize your “device driver” code.

    After the smoke cleared, we stepped back and discovered that VF was useful for a whole range of problems – all GUI’s are reactive, PDA’s, cell phones, networks, servers, i.e. just about every problem that exists today.

    VF does not use “processes” or “threads”. It simply feeds a chain of control down the line. On bare hardware, the chain begins with an interrupt. When VF is forced to live inside an O/S, you simply use it as any other programming language and the chain of control is derived from the “main windows loop” or some other invocation (a call).

    The combination of VF semantics gives you the same benefits that you receive by using a multi-tasking operating system without actually needing a multi-tasking kernel.

    Output queues were added for another reason. They further isolate components in the dimension of time. To allow a part to process an event to completion in a “fixed” amount of time, you must not spend time delivering and processing output events. So, a part can generate as many output events as it desires, but they are saved on an output queue. The VF “kernel” releases and distributes the output events only when the part has finished processing the input event. (The issue of whether the kernel releases events after one input event has been processed vs. waiting until all input events have been processed has been consciously left undefined – it is a scheduling policy which changes depending on the requirements of the system being designed. In fact, I believe that it matters only on bare hardware applications – it doesn’t matter what you do about this when running inside an O/S). Output queues allow device drivers to return to idle very rapidly. Also, without output queues, it is possible to design a chain of events that results in something akin to priority inversion (ask the Mars Rover people how much fun that can be :-).

    The output queue idea also manifests itself as “deferred conditions” in the Turing+ language (and I claim that it was my comments, about our solution to the problems of standard Hoare monitors vs. device drivers, over lunch with Ric Holt that caused this feature to be added to Turing+).

    The issue of multiple inputs dissolves with VF semantics. Multiple inputs are certainly encouraged.

    However, it is possible to design a system where the order of two input events is important, but in which they arrive “so closely together” in time that their order is ambiguous. This problem is left to the designer. VF semantics only guarantee that the events will arrive in the *same* order at all receivers and places no constraint on what that order actually is. (Practically, that’s all that you can wish for, living in our relativistic universe).

    Solutions to this problem are well understood and documented in the digital hardware world. The problem is called a “race condition” and is solved by things like Gray Code solutions (a design technique that ensures that only one input toggles at a time) or, simply by using state machines to track the state of the inputs (e.g. “wait for a or b”, if a arrives goto waiting-for-b elsif b arrive goto waiting-for-a).

    Causality means that you can determine that some event a caused an event b.

    VF semantics preserve causality.

    Causality is why it is so “refreshing” to be allowed to program with concurrent languages and threads.

    Causality IS encapsulation / modularity in the time domain.

    The only thing wrong with writing all programs as processes and threads is that these things are too expensive to use on a wholesale basis. For example the IDE we built with VF, running on Windows 3.11, had some 20,000 VF “processes” (i.e. reactive “parts” with VF semantics), and the embedded systems on 8-bit 8051’s had around 300 such “processes”.

    Call-return semantics interfere with causality. I *think* that a whole lot of bugs I solved over the years were caused[sic] by lack of causality, where I didn’t realize the effect of calling some routine and had to spend debug time inadvertently doing an impact analysis.

    Despite claims of uber-encapsulation, most OOP that I’m aware of still lacks causality.

    It looks like Esterel, and its variants, provides causality, but forbids it in certain cases (like feedback), although maybe that has been dealt with in more recent versions.

  26. @ Paul:
    You mentioned “When we designed hardware using “state of the art” techniques, the result was something that worked”. Curious to know at what level of granularity you worked: did you guys wind your own coils or bake your own semis (:/)? Because that is what we are doing in software — starting from ground up every single time.

    Lets face it: we are practitioners of an “arts and crafts” discipline subject to industrial consumer demands.

    @ Jonathan: Just followed the trail from the subtext demo video to here. (yep… under a rock) Great stuff! My suggestion for you/subtext (juncture + logo: no go ..) is to consider the textual code as the initial sketch on the napkin and have subtext slurp the code and quickly setup the 2-D model: basically a visual REPL.

    /Regards

  27. Dercsár Péter:
    I have to disagree. It really is all about presentation and/or appearance. For example, In my opinion, Java is a horribly inferior language based on pretty bad design decisions. Yet it is hugely popular. Why? Because of clever marketing! (Not trying to start a flamewar, just my opinion).

    Juncture might just work for that exact reason – its cleverly deceptive, masquerading as something people want. The catch? It contains (or I assume it will) so much more because its based on a (hopefully) superior base (subtext).
    The truth is, most people (programmers or otherwise) don’t know what they want. They think they do and they jump at anything which promises to give them that while ignoring the things which would actually be whatever it is they really need. Assuming that subtext really is the way to go, and I’m pretty positive about it at the moment, Juncture will hopefully be a friendly introduction which entices developers to at least give it a chance.

    Well, we can hope anyway, its about time for a paradigm shift in programming technology.

  28. I’m happy to see anything happen with this project. It’s the future programming(or construct configuration as I put it.) Good job and good luck for the future!

  29. Jonathan, forgive me if you have covered this already, but I see no easy way to find all the documentation/discussion there is about Subtext/Juncture.

    There are a few benefits that plain text has which I would want from a system like Juncture:
    1. Easy change tracking, using some sort of source control system. As far as I can understand, the ability to “diff” is crucial to how the human brain works; if your new language cannot support this _effortlessly_, I think it is doomed to obscurity.
    2. Easy manipulation. I need a Turing-complete way to manipulate the “code”, whatever form it may be. No GUI is going to provide for this — GUIs are inherently declarative* (unless you get funky). Therefore, I need programmatic access to the code. Do you have plans for this? (I envision your solution being some sort of tree manipulation.)

    Have you discussed either of these?

    * I typically don’t think “Turing-complete” when I think “declarative”. Strictly speaking, one could call declarative “without time”, but almost every declarative language/system I have seen has not been Turing-complete.

    [Luke – You bring up crucial issues which I have not addressed yet. All code is a data structure, and thus can be directly generated and modified. Likewise diffing is far more powerful than the text-based forms because I track actual structural changes, not changed lines. But I haven’t gotten to those issues yet. I have been focusing on basic semantic and syntactic issues. You can’t reinvent the entire world all at once. Unfortunately, that is exactly what most people will require before they take this kind of work seriously. Especially academic Computer Science.

    BTW, I think that in most cases “declarative” does mean timeless, and pure functional and logic programming languages are examples of Turing-complete declarative languages.

    — Jonathan]

  30. > I have been trying to solve problems that people don’t know they have, or won’t admit they have.

    Unless we have some people in the crowd interested in sales and marketing, I see Subtext completely as a labor of love. Along with a few others, I believe this is the future. And that is all we need to move forward.

    Have you considered starting an open source project? To be honest, I’ve never contributed to any open source project ever. But I would _love_ to contribute to this. I don’t feel like I have the time right now to carry a whole project, but together…

    The smart people I’ve shown the schematic tables demo to all think it’s great (esp. the ones who get the idea that code is data), but they don’t know what to do with it. As you said, they’re not used to it; their brains have been programmed to think in text. Plus, they can’t play with it.

    If we could make something that we — and other early adopters — could play with, we can unleash the imagination of nature, and Subtext can take on a life of its own.

    I think you are truly a visionary. As in a previous comment, people don’t know what they want. It’s up to the designer to give them what they really need. When you re-imagine things from scratch, throw away the onion, people are bound to not understand. But the great thing about software is that we can build something and show them.

    But if Juncture is your way of doing this… I wish you the best, as I and others like me will probably have to watch from the sidelines.

    [Jonathan –

    Thanks for the encouragement. I like that Shirky clip a lot. My primary goal now is to come up with something that other people can contribute to. I have realized that this is not going to happen inside Computer Science, at least not at first. The idea with Juncture is to build something that might be really useful, and is complete enough to allow other people to extend it.

    Regards, Jonathan]

  31. Jonathan, RE my comments on diffing and non-GUI code manipulation:

    I understand that you have not figured out how to do what I mentioned, but I think the “more interesting” people would have much less to critique if they knew that these features were on the horizon. Hey, you might even find people (like me) who would be interested in developing those ideas further.

    You certainly can’t have the whole world at once, but perhaps you would be willing to give us a small outline of ideas that you are working on and ideas that you aren’t working on, but consider quite important? I suppose this might not be how traditional research is done…

  32. @Joubin “Æ·” Houshyar

    I don’t think you are being fair to Paul, or those of us who think similarly to how Paul thinks.

    My only criticism of Paul’s technology is I can’t quickly “read” his visual language, but this is in part because he doesn’t show small-scale components as examples.

    @Paul

    What do you think of the Causality Domains concept as a way of accommodating a world clock?

  33. “Declarative” and “imperative” are not set limits, but “soft boundaries”, which are stretched at varying cost or consequences. Any compiler writer understands this well.

    Time is something you have to explicitly account for in order for it to gainfully mean something. Commonsense Reasoning is a good example. We can perform commonsense reasoning, but to have a computer do it requires applying architectural constraints so that it can be *automated*, so that it can be performed automatically by a computer. MIT’s Erik Mueller uses an event calculus to enable commonsense reasoning. Nonmonotonic reasoning and circumscription-based event calculi are two ideas.

    My best effort at explaining “declarative”, though, is that it is serializable to a data structure. A means-ends analysis approach to solving a rubik’s cube automatically by decomposing it into subgoals of primitive operators would not be declarative, because it would require some explicit stepping through the problem to reach the solution. In order to solve a rubik’s cube declaratively under my serializable constraint, you’d need to Learn to Solve Problems by Searching for Macro Operators. http://www.amazon.com/Learning-Solve-Problems-Searching-Macro-Operators/dp/B001OYN0N0/

    Still, it is a soft boundary. Similar to how MIT has conferences on “agile techniques”, “little languages”, and “lightweight languages”, these things are best defined by examples rather than theory. And the best thing is to define something in terms of what it’s clearly not: What is declarative? The opposite of imperative!

  34. Hi!

    Too bad you’re giving up on Subtext. I only recently stumbled about it and it would make a wonderful refactoring tool, if it could grok (even a single) C# method like this one:

    public static RelationType GetRelationType(this Relation rel)
    {
    if (rel.LeftPart.IsList == false && rel.RightPart.IsList == true) return RelationType.one_n;
    if (rel.LeftPart.IsList == true && rel.RightPart.IsList == false) return RelationType.one_n;
    if (rel.LeftPart.IsList == true && rel.RightPart.IsList == true) return RelationType.n_m;
    if (rel.LeftPart.IsList == false && rel.RightPart.IsList == false) return RelationType.one_one;

    throw new InvalidOperationException(“Unable to find out RelationType”);
    }

    and help me prove that the throw can be safely deleted.

    Anyways, I’m looking forward to whatever your next project is. It’ll be great for sure, if you can avoid the quagmire of trying to define what MVC is 😉

    Regards, David

  35. I am really confused by this. You say that “It is stereotyped as a Visual Programming Language, and as such will never command respect.” You also said “The really fundamental problem is that I have been trying to solve problems that people don’t know they have, or won’t admit they have.” Well maybe I am just out of the loop but this confuses me. I was under the impression that you claim that you are not finished with Subtext yet and it is not usable to create anything useful yet. Maybe it would “command respect” if it were finished and useful. Exactly in what way were you hoping it would foster more respect and how does this lack of respect impede your progress or goal? Are failing to find sought after help or funding to finish the project? I am not trying to say you are stupid. These are not meant to be rhetorical questions. I think I am probably just missing information. Please help me to understand.

    [Jesse – Fair question. The ultimate problem is that I am not satisfied with Subtext. It demonstrates some interesting ideas about program representation and IDE design. But there are deeper ideas I haven’t worked out yet, about expressing programs with a fundamentally different semantics. Focusing on representation issues has distracted me from working on the semantics. – Jonathan]

Comments are closed.