It’s great to be coding again, after spending so long lost in thought and theory. This has been my longest break from programming since I started at 13. I now have a working virtual machine that supports the essential semantics of the language. The next step is a compiler. The good thing about having a long break from programming is that it made the experience of starting up again quite vivid. What was most vivid was the many ways I make programming needlessly harder for myself.
It was insane to try using Scala instead of Java. A language so complicated that few people can fully understand it, with spotty documentation, and immature buggy IDE support. How could that possibly be a win over a language like Java that I already know cold and which has state of the art tool support? I was bored to death with Java, and I was seduced by Scala’s cleverness; it’s ability to express things very succinctly, and it’s more powerful abstraction capabilities. When you think about programming in the abstract, such issues loom large. But when you actually have work to do, these idealisms pale in comparison to having trusty, well understood, well supported tools. We programmers love abstraction and elegance, but it is often a siren song leading us to disaster.
An even graver mistake I found myself making is compulsive performance optimization. I know well the evil of premature optimization. It is even central to my philosophy of programming language design — I believe that existing languages are far more complicated than they have to be because they are obsessed with letting programmers hand-optimize everything. I am building a prototype VM where performance is largely irrelevant. Yet still I repeatedly found myself optimizing: caching, interning, fast-pathing, etc. I just can’t stop myself. There is something particularly insiduous about performance. I think it is that it is so much easier to think about performance than subjective design issues. Performance is objective — you can actually quantify it, unlike most of the issues that really matter: subjective design issues with no clear cut answer, which can be almost impossible to evaluate while you are still developing. As engineers we crave the objective and quantifiable world of performance. So we focus on that even when it isn’t important. Most of the time it isn’t important. It reminds me of the joke where a drunk loses his car keys in an alley, but looks for them under the lampost because the light is better there.
Cleverness and performance are major delusions of programming. We let ourselves be seduced by them to the detriment of our code. Unfortunately these are exactly the issues driving much research and development these days. It seems everyone is working on fancy complicated techniques to exploit the performance of multicore systems. I guess that just shows how much I am out of step with the times. So be it.
Update. I apologize to all the Scala enthusiasts who were upset by this post. Scala is an awesome language. But it is new, and it is complicated, and it is way overkill for my needs. Building a VM is really more of a job for C, and Java kind of feels like C to me now: low-level and simple. The point I was trying to make was that I was seduced by the sexy advanced features of Scala even though I didn’t really need them and using it would end up slowing me down.
I also want to thank the people who have stepped in and tried to keep the discussion on a mature level. You don’t see that too often on the interwebs. It is gratifying that my last several posts have attracted some nice thoughtful discussions. Good work guys.
I’d like to see this kind of post substantiated with examples or references to articles.
You’re writing on cleverness and performance as delusions but maybe both topics are related?
For example with Scala 2.9 collections you can have parallel computations just by adding a single method call in the right place. In that case abstraction and elegance allow to think about performance later in the cycle.
Or maybe your point is that all of this is irrelevant if the vm knows all about resources and performance goals?
His point was that Scala’s shiny features are outweighed by its immaturity (as a language).
To quote something I read in a VPRI report:
“User interface design is the little things —
the hundreds and hundreds of little things.”
— Cliff Shaw
It takes a while for a programming language’s ecosystem to expand to the point where you can be really productive — for those hundreds and hundreds of little details to be fleshed out. 🙂
To be fair scala ecosystem is not totally unrelated to java’s. Of course there is not a scala library for everything but there are plenty of java libraries. Once the tools are production ready (http://www.scala-ide.org/), even by using java libraries, you should be at least as much productive with scala than with java.
I totally agree with this post. Developers should be much more aware of why they actually get payed to do what they do. It is all because they can (but not always do) add value for their boss and/or customers/users.
Developers should always be looking to add as much value as possible in the shortest time (read smallest effort) possible.
It’s not helping anyone to never question your own tools and ways of thinking.
The best a developer can deliver is something that is easy to modify and peforms outstandingly in any browser/platform, by which I mean the code is clever and does more with less lines. If you simply forget about these two factors in order to spit out something fast you’ll end up re writting it in a few months.
Yeah they might get payed to deliver products but if those products aren’t maintainable then you are just going to end up spending more money in the future.
I disagree with your complaints about scala. I find the expressivity of modern languages like scala, ruby and coffeescript result in programs that are slightly less readable but way more concise. Conciseness brings all sorts of benefits in testing and refactoring. Having a small program lets you reason about it more easily (the whole thing fits in your head for longer). You can iterate faster because there is way less code. The benefits of succinct software compounds with itself because each module can comfortably do more before it needs to be broken into multiple modules. You end up with a lot less overhead from composition.
A recent java project I worked on had >1M lines of java. Can you believe that? The amazing thing isn’t that iteration was slow, it was that we iterated at all. Programming stopped feeling like reaching to a machine with the hand of god, and started feeling like we spent our lives tightening bolts in the Titanic.
Maybe I’m just a victim of my last failure, but I would take coffeescript + vim over java + eclipse any day. There’s a huge learning curve with scala. But if you don’t get up to speed, you have no reason to think the programming environment you’re making will be any better.
Excellent example that shows our differences in values. You couldn’t make me go back to a text editor and a language without source-level debugging.
Debuggers are a very interesting dividing line for developer culture in that almost everyone either loves them or hates them, with relatively few people in between.
For my own purposes, I’ve always found debuggers to be almost irrelevant because I favor immutability, functional purity and unit testing. Debuggers are a great solution for the problem of having tons of mutable state and not knowing what values different variables are holding; the compositional and functional approach just avoids the problem entirely.
If you like debuggers, I can see why you like Java over Scala – Scala’s idioms probably just don’t fit into your personal way of thinking about problems. When I came to the language, I’d already gone immutable and functional in my Java code (and never used debuggers) and so the transition to Scala just meant that the pain of the Java boilerplate disappeared.
I think that line exists even within a given (procedural) language. Reliance on debuggers can be a sign that the code is out of control, or at least not well understood by he who is charged with finding the bug.
Good insight Kris. I agree that debugging is a sort of litmus test of programmers. It correlates somewhat with functional programming, but plenty of imperative programmers also disdain debuggers.
I observe that the difference is whether or not the programmer tends to have well defined specifications up front. A well defined program can be thought through carefully and constructed very cleanly and reliably without the need for much debugging.
But having specs is just a Utopian fantasy for most programming jobs. In my experience, programming to specs is relatively trivial compared to real world programming.
Is it related to specification, or orthogonality of design elements? In my 13 years in the industry, I’ve never worked in a position where there was anything resembling a good specification (or, at least, where the specification actually reflected what needed to be built.)
The situations I’ve encountered where debuggers have been invaluable have almost all been for low-level bit twiddling applications; for any modern business application that sits atop mountains of framework code, my experience has been that debugging my way through a program to figure out what is going on is almost always slower and more confusing than simply reading the code and writing tests if they don’t exist, or enhancing them if they do. Perhaps with a sufficiently large and insufficiently decoupled codebase this approach will cease to work, and on that day I’ll learn to appreciate debuggers. 🙂
Indeed unit tests are enormously important. But I still use a debugger to diagnose test failures other than the immediately obvious. It just isn’t worth my time pondering the code when I can go in and look at its execution. Some people seem to enjoy the intellectual challenge. I prefer to get work done. In fact my breaking point with Scala came when I discovered IntelliJ didn’t support debugging of unit tests.
A good IDE integrates debugging and testing, and moves breakpoint/watch analysis into its own chalkboard like stack inspection window See Smalltalk variants.
Very weird. I had none of your problems and I started using it when 2.7 was still new.
I love the mature implementation, the great IDE support (IntelliJ) and the stable libraries.
I guess our mileage varies, because I found the exact opposite.
I’m all for simplicity and avoiding delusions that can lead to disasters. But don’t you think what got you thinking of Scala instead of Java was the lack of efficiency when programming in Java.
I know this feeling about fine tuning, performance, and what you call your delusions, but as i’m programming in Java by day and programming in Python at night. Every night, for me, these feelings disappear, just because i’m more efficient and doing more with less.
In regards of the “performance” delusion, to me it’s just that when you’re doing code over code over code etc… you find it more enjoyable to focus on a single piece of code instead of creating another 3 classes of java Beans…
What do you think ?
Agree about performance optimization but disagree on cleverness. Not sure how things being more concise is a bad thing.
Conciseness is bad when it impedes understanding and change. But I am more concerned with the conceptual complexity of clever languages. Complexity and learning curves have a big cost. That’s why I never use C++. Not worth my time.
There’s a learning curve with increased conciseness no doubt. But I’d argue there’s a higher learning curve in learning how someone implemented something with more lines of code as opposed to less. You still have to figure out what they’re trying to do. Don’t fight cleverness and conciseness. It’s a brave new world, but it’s also a better world.
You say you’re a compulsive optimizer. I’ve always assumed (based on no research) that this compulsion results from being cradled in the computing equivalent of the Depression Era, resource-wise. It’s hard to change your spots. Since then, the programming whippersnapper has grown up fat on the rich bounty of Moore’s Law, and a lot of modern fatware is the result.
Right-programming is the art of finding the inflection point between obfuscation and efficiency. Anyone is capable of this, regardless of birth date.
Good call. My formative years were spent programming assembly language on minicomputers with <= 1 MIPS. But I still see lots of young programmers who are obsessed with performance, and it is the point of a lot of current research.
I have seen this, too. Anecdotally, it has been driven by greybeard managers from the previous era, though a lot (for me as well) depends on whether the developer is also a user. Even if I am not a target user, I want speed (given X space) because I am running the thing during its development and testing. In that regard, faster code makes for faster development. Again, up to that inflection point.
Yet the UX on a contemporary software platform involves a whole lot of waiting for the computer.
Surely some people over-optimize, but the opposite problem is probably more severe.
I once told my dev team that I was starting a competition. The first developer to find a way of including in their designs every new language feature in the latest release of Java would win a prize … and that prize was the sack! (i.e. you got fired).
I cannot comment on the merits of Scala versus Java, and to do so actually misses the two points being made:
Firstly, writing code that you think is ‘clever’ is often viewed as the complete opposite by your employer and teammates when they have to work with your ‘clever’ code. As Martin Fowler once wrote, “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” (Refactoring: Improving the Design of Existing Code).
Secondly, beware spending time optimizing code until you know when and where you actually have performance problems.
I certainly have no quarrel with those points.
Steve
Stephen R. Palmer
Certified Scrum Practitioner 2009, FDD Associate, Coad-Certified Mentor
http://www.step-10.com
http://www.twitter.com/stephenrpalmer
There’s got to be a balance, though. The competition comment reminded me of my previous workplace–use of Java 1.5 (or beyond) features was strongly discouraged, despite it being recent enough that we were using Java 1.6, because the lead did not understand them. Use of things like Sets or Maps was considered “clever.” Unit testing was considered “clever.”
Seems like lot of your readers work in a single programmer environment or in teams totally without not-so-super-clever developers 😉
Great post…
I see development teams fall into such traps over and over again. They loose sight of the big picture. Understanding the customer value associated with every single decision and going through the cost/benefit analysis of such decisions is crucial.
Serdar
Great post. Totally agree.
Ignore the Scala fans rant – no one seems to understand that it is irrelevant to the point you are trying to make 😉
Is it an age thing?
A lot of developers grew up in an age where performance really mattered. Speed was a constant struggle and it became second nature to optimize as we coded.
Nowadays of course the optimizations are probably not where we think they are, and mostly are not needed, but we may still have the mindset of 20 years ago.
Do younger programmers still do the same?
At 23 I think I would be considered a “young programmer” and I would say that for younger developers, consideration of efficiency will vary widely depending on their field and educational background. As you say, performance is less of an issue with the computing power of today. There are many situations where you can completely disregard performance and still write a perfectly usable piece of software. I do try to make my code fairly efficient as I write it, as any good developer should, but my view is that in most cases readability and overall productivity are more important than performance.
Also, I do find myself worrying about things which I know I don’t need to care about such as “Can I use a bit shift instead of a divide there” or “should I nest these conditions to avoid repeating that comparison”. I put this down to my attention to detail and the fact that I studied a good course at a good university which taught me about more than just writing code that works.
You seem to be conflating Scala with premature optimisation, and badly misrepresenting it in the process. I can only assume that you’ve been using Eclipse as your IDE, which has indeed been immature until *very* recently – many Scala devs are now using IntelliJ, which is absolutely at a level where it can be used in production.
My own experience of Scala is that it leads to far cleaner code, which expresses the core logic of an application far more readily, no more tracking down exactly where a particular variable was mutated, or trying to read through all the boilerplate; the endless repetition of types, broken equals/hashCode implementations, screenfulls of getters/setters, and fat bloated SAM types.
With Scala, I often find that it’s obvious figuring out how to do something simply and elegantly, usually as a test in the REPL first (far more powerful than any debugger) With Java, I often find that I forget the original goal once I’ve finally got all the boilerplate in place.
I had exactly the same experience with Scala. I had high hopes because the language is so elegant in principle (notwithstanding the complexity of the type system). But alas the tooling made the whole project far more difficult and painful then it needed to be.
I hope you come up with a Java library which I can use from Scala in the end.
The main reason why I liked Scala for your software is that its functional, and that one of your papers is dealing with the opposite of functions, reactions. A reaction library for Scala would be very useful.
Also Scala is so powerful now, Scala 2.9 has Parallel collection library.
OpenCL in Scala allowing you to use the Graphics card to do computation.
Gridgain to allow you to distribute your computation to any number of computers in your “cloud”/network LAN. Program one Scala program and it will distribute the class files out to all the other nodes to run immediately.
(Took off the http:// of the following due to your spam filter)
code.google.com/p/scalacl/
kotek.net/blog/quick_look_at_upcoming_parallel_collections_in_scala_2.9
http://www.gridgain.com/
This is all very high level but assuming you had a large number of computers in a LAN, some with Graphics cards and multi-threaded CPU’s, you could do a lot of very useful computation from one single Scala program.
I understand the desire to optimize, sometimes I just want to program back in Assembly, but then I realize sadly how restricted I would be. It would be easy to write very efficient code in Assembly and use the powerful instruction set of the 64 bit CPU such as the SIMD instruction set, but then I would not be able to do higher level things easily.
http://www.returninfinity.com/baremetal.html
My experience with Scala was initially quite rough. I started learning Scala in 2008. At that time IDE suppory was limited and – more importantly – I lacked a compelling use case for Scala. Like a hammer seeing nails where none existed I was bound for disappointment.
The biggest problem for me was getting into a giant toybox of language features an wanting to try them all. However, most developers aren’t like that and care most about getting things done and have laser focus on that. For these people Scala will work brilliantly. For people like me, all it takes is a reminder that abstractions only have value when they improve readability, maintainability or both. That’s actuallu rather easy and I think fears of unfettered abstraction complexity are largely FUD.
The epiphany happened for me when I needed to write complex HTML email notifications for the commercial product I work on. I initially built the emails using StringBuilder, then tried JSP, thebcwrote a very simplistic Django-esque template engine, then tried jangod and finally remembered Scala’s exceptional literal XML support. Using Scala’s XML I was able to factor the HTML email notifications into configuration-like classes that effectively filled in the blanks in a common HTML template. The result was compile-time verified HTML emails with very little code beyond the essential HTML itself.
The result of using Scala XML for HTML emails has been so compelling that several of my colleagues have asked if it can be used for rendering the web app. I responded that I did not think it would be effective since it lacks the convenience and safety of tested visual components we use in JSF.
I use JetBrains IntelliJ IDEA 10.5 EAP with the latest nightly Scala plugin. So far I have seen no show-stoppers.
I think the key to adoption of Scala in a Java world is moderation. Like any other tool it should be used only where it improves productivity and maintainability. There are undeniably many places where judicious application of Scala has undeniable benefits.
Is Scala really “so complicated”.
I don’t find it complicated at all and I picked it up very quickly. Maybe you just aren’t very intelligent.
Show some respect if you intend to get any.
Mmm tasty replies.
Short of spam, that is quite simply the most useless response I could imagine, and clearly doesn’t show off the intelligence that you claim to have so much of.
I sincerely hope that other readers are able to demonstrate a great deal more wisdom and not take your attitude as being representative of the Scala community, if this sort of thing were typical then I most certainly wouldn’t be involved myself.
Rest assured, it *is* possible to write some extremely complex code in Scala, typically to deal with problems that you would only partially solve in Java, or use a *lot* of duplication, or sacrifice type-safety.
Troll successful.
Personally I find comments such as “It was insane to try using Scala instead of Java. A language so complicated that few people can fully understand it” more annoying than you find my trolling.
Perhaps you do, but that’s not the correct way to respond, and it absolutely doesn’t justify getting those of us who successfully use Scala in production branded as abusive trolls.
Stop being such a muppet and grow up. One day, your resumé will be sitting across the desk of someone who has most likely read this blog post – and I’m quite sure that the resulting experience won’t be justified by the few moments of amusement that you’ve found here.
All trolling aside I find your suggestion that the actions of a single troll reflect poorly on an entire community of developers ludicrous. I dislike even the tenuous link you have formed between programmers and racists.
Regardless of these points I stand by my initial statement both logically and trollogically.
I might also point out that it’s very funny that you keep replying.
Update: I apologize to all the Scala enthusiasts who were upset by this post. Scala is an awesome language. But it is new, and it is complicated, and it is way overkill for my needs. Building a VM is really more of a job for C, and Java kind of feels like C to me now: low-level and simple. The point I was trying to make was that I was seduced by the sexy advanced features of Scala even though I didn’t really need them and using it would end up slowing me down.
I also want to thank the people who have stepped in and tried to keep the discussion on a mature level. You don’t see that too often on the interwebs. It is gratifying that my last several posts have attracted some nice thoughtful discussions. Good work guys.
It is impossible to have an unemotional debate about Scala: once you get addicted to Scala it is difficult to go back. Having gone through withdrawal myself I know this first hand. Scala has so many great features that you want to use them in your program, you spend time thinking about the most elegant way to do something because you can express elegance in Scala, you spend time hacking your types to get them to be just right. Scala is very much like C++ in this regard, both are very clever languages that tend to attract clever programmers who are pursuing elegance. I would still give Scala an advantage over C++ in that Scala has a much better and safer design, but the fundamental power issues are similar.
When I was forced to move to C# (job requirement being that I started working for MS), I went through a lot of withdrawal pains because C# just lacks so many features compared to Scala: no traits, no pattern matching, no type variables, etc. But then I discovered that less can be more. I spent less time pondering how to leverage C#’s features to write elegant code, because frankly there weren’t that many, and more time hacking out functional but inelegant code.
There have been claims that a programmer is more efficient when they use Scala. That is definitely true when you consider keystrokes made, but that isn’t a really big part of programming. Scala does allow for better/more reuse of code, so I’m sure there is a savings there, but you have to spend a lot of time coming up with abstractions for the code you will reuse, so you lose time there.
I still miss programming in Scala, and I’m not entirely sure that my work is really easier with C#. However, I do believe that a “simpler Scala with fewer choices” would be a better Scala for most people.
Well said, Sean.
> simpler Scala with fewer choices” would be a better Scala for most people.
Mirah? Clojure? 🙂
>but you have to spend a lot of time coming up with abstractions for the code you will reuse, so you lose time there.
But wasn’t that the case with OO too? It seems to me the most common argument against Scala (paraphrased) “I’m not used to learning while I program anymore, and it’s slowing me down”.
Yes, your productivity goes backwards because you’re *learning as your going*, just like when you were brand new to OO or programming in general. It doesn’t stay like that forever, and I look at each time I spend 3x as long on a Scala program as an *investment* into my abilities.
>However, I do believe that a “simpler Scala with fewer choices” would be a better Scala for most people.
I’ve been pushing this for a while. I’d love it if there was a way to make the Scala compiler enforce a ‘Scala-lite’ version. One that did away with implicits, operator overloading, syntactic sugar, etc. Sure those features are great when creating a DSL, but 90% of the time they aren’t needed, and nobody wants the new guy fresh out of school to start using them in his CRUD code.
Maybe there can be made compiler plugin, prohibiting some certain features ?
BTW, for simple tasks, procedural approach may really be better, than OOP boilerplates. For example – command-line scripts, be it in Python, JavaScript or PowerShell, i think en masse are more procedural than OOP. Even by people knowing OOP.
// And even the famed “concise” functional programming, may we sometimes see it as “more procedural, than procedural” ? 🙂 //
So, if to agree with Scala similarity to OOP dawn, there probably are really many-any tasks, where Scala is too much, even if person is no more learning Scala and time goes to programming only.
ok, Scala.Net always seemed born dead to me.
But like JVM has Java/Scala pair, the same way .Net has C#/Nemerle pair.
Maybe you’d look at nemerle.org and find the same thrills of advanced language and immature tools, like you had with Scala ? All this on you .Net platform of work 🙂
“unlike most of the issues that really matter: subjective design issues with no clear cut answer, which can be almost impossible to evaluate while you are still developing”
I think user interface designers would disagree that the issues that really matter — which I read to mean language design with particular focus on the user experience — are entirely subjective and impossible to evaluate.
In fact, UX designers try to evaluate their designs as EARLY as possible, using lo-fi techniques such as paper prototyping, and continually refine their designs during product development guided by empirical experiments.
Are programming languages any different? If so, why?
(Personally I do not believe they are).
Good point, Nat. Actually I was referring mostly to internal design issues, those observable only by the programmers. External interfaces are easier, because we have fewer degrees of freedom, and we often have imposed constraints.
I do agree that the design of programming languages is a special case of UI design. In fact a major thrust of my research is that we have let the assumption of a text editor UI tacitly constrain the design of our languages. However I am not a big fan of traditional UI prototyping and evaluation, which I believe only works well for relatively “shallow” issues like layout and mechanical interaction. Deeper issues of conceptual design are much murkier, and not readily resolved by empirical methods.
So, at http://cscott.net/Projects/TurtleScript/ I’m taking the radical/traditional approach of assuming that the text-based representation of programming languages is actually pretty good UI, and just revisiting the “typing in all the code” keyboard part. Dunno how that will turn out yet…
Interesting, good luck.
Nat, I was agreeing with you right until you mentioned empirical experiments. The design studios that I’m familiar with will prototype designs using a combination of lofi and medium fidelity techniques, and they will then try the prototypes out and get qualitative feedback from product stakeholders (user surrogates) as well as actual users (when appropriate). Empirical testing is very expensive and done mostly in the field of HCI research.
By “empirical” I didn’t mean to imply only experiments that produce quantitive results. I just meant that they actually observe the UI in use in as realistic a situation as they can create, and adapt the design based on those observations.
Which is quite different from how most programming languages are designed.
Sorry for the two-comment reply… I hit enter too fast.
Also, I think more places do quantitive research than you say. For example, many websites do A/B testing, which is quantitive, empirical research of their UI design options using their actual users as a cohort.
I’ve been doing games programming for some years now, but along the way have dabbled with language design among other things. I’ve gone through a long process of iterating ongame engine design, often turning my previous thinking upside down.
The one thing that stands out to me now is how important it is to be able to modify code later. Getting the architecture nailed and watertight – not really, it can be 80% appropriate and the risk of catastrophe will be low. Getting it fast – again, “within the ballpark” is usually OK, even in the Flash-based stuff I’ve been working with. I used to fret a lot over every last division – haven’t done that in a while. But being able to reach in and change things – that’s really the key. And it usually works to only rely on simple bread-and-butter ideas – function calls, iteration, references, mutability. The only exception I’ve seen is of a few abstractions with massive application to the domain – in the case of games: tools to declare, queue and order real-time events, automation of the indexing and categorization of game entities, and parceling out entity data and behavior into a set of components.
Without those abstractions guiding things – ugly structure, spaghetti, poor reusability. With them – things work in a magical way.
On the language level, there also seem to be a few simple things that help me. Garbage collection. Static types – up to a point. Dynamic types – up to a point. (I work in haXe at present, which allows some mixing of static-compiled and runtime-dynamic – best of both worlds.) Serialization. Platform availability, build systems, packaging tools. And good libraries and APIs. There are some “can be nice” things but they are more domain oriented and can usually be had by dumping a little interpreter on top at some point.
Jonathan,
Have you heard about xtext?
Scala is a destined to be the C++ of JVM languages. Obviously, it is a powerful language. That’s not the issue. The issue is that, metaphorically speaking, effective use of Scala requires a commitment like marriage, where as at this point, most of us require a bevy of girl friends. And like most marriages, it is entirely a mystery looking from outside in!