Josh On Design

Is software getting better or worse? Some say we are making software ever more bloated. Some say we don’t care about quality anymore; that worse is better. Some say we haven’t changed how we write software in 40 years. It's still ASCII text on disk. (Yes that would be me, saying that). Certainly our programming languages haven’t improved. We still write billions of lines in glorified C code!

Despite the problems -- and all of the above complaints are both real and valid -- I think that yes, we do write better software than we did 40 years ago. We do make software that both does more and is more reliable than ever before. Furthermore, not all of the gain can be attributed to Moore’s Law.

The improvements have not come from improved programming languages. The early years of computing research (1950s-70s) had a Cambrian explosion of programming language design. Everything that’s come out since the early 80s has looked pretty familiar. Smalltalk was the last truly new take on PL design.

It’s the Tools

While we might be at a local maxima for programming language design, it’s actually a pretty good maxima.  I want to believe there is untapped potential in a brand new design,  but what we have today works pretty damn well. A developer from 1975 could look at modern Javascript and feel pretty comfortable. Garbage collection, runtime typing, closures, and object inheritance all existed in some form 40 years ago. We have a pretty mature system now. Software improvement has not come from better languages. It is the tools that have made us better.

I argue that almost all of the improvements have come from four areas: test driven development, version control, continuous builds, and library ecosystems. The use of these tools has let us build software that truly scales to billions of people.

Test driven development

Sometime in the early 2000s I started to care about the quality of the software I was writing. I hated fixing the same stupid bugs over and over. I’d read the patterns book and heard about Scrum and Agile, but the real break through for me was test driven development.

With TDD you don’t write the tests to detect problems in the code.  Instead you write the tests first to define what correct behavior actually is. Once that's done you can just hack on the code until all the tests pass. You can’t correctly write code until you know what it’s supposed to do.  It’s so obvious in retrospect, but this was a huge eye opener for me.

Version Control Systems

VCSs let us share code with others, and with our future selves. They let us go back in time. They let us make sense of how a code base changes and plan our future path in a reasonable way. Making VCSs distributed and decentralized (ala Git & Mercurial) has certainly helped, but it’s the sharing over the network that makes them truly valuable. I think I first used CVS in the late 90s and never worked on an un-versioned project again. I create git repos even for code I never plan to share. It's just damn useful and essentially free. There's no excuse not to use a VCS now.

Continuous Build Systems

The two previous tools enable this third productivity booster: continuous building. Once your code is on the network with automated tests the next obvious step is to run those tests as much as possible. We can catch errors, regressions, and performance problems immediately when they happen, or as close to immediately as possible. Standardized build systems (which all still suck, BTW) have really helped here.

Ecosystem of reusable libraries

I feel like Java really kicked off the trend of wide scale reusability with it’s built in standard library, but it was websites sharing jars that really made a difference. Apache Jakarta and public code hosting (alas, SourceForge, we miss thee) took the ecosystem to the next level. Once people could easily share code there was an explosion of libraries. Developers now pick a framework or language based not on it’s technical merits, but on the community of developers and libraries around it.

Most importantly almost all of these libraries are open source. In theory an open license ensures all bugs are shallow to enough eyeballs, but in practice we still see egregious bugs in open source code. It’s not magic pixie dust. The magic part is simply that open source licenses like Apache and MIT lower the friction to trying out a new library. Friction is now so low that we have networked module systems that simply import the library on demand with no extra interaction. Type in the lib you want and you get it automatically from the Internet. No muss, no fuss. The growth of ecosystems has proven far more transformative than the growth of languages.

Things that haven’t helped

IDEs:  Okay. I suppose they’ve helped a little. I certainly feel more productive with IntelliJ or Webstorm than with a plain text editor.  I do love my code completion, but I don’t think the improvement is on the same scale. IDEs are a 2x or 3x productivity improvement, not the 10x gain from an open library ecosystem. The best code is the code you don’t have to write at all.

Language design:  While I’m hopeful that Rust will break the stranglehold of C... I’m not holding my breath.  We continue to use unsafe programming languages (yes, even Java and C# have unsafe corners).  We’ve worked around these problems with tools and systems and libraries, not by improving the language. 

Better computers:  If anything, the continual gains of Moore’s law has made software worse. We can afford to ship unoptimized broken software and just patch it later. Who cares if an editor takes up half a gig of ram? My new laptop came with eight!

What about the next 40 years?

So where do we go from here? I know that forty years is a very long time but I’ll make a few meager predictions anyway. After all, the Starship Enterprise won't run on C code, right? (..right?! ..please?!)

Abstract Syntax Tree Editors: The time for AST based editors has come but the benefits will be not from a new representation on screen. You will still see code that mostly looks like ASCII text. No crazy blocks and lines diagrams. The benefits will come from other tools that we can build around the AST language.  

What if you could pattern match all of the code in Github? What if your language could support any Domain Specific Language extensions without modifying the underlying language? What if you could share code with people writing in other languages on the other side of the planet, and have it translated, built and tested on the fly?  An AST language will let IDEs really shine with structure visualization, intelligent debuggers, and shared editors.

Check out this video of Moonchild by Patrick Dubroy.

Call by Meaning: Take module systems to the next level with Call by Meaning by Hesam Samimi, Chris Deaton, Yoshiki Ohshima, Alessandro Warth, and Todd Millstein. Discover and stitch components together by what they mean, not their particular API. It will be mind blowing if they can get this to actually work.

Testing Engines:  Right now you write tests specific to your code. Imagine intelligent testing software that puts your code through a battery of automated tests built on common design patterns. 

Consider physical testing tools. They sure the object can handle high and low temps, high pressure, and stress each side until it breaks. We could do the same thing with software. 

Put your codebase through a battery of tests to determine how much load it can handle, what files and network resources it needs, and what conflicts come from changing operating systems and versions.  A lot of these tests are done today but in a very ad hoc way. I think we’ll see it happen in a more systematized way as tools like Docker and the Android emulator become more common.

Teaching Kids to Code: Finally I think we’ll write better software just because we will be teaching more kids how to code at a much younger age. I don’t mean teaching how to use a particular language. While that is useful, the real value comes from the problem solving skills.  

Given a coding problem the child must figure out what resources they have, what libraries to use, and to carefully define the problem in a testable way. Then the kid can start actually writing code, with lots of experimentation and debugging along the way. Iteration is the key. Coding is problem solving, not learning how to match up curly braces.

Software really has greatly improved in forty years; but it's the tools and patterns that have gotten us there, not the languages and compilers.