I need to move on to other projects so I’m wrapping up the rest of my ideas in this blog. Gotta get it outta my brainz first.

The key concept I’ve explored in this series is that the code you see in an editor need not be identical to what is stored on disk, or the same as what is sent to the compiler. If we relax this constraint then a world of opportunity opens up. We’ve been writing glorified text files for 40 years. We can do better. Let’s explore.

Keywords

Why can’t you name a variable for? Because in many common languages for is a reserved word. You, as the programmer, aren’t allowed to use for because it represent a particular loop construct. The underlying compiler doesn’t actually care of course. It doesn’t care about the name of any of your variables or other words in your code. The compiler just needs them to be unique symbols, some of which are mapped to existing constructs like conditionals and loops.

If the compiler doesn’t care then why can’t we do it? Because the parser (the ‘front end’ of the compiler) does care. The parser needs to unambiguously transform a stream of ASCII text into an abstract syntax tree. It’s the unambiguous part that’s the trouble. The syntax restrictions in most common languages are there to make the parser happy. If the parser was magical and could just "know what we meant" then any syntax could be used. Perhaps even syntax that made more sense to the human rather than the computer.

Fundamentally, this is what typographic programming does. It lets us tell the parser which text is what without using specific syntax rules. Instead we use color or font choices to indicate whether a given chunk of text is a variable or keyword or something else. Of course editing in such a system would be a pain, but we already know how to solve that problem. Graphical word processors are proof that it is possible. Before we get to how we solve it let us consider why. Would such a system have enough benefits to outweigh the cost of building it. What new things could we do?

Nothing’s reserved

If we use typography to indicate syntax, then keywords no longer need to be reserved. Any keyword could be used as a variable and any text string could be used as a keyword. You could swap for with fore or thusly. You could use spaces in keywords as for each of. These aren’t very useful examples but the compiler could easily handle them.

With the syntactic restrictions lifted we are free to explore new control flow constructs. How about forever to mean an infinite loop and 10 times for standard for fixed length loops? It’s all the same to the compiler but the human reading it would better understand the meaning.

Custom Operators

If nothing is reserved then user defined operators become easy. After all; what is an operator but a function with a single letter name from a restricted character set. In Python 4 + 5 is just sugar for add(4,5).

With no syntax rules anything could be an operator. Operators could have multiple letter names, or other symbols from the full unicode set. The only reason operators are given special treatment to begin with is because they represent functions which are so commonly used (like arithmetic) that we want a shorthand. With free syntax we can create a shorthand for the functions that are useful to the task at hand rather than the abstract general purpose tasks the language inventors imagined.

Let’s look at something more concrete. Using complex numbers and vectors is common in graphics programming, but we have to use clumsy and verbose syntax in most languages. This is sad. Mathematics already has invented compact notation for these concepts but we can’t use them due to ASCII limitations. Without these limitations we could add complex numbers with the plus sign like this:

A +B

instead of

complexAdd(A,B)

To help the programmer remember these are complex numbers they could be rendered in a different color.

There are two ways to multiply vectors: the dot product and the cross product. They have very different meanings. With full unicode we could use the correct symbols like this:

A ⋅ B  // dot product
A ⨯ B // cross product

No ambiguity at all. It would be too much to expect a language to support every possible notation. Much better instead to have a language that lets the programmer create their own notation.

Customization in Practice

So how would this work in practice? At some point the code must be transformed into something the compiler understands. Let’s postulate a hypothetical language called X. X has no syntax rules, only the semantic rules of it’s AST. To tell the complier how to convert the code into the AST we must provide our own rules. Something like this.

fun => function
cross => cross
dot => dot
|x| => magnitude(x)

fun intersection(V,R) {
     return V dot R / |V|;
}

We have now defined a mini language in X which still compiles to the same syntactic structure.

Of course typing all of these rules in every file (or compilation unit) would be a huge pain, so we could include them much as we include external libraries.

@include math.rules

fun intersection(V,R) {
     return V dot R / |V|;
}

Most importantly, not only would the compiler understand these rules but so would the editor. The editor can now indicate that V ⋅ R is valid only if they are both vectors. It could enforce the rules from the rule file. Now our code is limited only by the imagination of our rule writers, not the fixed compiler.

In practice, were X to become popular, we would not see everyone making up their own rules. Instead usage would gather around a few popular rulesets much as JavaScript gathered around a few popular libraries like JQuery. We might call each of these rulesets dialects, each a particular flavor derived from the base X language. Custom DSLs would become trivial to implement. It would be common for developers to use one or two "standard" dialects for most of their code but use a special purpose dialect for a particular task.

The important thing here is that the language no longer has a fixed syntax. It can adapt and evolve as needed. All without changing the compiler.

How do you edit?

I hope I’ve convinced you that a flexible syntax delimited by typography is useful. Many common idioms like iteration, accumulation, and data structure traversals could be distilled to concise syntax. And if it has problems then we can tweak it.

There is one big problem though. How would you actually edit code like this?

Fortunately this problem has already been solved by graphical word processors. These tools use color, font, size, weight and spacing to distinguish one element from another. Choosing the mode for a variable is as simple as selecting it with the cursor and using a drop down.

Manually highlighting a entire page of code would quickly grow tedious, of course. For common operations, like declaring a variable, the programmer could type a special symbol like @. This tells the editor that the next letters are a variable name. The programmer ends it with @ or by pressing the spacebar or return key. This @ symbol doesn’t exist in the code. It is simply to indicate to the editor that the programmer wants to be in ‘variable’ mode. Once the mode is finished the @’s go away and the text is rendered with the ‘variable’ font. This is no different than using star word star to indicate bold in Markdown text. The stars never appear in the rendered text.

The choice of the @ symbol doesn't matter as long as it's easy with the user's native keyboard. @ is good for US keyboards. French or Russians might use something else.

Resolving Ambiguity

Even using manual markup might become tedious, though. Fortunately the editor can usually figure out the meaning of any given token by using the dialect rules. If the rules indicate that # equals division then the editor can just do the right thing. Using manual highlighting would only be necessary if the dialect itself introduces an ambiguity. (ex: # means division and also the start of a hex value)

What about multiplying vectors? You could type in either of the two proper symbols, but the average keyboard doesn’t support those directly. You’d have to memorize a unicode code point or use a floating dialog. Alternatively, we could use code completion. If you type * then the editor knows this must be either dot or cross product. It provides only those two choices in a drop down, much as we auto-complete method names today.

Using a syntax free language does not fully remove the need to resolve ambiguity, it just moves the resolution process to edit time rather than compile time. This is good. The human is present at edit time and can explain to the computer was is correct. The human is not there at compile time, so any ambiguity must result in an error that the human must come back and fix. Furthermore, resolving the ambiguity need only happen once, when the human types it, not every time the code is compiled. This will further reduce code regressions when other parts of the system change.

Undoubtedly we would discover more edge cases, but these are all solvable. Modern GUI word processors and spreadsheets prove this. A more challenging issue is version control.

Versioning

Code changes over time. It must be versioned. I don’t know why it took 40 years for us to invent distributed version control systems like Git, but at least we have it now. It would be a shame to give that up just as we’ve gotten the world on board. The problem is Git and other VCSs don’t really understand code. They just understand text. There are really only two ways to solve this:

1) modify git, and the other tools around it (diff viewers, github’s website, etc.) to support binary diffs specific to our new system.

2) make the on disk format be pure text.

Clearly option 1 is a non-starter. One day, once language X takes over the world, we could ask the GitHub team to add support for X diffs, but that’s a long ways off. We have to start with option 2.

You might think I’m going back on what I said at the start. After all, I stated we should no longer be writing code as text on disk, but that is exactly what I am suggesting. What I don’t want is to store the same thing that we edit. From the VCS’s point of view the editor and visual representation are irrelevant. The only thing that matters is what is the file on disk. X needs a canonical on serialization format. Regardless of what tool you use to edit X, as long as it saves to the same format we are fine. This is no different than SQL or HTML. Everyone has their favorite tool, but they all write to the same format.

Canonical Serialization Format.

X’s serialization format should obviously be plain text. < 128bit ASCII would be fine, though I think we could handle UTF8 easily. Most modern diff tools can work with UTF8 cleanly, so Japanese comments and math symbols would come through just fine.

The X format should also be unambiguous. Variables are marked up explicitly as variables. Operators as operators. There should be no need for the parser to guess at anything or interpret syntax rules. We could use one of the many existing formats like JSON, XML, or even LaTex. It doesn’t really matter since humans will rarely need to look at them.

But.... since we are defining a new serialization format anyways, there are a few useful things we could add.

Code as Graph

Code is really just a graph. Graphs can be serialized in many ways. Rather than using function names inline they could be represented by identifiers which point to a lookup table. Then, if a function is renamed the code only changes in one place rather than at every point in the code where the function is used. This creates a semantic diff that the diff tool could render as ‘function Y renamed to Z’.

v467 = foo
v468 = bar
v469 = baz

fun v467 () {
return v468 + v469;
}

Semantic diff-ing could be very powerful. Any refactoring should be reducible to its essential meaning: moved X to a new class or extracted Y from Z. Whitespace changes would be ignored (or never stored in the first place). Commit messages could be context aware: changed X in the unit test for Y and added logging to Z. Our current tools just barely understand when a file has been renamed instead of deleted and a new one added. There’s a lot of room for innovation here.

WrapUp

I hope I’ve convinced you there is value in this approach. Building language X still won’t be easy. To be viable we have to make a compiler, useful dialect definitions, and a visual editor; all at the same time. That’s a lot of work before anyone else can use it. Building on top of existing tools like Eclipse or Atom.io would help, but I know it’s still a big hill to climb. Trust me. The view will be worth it.

Note: I’m a research at Nokia but this blog does not represent my employer. I didn’t move to Microsoft and I’ve never been on the Windows Phone team. These ill considered opinions are my own.

Windows 10 seems nice and all, but it doesn’t do anything to make me care. Fortunately Microsoft can fix all of Windows problems if only they follow my simple multistep plan. You’re welcome.

First, Fix the damn track pads.

The problem: my employer gave me a very nice, probably expensive, laptop. It’s name rhymes with a zinc fad. It’s specs sure are nice. It’s very fast. but the track pad is horrible. Every Windows laptop I’ve tried (which is a lot because Jesse likes to ‘do computers’ at Costco) has a horrible trackpad. why is this so hard? I simply can’t bear to use this laptop without a mouse. the cursor skips around, gestures don’t work all the time, and it clicks when i don’t and it doesn’t click when I do.

The fix: Take over trackpad drivers and make a new quality test for Win10 certification. It used to be that every mouse and keyboard needed it’s own driver, and they were usually buggy. I bought Logitech trackballs in the 90s because they seemed to be the only guys who cared to actually test their drivers (and the addon software was mildly useful). Sometime in the early USB days (Win98ish?) MS made a default mouse and keyboard driver that all devices had to work with. Since then it’s never been an issue. Plug in any mouse and it works perfectly. 100% of the time. MS needs to do the same for trackpads.

Please write your own driver for the N most popular chipsets, standardize the gesture support throughout the OS, then mandate a certain quality level for any laptop that wants to ship windows 10. Hey OEM: If it’s not a Macbook Air quality trackpad experience then no Windows 10 for you.

Make A Proper Command Line Shell

Hide PowerShell, Cygwin, whatever Visual Studio ships with (it has a shell, right?) and the ancient DOS prompt. Make a proper terminal emulator with Bash. (Fix the bugs first). Build it in to the OS, or at least as a free developer download (one of those MS Plus thingies you promised us).

This shell should be fully POSIX compliant and run all standard Unix utilities. I understand you might worry that full POSIX would let developers port code from another platform instead of writing natively for you. That is very astute thinking… for 1999. Unfortunately we live in the futuristic hellscape that is 2014. You need to make it as easy as possible for someone to port code to Windows. Eliminate all barriers. Any standard Unix command line program should compile out of the box with no code changes. Speaking of which..

Give your C++ compiler a GCC mode.

For some reason all ANSI C code compiles perfectly on Mac and Linux but requires special #IFDEFs for Windows.h. Slightly different C libs? Sightly different calling syntax? None of this cdecl vs stdcall nonsense. Make a --gcc flag so that bog standard Linux code compiles with zero changes. Then submit patches to GNU Autoconf and the other make file builders so that this stuff just works. Just fix it.

Build a Package Manager

Now that we have a proper command line Windows needs a package manager. I use Brew on Mac and it works great. I can install any package, share formulas for new packages, and keep everything up to date. I can grab older versions of packages if I want. I can switch between them. Everything all works. Windows needs this, and it should work from both a GUI and CLI.

I know Windows has NuGet and Chocolatey and supposedly something is coming called OneGet. There needs to be one official system that really works. It handles all dependencies. And it should be easy to use with no surprises.

"What surprises?" I hear you say? I wanted to install Node. I couldn’t figure out which package manager to use so I chose Chocolatey since it seemed to be all the new hotness. I go to their website and find four different packages: Node JS (Install), Node JS, Node JS (Command Line), Node Package Manger. What? Which do I choose? They all have a lot of downloads. On every other platform you just install Node. NPM is built in. There are no separate packages. It’s all one thing because you can’t use part of it without the rest.

NodeJS is an alias for NodeJS.commandline. NodeJS.commandline installs to the Chocolatey lib dir. NodeJS.install installs to a system dir. It turns out Chocolatey has both installable and portable packages. As near as I can tell they are identical except for the install path, which is something I shouldn’t have to care about anyway. Oh, and one way will add it to your path and the other won’t. What? Why should I have to care about the difference? Fix it!

I really hope OneGet straightens all of this nonsense. There should be just one way to do things and it must work 100% of the time. I know Microsoft has MSDN subscriptions to sell, but that’s just another repo source added to the universal package manager.

Make Visual Studio be everywhere.

Visual Studio is really an impressive piece of software. It’s really good at what it does. The tooling is amazing. Microsoft needs to let the world know by making it be everywhere.

If you are on Windows, you should get a free copy of VS to download. In theory this is the idea behind Visual Studio Express. So why do I still use Atom or Sublime or even JEdit on Windows? Partly because of the aforementioned package manager problem, but also because Visual Studio isn’t good for all kinds of coding.

Visual Studio is primarily a C/C++ editor, meant for MS’s own projects (and now hacked up for WinPhone and presumably WinRT). They should make it good for everything.

Are you a Java programmer? VS should be your first choice. It should have great Java language support and even version the JDKs with the aforementioned package manager.

Are you a web developer? VS should have awesome HTML and JavaScript support, with the ability to edit remote files via sftp. (Which Atom still doesn't have, either, BTW).

And all of this should be through open hackable plugins, also managed through the package manager. VisualStudio should be so good and so fast that it’s the only IDE you need on Windows, no matter what you are coding.

Why should Microsoft do this? After all, they would be putting a lot of effort into supporting developers who don’t code for their platform. Because Microsoft needs developer mindshare.

I know very few topflight developers who use Windows as their main OS. Most use Macbook Pros or Linux laptops. MS needs to make a development experience so good that programmers will want to use Windows, even if it’s just for web work.

Once I use Windows every day I might take a look at MS’s other stacks. If I’m already using Visual Studio for my JavaScript work then I’d be willing to take a look at developing for Windows Phone; especially if it was a single download within a program I already have installed. Fix it!

Buy VMWare.

You want me to test new versions of Windows? It should be a single click to download. You want me to use your cloud infrastructure? If I could use Visual Studio to create and manage VM instances, then it’s just a single button to deploy an app from my laptop to the cloud. MS’s cloud, where they real money from the movie is made. Buy VM ware to make it happen if you need to. I don’t care. Just fix it!

Be open and tell the world.

MS has always had a problem with openness. They have great technology but have always felt insular. They build great things for Windows Devs and don’t care about the rest of the world. Contribute to the larger community. Make Visual Studio for all sorts of development, even those that don’t add to the bottom line.

Maybe Visual Studio Express already has all sorts of cool plugins that make web coding awesome. I will never know because MS doesn’t market to me. All I hear is “please pretty please make some crappy Windows Phone apps”.

Maybe OneGet fixes all of the package management problems, but I didn’t even know about it until I was forced to use a Windows laptop and did my own research.

Fix It !

Here is the real problem. MS has become closed off. An ecosystem unto itself. This is a great strategy if you are Apple, but you aren’t. Your a software company become a cloud company. You must become more open if you expect your developer mindshare to grow. And from that mindshare new platforms will grow. When MS releases their Windows smart watch, or the Windows Toaster, they will find it a lot easier to get developers on board if they’ve built an open community up first. Efforts like CodePlex are nice, but this philosophy has to come from the top.

Good Luck Nadella.

Lately I've been digging into Rust, a new programming language sponsored by Mozilla. They recently rewrote their docs and announced a roadmap to 1.0 by the end of the year, so now is a good time to take a look at it. I went through the new Language Guide last night then wrote a small ray tracer to test it out.

One of the biggest success stories of the last three decades of programming is memory safety. C and C++ may be fast but it's very easy to have dangling pointers and buffer overflows. Endless security exploits come back to this fundamental limitation of C and C++. Raw pointers == lack of memory safety.

Many modern languages provide this memory safety, but they do it at runtime with references and garbage collection. JavaScript, Java, Ruby, Python, and Perl all fall into this camp. They accomplish this safety at the cost of runtime speed. While they are typically JITed today instead interpreted, they are all slower than C/C++ because of their runtime overhead. For many tasks this is fine, but if you are building something low level or where speed really matters, then you probably go back to C/C++ and all of the problems it entails.

Rust is different. Rust is a statically typed compiled language meant to target the same tasks that you might use C or C++ for today, but it's whole purpose in life is to promote memory safety. By design, Rust code can't have dangling pointers, buffer overflows, or a whole host of other memory errors. Any code which would cause this literally can't be compiled. The language doesn't allow it. I know it sounds crazy, but it really does work.

Most importantly, Rust achieves all of these memory safety guarantees at compile time. There is no runtime overhead, making the final code as fast as C/C++, but far safer.

I won't go into how all of this work, but the short description is that Rust uses several kinds of pointers that let the compiler prove who owns memory at any given moment. If you write up a situation where the complier can't predict what will happen, it won't compile. If you can get your code to compile then you are guaranteed to be memory safe.

I plan to do all of my native coding in Rust when it hits 1.0. Rust has a robust FFI so interfacing with existing C libs is quite easy. Since I absolutely hate C++, this is a big win for me. :)

Coming into Rust I was worried the pointer constraints would make writing code difficult, much like puzzle languages. I was pleasantly surprised to find it pretty easy to code in. It's certainly more verbose than a dynamic language like JavaScript, but I was able to convert a JS ray tracer to Rust in about an hour. The resulting code roughly looks like what you'd expect from C, just with a few differences. Let's take a look.

First, the basic type definitions. I created a Vector, Sphere, Color, Ray, and Light class. Rust doesn't really have classes in the C++/Java sense, but it does have structs enhanced with method implementations, so you can think of them similar to classes.

use std::num;


struct Vector {
    x:f32,
    y:f32,
    z:f32
}

impl Vector {
    fn new(x:f32,y:f32,z:f32) -> Vector {
       Vector { x:x, y:y, z:z }
    }
    fn scale(&self, s:f32)    -> Vector  { 
       Vector { x:self.x*s, y:self.y*s, z:self.z*s }
    }
    fn plus(&self, b:Vector)  -> Vector  { 
      Vector::new(self.x+b.x, self.y+b.y, self.z+b.z)
    }
    fn minus(&self, b:Vector) -> Vector  {
       Vector::new(self.x-b.x, self.y-b.y, self.z-b.z) 
    }
    fn dot(&self, b:Vector)   -> f32     { 
      self.x*b.x + self.y*b.y + self.z*b.z 
    }
    fn magnitude(&self)       -> f32     { 
      (self.dot(*self)).sqrt() 
    }
    fn normalize(&self)       -> Vector  { 
      self.scale(1.0/self.magnitude())  
    }
}

struct Ray {
    orig:Vector,
    dir:Vector,
}

struct Color {
    r:f32,
    g:f32,
    b:f32,
}

impl Color {
    fn scale (&self, s:f32) -> Color {
        Color { r: self.r*s, g:self.g*s, b:self.b*s }
    }
    fn plus (&self, b:Color) -> Color {
        Color { r: self.r + b.r, g: self.g + b.g, b: self.b + b.b }
    }
}

struct Sphere {
    center:Vector,
    radius:f32,
    color: Color,
}

impl Sphere {
    fn get_normal(&self, pt:Vector) -> Vector {
        return pt.minus(self.center).normalize();
    }
}

struct Light {
    position: Vector,
    color: Color,
}

Without knowing the language you can still figure out what's going on. Types are specified after the field names, with f32 and i32 meaning integer and floating point values. There's also a slew of finer grained number types for when you need tight memory control.

Next up I created a few constants.

static WHITE:Color  = Color { r:1.0, g:1.0, b:1.0};
static RED:Color    = Color { r:1.0, g:0.0, b:0.0};
static GREEN:Color  = Color { r:0.0, g:1.0, b:0.0};
static BLUE:Color   = Color { r:0.0, g:0.0, b:1.0};

static LIGHT1:Light = Light {
    position: Vector { x: 0.7, y: -1.0, z: 1.7} ,
    color: WHITE
};

Now in my main function I'll set up the scene and create a lookup table of one letter strings for text mode rendering.

fn main() {
    println!("Hello, worlds!");
    let lut = vec!(".","-","+","*","X","M");

    let w = 20*4i;
    let h = 10*4i;

    let scene = vec!(
        Sphere{ center: Vector::new(-1.0, 0.0, 3.0), radius: 0.3, color: RED },
        Sphere{ center: Vector::new( 0.0, 0.0, 3.0), radius: 0.8, color: GREEN },
        Sphere{ center: Vector::new( 1.0, 0.0, 3.0), radius: 0.3, color: BLUE }
        );

Now lets get to the core ray tracing loop. This looks at every pixel to see if it's ray intersects with the spheres in the scene. It should be mostly understandable, but you'll start to see the differences with C.

    for j in range(0,h) {
        println!("--");
        for i in range(0,w) {
            //let tMax = 10000f32;
            let fw:f32 = w as f32;
            let fi:f32 = i as f32;
            let fj:f32 = j as f32;
            let fh:f32 = h as f32;

            let ray = Ray {
                orig: Vector::new(0.0,0.0,0.0),
                dir:  Vector::new((fi-fw/2.0)/fw, (fj-fh/2.0)/fh,1.0).normalize(),
            };

            let mut objHitObj:Option<(Sphere,f32)> = None;

            for obj in scene.iter() {
                let ret = intersect_sphere(ray, obj.center, obj.radius);
                if ret.hit {
                    objHitObj = Some((*obj,ret.tval));
                }
            }

The for loops are done with a range function which returns an iterator. Iterators are used extensively in Rust because they are inherently safer than direct indexing.

Notice the objHitObj variable. It is set based on the result of the intersection test. In JavaScript I used several variables to track if an object had been hit, and to hold the hit object and hit distance if it did intersect. In Rust you are encouraged to use options. An Option is a special enum with two possible values: None and Some. If it is None then there is nothing inside the option. If it is Some then you can safely grab the contained object. Options are a safer alternative to null pointer checks.

Options can hold any object thanks to Rust's generics. In the code above I tried out something tricky and surprisingly it worked. Since I need to store several values I created an option holding a tuple, which is like a fixed size array with fixed types. objHitObj is defined as an option holding a tuple of a Sphere and an f32 value. When I check if ret.hit is true I set the option to Some((obj,ret.tval)), meaning the contents of my object pointer and the hit distance.

Now lets look at the second part of the loop, once ray intersection is done.

            let pixel = match objHitObj {
                Some((obj,tval)) => lut[shade_pixel(ray,obj,tval)],
                None             => " "
            };

            print!("{}",pixel);
        }
    }

Finally I can check and retrieve the option values using an if statement or a match. Match is like a switch/case statement in C, but with super powers. It forces you to account for all possible code paths. This ensures there are no mistakes during compilation. In the code above I match the some and none cases. In the Some case it pulls out the nested objects and gives them the names obj and tval, just like the tuple I stuffed into it earlier. This is called destructuring in Rust. If there is a value then it calls shadepixel and returns character in the look up table representing that grayscale value. If the None case happens then it returns a space. In either case we know the pixel variable will have a valid value after the match. It's impossible for pixel to be null, so I can safely print it.

The rest of my code is basically vector math. It looks almost identical to the same code in JavaScript, just strongly typed.

fn shade_pixel(ray:Ray, obj:Sphere, tval:f32) -> uint {
    let pi = ray.orig.plus(ray.dir.scale(tval));
    let color = diffuse_shading(pi, obj, LIGHT1);
    let col = (color.r + color.g + color.b) / 3.0;
    (col * 6.0) as uint
}

struct HitPoint {
    hit:bool,
    tval:f32,
}

fn intersect_sphere(ray:Ray, center:Vector, radius:f32) -> HitPoint {
    let l = center.minus(ray.orig);
    let tca = l.dot(ray.dir);
    if tca < 0.0 {
        return HitPoint { hit:false, tval:-1.0 };
    }
    let d2 = l.dot(l) - tca*tca;
    let r2 = radius*radius;
    if d2 > r2 {
        return HitPoint { hit: false, tval:-1.0 };
    }
    let thc = (r2-d2).sqrt();
    let t0 = tca-thc;
    //let t1 = tca+thc;
    if t0 > 10000.0 {
        return HitPoint { hit: false, tval: -1.0 };
    }
    return HitPoint { hit: true, tval: t0}

}

fn clamp(x:f32,a:f32,b:f32) -> f32{
    if x < a { return a;  }
    if x > b { return b; }
    return x;
}

fn diffuse_shading(pi:Vector, obj:Sphere, light:Light) -> Color{
    let n = obj.get_normal(pi);
    let lam1 = light.position.minus(pi).normalize().dot(n);
    let lam2 = clamp(lam1,0.0,1.0);
    light.color.scale(lam2*0.5).plus(obj.color.scale(0.3))
}

That's it. Here's the final result.

text

So far I'm really happy with Rust. It has some rough edges they are still working on, but I love the direction they are going. It really could be a replacement for C/C++ in lots of cases.

Buy or no buy? Buy! It's free!

After the more abstract talk I’d like to come back to something concrete. Regular Expressions, or regex, are powerful but often inscrutable. Today let’s see how we could make them easier to use through typography and visualization without diminishing that power.

Regular Expressions are essentially a mini language embedded inside of your regular language. I’ve often seen regex written like this,

new Regex(“^\\s+([a-Z]|[0-9])\\w+\\$$”) // regex for special vars

then sometimes reformatted like this

//regex for special vars
new Regex(
     “^” // start of line
     +“\\s+” // one or more whitespace
     +“([a-Z]|[0-9])” //one letter or number
     +”\w+” // one word
     +”\\$” // the literal dollar sign
     +”$” // end of line
)

The fact that the author had to manually split up the text and add comments simply screams for a better syntax. Something far more readable but still terse. It would seem that readability and conciseness would be mutually exclusive. Anything more readable would also be far longer, eliminating much of the value of regex, right?

Au contraire! We have a special power at our disposal: we can render whatever we want in the editor. The compiler doesn’t care as long as it receives the description in some canonical form. So rather than choosing readable vs terse we can cheat and do both.

Font Styling

Let’s start with the standard syntax but replacing delimiters and escape sequences with typographic choices. This regex looks for variable names in a made up language. Variables must start with a letter or number, followed by any word character, followed by a dollar sign. Here is the same regex using bold italics for all special chars:

text

Those problematic double escaped backslashes are gone. We can tell which dollar sign is a literal and which is a magic variable thanks to font styling.

Color and Brackets

Next we can turn the literals green and the braces gray, so it’s clear which part is actual words. We can also replace the $ and ^ with symbols that actually look like beginning and ending of lines. Open and close up brackets or floor brackets.

text

Now one more thing, let’s replace the w and s magic characters with something that looks even more different than plain text: letter boxes. These are actual unicode characters from the "Unicode Characters in the Enclosed Alphanumeric Supplement Block".

text

Now the regex is much easier to read but still compact. However, unless you are really familiar with regex syntax you may still be confused. You still have a bunch of specific symbols and notation to remember. How could we solve this?

Think Bigger

Let’s invent a second notation that is even easier to read. We already have a clue of how this should work. Good programmers write the regex vertically with comments as an adhoc secondary notation. Let’s make it official.

If we have two views then the editor could switch between them as desired. When the user clicks on the regex is will expand to something like this:

text

We get what looks like a tiny spreadsheet where we can edit any term directly, using code completion so we don’t have to remember the exact terms. Furthermore the IDE can show the extra documentation hints only when we are in this "detailed" mode.

Even with this though, there is still a problem with regex. Just by looking at it you can’t always tell what it will match and what it won’t. Without a long comment, how do programmers today know what a regex will match. They use their brains. Yes, they actually look at the text and imagine what it will match in their heads. They essentially simulate the computer the regex mentally. This is totally backwards! The whole point of a computer is to simulate things for us, not the other way around. Can’t we make the computer do it’s job?!

Simulate

Instead of simulating the regex mentally, let’s give it some strings and have the computer tell us what will match. Let’s see what that would look like:

regex tester

We can type in as many examples as we want. Having direct examples also lets us test out the edge cases, which is often where regexes fail.

Hmm. You know… this is basically some unit tests for the regex. If we just add another column like this:

regex unit tests

then we can see at a glance if the regex is working as expected. In this example the last test is failing so it is highlighted in red.

Most importantly, the unit tests are right next to the regex in the code; right where they are used. If we collapse the triangle then everything goes away. It’s still there, just hidden, until we need it.

This is the power of having a smart IDE and flexible syntax. Because these are only visualization changes it would work with any existing compiler. No new language required.

I’ve talked a lot about ways to improve the syntax and process of writing code. In Typographic Programming Language, Fonts, and Tabs Vs Spaces I've talked about the details of how to improve programming. However, I haven't really talked about my larger vision. Where am I actually going with this?

My real goal is to build the Ultimate IDE and Programming Language for solving problems cleanly and simply. Ambitious much?

Actually, my real goal is to create the computer from Star Trek (the one with Majel Barrett's voice).

Actually, my real goal is to create a cybernetically enhanced programmer.

Okay. Let’s back up a bit.

The Big Picture

When we are programming, what are we really doing? What is the essence? Once we figure that out we should be able to work backwards from there.

Programming is when you have a problem to solve and you tell the computer how to solve it for you. That’s really it. Everything we do comes back to that fundamental. Functions are ways of encapsulating solving stuff in small chunks so we can reason about it. The same thing with objects (often used to "model" what we are solving). Unit tests are to help verify what we are solving, though it often helps to define the problem as well. At the end of the day it's all to serve the goal of teaching the computer how to solve problems for us.

So how could we compare one PL against another?

Metrics

From here out I'm going to use Programming Language, PL, to mean the entire system of IDE, compiler, editor, build tools, etc. The entire system that lets you have the computer solve a problem for you.

How can we say that one PL is better than another? By what metric? Well, how about what lets you solve the program the fastest, or the easiest, or with the least headache. That sounds nice and is certainly true, but it's rather subjective. We need something more empirical.

Hmm. That's not very helpful. Let's back up again.

Paper and Pencil

What is a pad of paper and a pencil? It’s a thinking device. If you want to add up some numbers you can do it in your head, but after more than a few numbers it becomes tricky to manage them all. So we write them down. We outsource part of the process to some paper.

If we want to do division we can do it with the long division process. This is actually not the most efficient way to divide numbers, but it works well because you can have the paper manage all of the state for you.

What if you need to remember a list of things to do? You write it down on paper. The paper becomes an extension of your brain. It is a tool for thinking. This perhaps explains some people’s fetish over moleskin style sketch books (not that I would ever invest in a Kickstarter for sketchbooks.

A Tool for Thinking

If we think of a programming language as a tool for thinking, then it becomes clear. The PL helps you tell the computer what to do. So a good PL would help you around the parts of programming that are hard: namely keeping state in your head. A bad PL requires you to remember a lot of things.

For example, suppose you write a function that calls another function named 'foo'. You must remember what this other function does and what it accepts and what it returns. If the function is named well, say 'increment', then the function name itself helps your brain.You have less to remember because the function name carries information.

Now suppose we have type information in our PL. The function increment only takes numbers. Now I don’t have to remember that it only takes numbers; the compiler will enforce it for me. Of course I only find this out when I compile the code. To make the PL give me less to remember the IDE can compile constantly, giving me an immediate warning when I do something bad. Code completion can also help by only suggesting local values that are numbers.

With these types of features the PL acts as a tool for thinking. An extension of the brain by holding state.

So we can say a PL is “better” if it reduces the cognitive load of the programmer. Things like deterministic performance and runtime speed are nice, but they are (or at least should be) secondary to reducing the cognitive load of the programmer. Of course a program which is too slow to be used is a failure, so ultimately it does matter. However computers get faster. The runtime performance is less of a concern than the quality of the code. Perhaps this explains the resurgence in functional programming styles. The runtime hit matters less than it did 30 years ago, so we can afford to waste cycles on things which reduce the cognitive load of the programmer.

Regex

Now, where were we? Oh, right; the Fire Swamp.

Let's look at a more concrete example. Regular expressions. Regexes are powerful. They are concise. But a given regex is not easy to read if you didn’t write it yourself; or even if you did but you didn’t just write it. Look at your regex from 6 months ago sometime. I hope you added good documentation.

A regex conveys a lot of information. You have a lot to load up into your brain. When you look at a regex to see what it does you have to start simulating it in your brain. Your brain basically becomes a crappy computer that executes the regex on hypothetical strings.

That's crazy. Why are we simulating a state machine in our heads? That’s what we have computers for! To simulate things. We should have the PL show the regex to us in an easier to understand form, or perhaps even multiple forms. With unit tests. And visualizers. Or an embedded regex simulator to show how it works. I have a lot to say about regular expressions in an upcoming blog, but for now I'll just say they are some extremely low hanging fruit in the mission to reduce cognitive load.

Expanding Complexity

Now you might think, if we make a PL which sufficiently reduces cognitive load then programmers will have little to do. Programming will become so easy that anyone could do it. True, this might happen to some degree. In fact, I would argue that letting novices program is actually a good thing (though we would call it problem description rather than programming). However, in general this won’t happen. As we decrease the cognitive load we increase the complexity of the programs we can make.

There seems to be a limit to how much complexity the human brain can handle. This limit varies from person to person of course. And it is affected by your health, hunger level, stress, tiredness, etc. (Never code on an empty stomach.) But there is a limit.

Historically better tools have reduced the complexity at hand to something below the cognitive limit of the programmer. So what did we do? We tackled more complex tasks.

Josh’s first postulate of complexity: Just as data always expands to fill available disk space, programing tasks always increase in complexity to fit our brain budget.

Offloading complexity to tools merely allows us to tackle bigger problems. It’s really no different then the brain boost we got from inventing paper and pencil, just separated by a few thousand years. [1]

Now I think we can agree that a PL which reduces more cognitive load of the human is better than one which reduces less. That's our metric. So how can we turn this into something actionable? Does it suggest possible improvements to real world programming languages?

The answer is yes! If we stop focusing on PL implementations but rather the user experience of the programmer, then many ideas become readily available. Everything I’ve discussed in previous blogs is driven by this core principle. Showing a color or image inline reduces cognitive load because you don’t have to visualize in your head what the color or image actually looks like. The editor can just do it for you. This is just the beginning.

My Forever Project

These are ideas I’ve been working on a long time. In fact I recently realized some of these ideas have been in my head for twenty years. I found evidence of tools I (attempted) to write from my college papers. It’s only recently that everything has started to gel into an expressible form. It’s also only recently that we’ve had the "problem" of computers with too much computational power going to waste. This is my Forever Project.

There’s so much possibility now. If a notepad and a text editor are our current cybernetic enhancements, what else could we build? Could Google Glass help you with programming? How about an Oculus Rift? Or using an iPad as an extra screen? The answer is Yes! We could definitely use these to reduce cognitive load while interacting with a computer. We just might not call all of these tasks "programming" but they are (or will be shortly as software continues to eat the world).

deep breath

My concept summarized: Programming systems should not be thought of as ways to make a computer do tricky things. They are ways to make a computer solve problems for you. Problems you would have to do in your head without them (assuming you could do them at all). Thus PLs are tools for thinking. This gives us a metric. Do particular PLs help us think better and solve problems better? As it turns out, most PLs fail miserably by this metric. Or at least they could be a whole lot better. Many ideas from the 60s and 70s still remain unimplemented and unused.

But don’t be sad. Our field is less than 100 years old. We are actually doing pretty well. It took a few thousand years to invent suspension bridges and they still don’t work 100% of the time.

So from now on let us consider how to make better tools for thinking. Everything Bret Victor has been doing, and Jonathan Edwards, and even the work from the 70s & 80s that Alan Kay did with Smalltalk and the Dynabook come back to the same thing: building better tools for thinking. With better tools we can think better thoughts. With better tools we can solve bigger problems.

So let’s get crackin'!

footnote: [1] Before someone calls me on this, I have no idea when paper and pencil were first invented for the purposes of reducing problem complexity. Probably in ancient Egypt for royal bookkeepers. I'll leave that as an exercise to the reader.