Josh On Design

I am giving a presentation on the future of desktop interface at OSCON in a few weeks. To help prepare for the session I'd like to use you, gentle readers, as my guinea pigs. The following essay is an extremely rough version of what I will be presenting. Please imagine it with humorous illustrations and no grammatical errors. I will greatly appreciate your feedback. What parts should I expand? Where are my arguments unclear or flawed? Would you come to see this talk?

Welcome to The Future of Desktop Interfaces. I'm afraid your presence is actually a ruse on my part. I am *not* going to tell you what the future of desktop interfaces will be. Rather my goal is to trick you into inventing the future for me. So really you are here because I am lazy. Remember, necessity is the mother of invention. Laziness is the father.

Introduction

It is my belief that in a decade 90% of people will use a tablet, smartphone or other non-desktop device as their primary computing interface. I actually made this prediction two years ago so have only eight years left, and the success of the iPad implies that we may actually be ahead of schedule. At this point I think my prediction is relatively uncontroversial. Tablets and smartphones have a managed or 'curated' computing experience. These devices actually meet the computing needs of most people better than a traditional desktop PC. Note: for the purposes of this discussion I mean both laptops and desktop computers running MacOSX, Windows, or a Linux Desktop. anything with an exposed filesystem. These managed computing devices will continue to get better and more powerful, mostly replacing the PC.

So, that's great for the 90%. They will be able to get their jobs done with no fuss. But what about the 10%? What about the people who create content professionally? What about the programmers? The digital artists? The data analysts? The hackers? Are PCs going to atrophy as all attention moves to the 90%? I don't think so. In fact, I think this could be a renaissance of desktop computing. We've been kind of stuck in a rut for the past decade. Desktop interfaces haven't changed much since about 2000. Certainly they are prettier, but they haven't' really gotten better or more productive. I think we could have some really interesting things coming. But first, a diversion.

This is a cave painting [image]
This is a painting from the wall of pompeii, circa 70 ad [image]
This is a painting from the middle ages [image]
This is a painting from the high renaissance [image]
This is a painting from the 1850s [image]

We can see a trend of greater and greater realism built over the centuries leading up to height of the great renaissance painters like Michaelangelo. Then a plateau. Once we had achieved realism where do we go? More portraiture. More landscapes. Painting started to get boring. Then something happened in the 1870s. All of a sudden we get impressionism, cubism, surrealism, and abstract modernism. we see more change in a 70 year period than the previous 700. What happened?

[pictures of Van Gogh and Picasso paintings]

Photography was invented. Until the photograph the commercial purpose of painting was to capture and recreate reality. Most paintings were portraits of rich people. But no painting could compare with the photograph at recreating reality, especially not for the price. The photograph made realistic portrait painting obsolete. This was both a blessing and a curse. Painters needed to dream up with something new to paint. And dream they did. Freed from the need to duplicate reality there was an explosion of new ideas and trends. Painting had new life, which lead to amazing things.

I hope the same thing will happen to desktop computers. PCs have been the primary computing interface for humans for the past thirty years. By definition the they have to serve the needs of all people. But if 90 % of people will use something else, then maybe desktop interface can evolve again. Maybe they can change to meet the needs of the 10% far better.

Trends

I can't reliably predict anything a decade out in our industry. Things change too quickly and there are too many unknowns. However, there are a few trends that I think we can take a look at. The next few years will be shaped by the following forces:

* Moore's law. We have a glut of CPU, GPU, and storage resources on our personal computers. A glut that we don't' really take advantage of yet, and this glut shows no sign of stopping.

* Mature software development tools and toolkits. One engineer can create an app in a few weeks that would have taken a team of programmers months just a decade or two ago. This is partly thanks to our tools like modern IDEs, version control systems, automated build systems, etc. We also have robust toolkits and APIs that let us code at a higher level. Combined with an excess of RAM we can build complex desktop software far faster.

* App stores: while I don't like the curated part of desktop app stores there is no denying that they open the market for smaller software developers. A small team can make an app that will appeal to a very niche market and still sell it profitably because they have access to the entire world. This makes narrower products far more feasible than was possible 10 years ago.

* Ubiquitous networking

* Info glut:

The 10 Percent

Before we talk about what the interface will look like, let's talk about the problems that the 10% have. When I first suggested this topic some people thought I meant that interfaces would become hard to use again. That usability is only for the 90%. Not true. the 10% needs quality software as well, but we need it to be deeper. We have specific needs that must be addressed and are willing to spend the time to learn more powerful interfaces. In particular, we have a torrent of information to be managed.

Every time I get a new computer the hard drive doubles and I always fill it up. Only half of this is photos and videos. I also have endless documentation sets, PDFs, gigabytes of emails, backups of my mobile devices, and word docs stretching back to the early nineties. I personally have more information to manage than most companies did thirty years ago. I need a way to manage it.

Along with this information I have lots of devices that I need to manage as well. iCloud is nice but it doesn't scale very well. I have an iPad, two iPhones (for me and my wife), an iPod touch for my son, and several test phones (Android, webOS, Windows Phone, and Meego), and a few cameras. And that's just the mobile devices. I now have a home media server and a Roku box for the TV. Soon I will have home automation components for my thermostat, to control the lights, water the garden, run the sprinklers, handle security, and watch the baby. Thanks to Moore's Law our homes will soon be awash in computing. That's a lot to manage. I can't do it all from an iPad.

So where do we start to address these needs? First: hide the filesystem.

information management

Hide the filesystem. Use robust shared data stores underneath. (Man, I really wished BeOS had survived). I'm not saying we need a database to replace the filesystem but at least hide it. I don't care how my stuff is stored as long as it works, and works quickly. The example most people are familiar with is iTunes. MP3 libraries were the first widespread case where we hit the complexity wall. Most people simply have too many songs to manage effectively using directories. Instead we need a database that can search and filter by multiple criteria and be very responsive. iTunes does this for music.

[picture of iTunes]

I'd like to see this interface style applied to more kinds of files. Here is an app for academic researchers to manage the many scholarly papers they have to read.

[picture of Papers]

[picture of Sparrow]

Sparrow is a great email client for Mac that is much faster than the system default. It sync with social networks to get avatars. It has intelligent threading. It's good enough that I spent 20$ to replace a free system app. But it's just email. It doesn't handle the rest of my communication. And while beautiful it's not very customizable. Customization matters as much as beauty.

I'd like to have a single place to store my external communication: IM logs, SMS logs, all tweets, all emails, all FB posts. One app. One interface. When I'm composing an email to someone I'd like to see all of the messages I've ever sent to them. And of course this should magically sync with all of my online services to keep up to date.

Idea I want a single place to see all of my code across all projects, be aware of all version control systems. My IDE only manages the projects I'm currently editing. I want something to manage *all* of my projects, both personal and professional, including my build services and the bug tracking systems I'm currently involved with. Most of these systems have APIs so I don't it would be difficult to build.

Apple is moving in the direction of system wide saved queries, similar to what iTunes offers, but they are doing it very cautiously. There is so much more we can do.

customization and automation

Just as great painters would make their own paints and canvases, we need the ability to create our own workflows and customize our tools.

* apps communicate together

* build new apps out of pieces of other apps

* customize the general computing environment, but still

* be able to handle anything

the line between a custom workflow and real programming is fuzzy

Let me give you some examples:

iTunes smart lists and IDE keymaps

We need to take this to the next level. I should have a keymap that works system wide in all apps. The Mac has smart lists that can be used in any app, but only for photos and albums. This should be available for any kind of media list.

I should be able to change the interface of any app. In a drawing program I can change the toolbar, but why doesn't *every* action in an app have a toolbar item? Why can't I create different toolbar sets for different kinds of projects? Then these sets should be shared across apps.

Automation

If every action has a toolbar item then we could take this to the next level and actually script our apps together visually. Mac OS X does this with automator but they never took it as far as it could go. Now it appears that Automator will be crippled by the new Mac App Store restrictions. This is a shame. I think Automator never really took off because not enough apps supported it and there was no easy way to share scripts between people. My ideal vision would be something like: visually create a script to take the current document of photoshop, convert to PNG, post to Flickr, send out a Facebook update and Tweet to the Flickr link *tomorrow morning at 9am EST*. This should not be hard to do in the 21st century, and yet our tools don't make this easy to do. This is really a problem with app communication. I'm not entirely sure how to solve it, but the potential is huge.

If we start to think of a desktop app as a collection of modules rather an a monolithic whole, then we can start doing these sorts of cool things.

With such a modular system the line between customization, automation, and real programming becomes very fuzzy. If we make it easy to create and share these scripts then we can make the desktop twice as powerful.

Integrate the web

Once we make our apps modular and scriptable they can start talking to the web. There are some simple things we can do today. Why don't more desktop apps have 'share via twitter' buttons? Why doesn't photoshop have it? I've been working on an open source drawing program called Leonardo Sketch. One of the first features I added was the ability to share a snapshot of what I'm working on right now through Flickr, Twitter, and Facebook.

I'd like to take this to the next level. I recently added an asset manager to Leo Sketch. It's an iTunes like view that shows you all of the clip art, fonts, palettes, and textures that I have to work with. But why should my asset collection be limited to what's on my computer? I've added a flickr search which will find creative commons licensed images based on keywords. next I want to integrate Google Fonts. Then I can access a huge collection of fonts from anywhere in the world, easily searchable.

How else could we ingrate the web into desktop apps? How about selecting text and having Bing translate it into another language, right from within my drawing. Or let me see the most popular color swatches that are tagged with 'summer'.

How about an IDE that will let me search for small open source code libraries. Say I'm working on an app that needs to parse a CSV file. Instead of jumping to a web browser I'd like to be able to search git hub for code snippets that match my current working language. It shouldn't be any more complicated than code completion. It would show me 8 snippets which take a file and return a string, along with their ratings. When I chose the one I want it downloads the code, compiles it, and puts it in my class path.

Context awareness Computers must learn from us and do things for us.

The other half of integration and automation is the computer doing things for us without having to request it. This trend is already happening from two very different directions. On one end our phones are learning more about us, which enables services like Siri. On the other end our IDEs are getting smarter and smarter. IntellJ IDEA not only knows what possible methods can fit the current spot in my code, but it will automatically add the imports for me. Because the editor has so much contextual knowledge about what I'm doing right now (I'm editing a particular place in a particular file in a particular project with a particular class path), it can do lots of things for me, or at least monitor what I am doing and give me advice. We should have all our apps doing this.

My word processor should let me type in an equation and offer to evaluate it for me. I should be able to type in a reference to a stock symbol and have it turn into a link with the current stock value. If I'm listening to music my calendar should tell me about an appointment 15 minutes before hand with a silent unobtrusive message, then increase to an alarm and turning off the music as I get closer to the appointment time. There should be a system wide switch to turn off all messages and alerts when I want to be in concentration mode.

My laptop knows where it is based on my phone's GPS and the local wifi access point. It should be able to adjust itself based on the location just as my phone does. The possibilities are huge here, but it worries me that almost all of the innovation is happening on the phone side, not the desktop side.

Identity Management and Service Sharing

Finally identity management. Originally we thought identity was just a handle. Then we thought it was your real name representing a single unified you. You have one identity on the web, period. But that was wrong too. Now we understand that identity is far more complicated. When I do something on a Google service am I acting as Josh, the author and open source coder, or am I acting as Joshua, the Nokia researcher? The answer is: it depends. I might be either of those, or some mixture.

We also have identity scattered across many places with no way to integrate them while protecting my privacy. This is a huge mess that is going to keep getting worse. I'd like to see an OS wide identity system. Any app, including the browser, which wishes to integrate with the network can ask the identity system for credentials. This allows any app to talk to anything, and at any time I can change which identity I'm using. This could be done per app or even for the desktop as a whole. I can imagine doing this with a virtual desktop system where one desktop is 'Josh the author' with a green background and menu bar. The other desktop is 'Josh the engineer' with a blue background and desktop. Each has credentials that can't be shared between them without explicit actions by me, the human.

We can see some of this in password managers and browsers that sync their settings, but this really needs to be system wide.

A plea to Desktop Linux

There will never be a year that Linux conquers the desktop. That era is over. The world is going mobile. Instead of focusing on a complete desktop OS, create an awesome desktop environment that can be seamlessly layered on top of any proprietary OS. Over time people will use more and more of your stuff, until they can switch entirely. Especially if you make the switch easy by having cloud based backup of everything (hello version control!).

My favorite app for Mac now is a command line tool called Brew. It uses community developed 'recipes' which download and compile lots of linux programs and libraries. With Brew I can be on a Mac but have ridiculously easy access to the rich ecosystem of linux tools like apache, imagemacgick, sdl, and node.

What I want is a graphical Brew app. I install it once and log in. I then downloads and configures my favorite email and browser apps, all of the command line programming tools I need, and anything else provided by the open source community. Even my settings are synced from a cloud server. *That* is how linux can win the desktop. You will never win over the masses. Focus on the hyper users and let them ease into a fully free desktop.

Conclusion:

I hope that you have gotten two things out of this essay:

First, most computing is going to managed devices but the desktop interface can have a rich future ahead of it, if we are willing to build it.

Second, I hope you come away with some ideas for how we can move the desktop forward. We are going to live in the future we make so we'd better make a good one. I understand it's going to be difficult, but this really needs to happen. We must accept that people are reluctant to change, but do it anyway. If you are right then we will win in the end. (or at least our ideas will).