Cookies are designed for communication between the browser and a server that persists between sessions. They’re typically used for identifying a user on return visits and storing details about that user. Cookies are sent between the browser and server in plain text, unencrypted, each time the user opens a page. So, unless an application encrypts cookie contents, these can be quite trivially be read particularly on public wifi networks, when used over standard HTTP (though less easily over encrypted HTTPS).

Storing all client data on the server creates usability issues as well, as users need to log in each time they use that site. And of course the heavy lifting of ensuring data is secure during transmission, and on the server is left to you as the developer. And it’s rather tricky to build apps which work when the user is offline if the user’s data is all stored on the server.

As web applications become increasingly sophisticated, developers need ways to keep data around in the browser (particularly if we want our applications to work when the user is offline).

Two closely related but slightly different W3C technologies exist to help keep track of information solely in the browser. They enable far more structured data than cookies, are much easier for us to use as developers, and the information stored can only be transmitted to a server explicitly by the application.

sessionStorage stores data during a session and is removed once a session is finished. localStorage is almost identical, but the data stored persists indefinitely, until removed by the application. Let’s start with sessionStorage, keeping in mind that we use localStorage almost identically.

sessionStorage

What is a session?

The key feature of sessionStorage is that data only persists for a session. But just what is a session? HTML5 has the concept of a “top-level browsing context”. This is, in essence, a window or tab. A session lasts for that top-level browsing context while it is open, and while that top-level browsing context is pointed at the same domain (or strictly speaking, the same origin).

During the session, a user could visit other pages of the domain, or other sites entirely, then return to the original domain. Any data saved in sessionStorage during that session will remain available, but only to pages in the original domain, until the tab or window is closed.

If the user opens a link to your site in another tab or window, then there is no access to this sessionStorage, since this new tab or window is a new session.
It’s worth noting that sessionStorage is also shared with pages inside subframes in the same domain as the top level document in the window.

So, if we

• visit http://webdirections.org in a tab and save data to sessionStorage
• then follow a link to http://westciv.com in this same tab
• and then return to http://webdirections.org in the same tab
• we return to the same session for http://webdirections.org
• the data in the original sessionStorage is still available

If however we

• visit http://webdirections.org in a tab and save data to sessionStorage
• then follow a link to http://webdirections.org in a new tab
• the data in the original sessionStorage is not available to this new tab (but is in the original tab)

The one exception to this is when a browser crashes, and is restarted. Typically, browsers will in this case reopen all the windows that were open when the browser crashed. The specification allows in this situation for sessionStorage to persist for reopened windows from before the crash (WebKit, Mozilla and Opera browsers support this, IE8 does not, though IE9 and up do).

Which may sound like a great boon for the user, but, as an application developer, you may wish to consider whether you in fact want to persist session data after a crash. A user may consider that when their browser crashes using a service like web mail or online banking at an internet café or other shared computer that their login details have been purged, but if these were stored in sessionStorage then the next user to launch the browser will resume the session that were current when the user crashed. Ruh-roh.

What good is sessionStorage?

So, what good is sessionStorage? Well, one very useful application would be to maintain sensitive information during a transaction, signup, sign in and so on, which will be purged as soon as the user closes a window or tab. It can be used to create a multi-page form or application, where the information in each page can persist, and then be sent to the server all at once when the transaction is complete. It also moves some of the heavy lifting for protecting sensitive data away from application developers to the browser developer.

Using sessionStorage

sessionStorage is a property of the window object in the DOM. Because it is as yet not universally supported, we’ll want to check that this property exists before we use it:

if (window.sessionStorage) {
  //we use sessionStorage
}

else {
  //we do something else, perhaps use cookies, or another fallback
}

Right, so now we have our sessionStorage object, how do we use it?

Key-Value Pairs

sessionStorage stores “key-value pairs”. Each pair is a piece of information (the value), identified by a unique identifier (the key). Both the key and the value are strings (more on the implications of this in a moment).

We use the setItem method of the sessionStorage object to store data like so:

//get the value of the input with id="name"
var name = document.querySelector('#name').value;

//store this value with the key "name"
window.sessionStorage.setItem('name', name);

Now we’ve stored the value of the input “name” in an item of the sessionStorage object called ‘name’. It will remain there until this window or tab is closed and it will then automatically be purged by the browser when the user closes the window or tab.

reading from sessionStorage

There’s not much point in storing these details if we can’t get them back at some point. We do this by using the function getItem of the sessionStorage object, using a single parameter, the key we used to set the item.

So, to get the value of the name, we’d use:

var name = window.sessionStorage.getItem('name');

Non-existent items

Now, what happens if for some reason there’s no item in sessionStorage with the key we are trying to access? In place of a string value, it returns null, not the empty string. So, it’s worthwhile testing whether the result returned is not null before using it

var email = window.sessionStorage.getItem('email');
if(email!==null){
  document.querySelector('#email').innerHTML = email;
}

Saving Data Between Sessions

When information is less sensitive, it may make sense to store it between sessions. Particularly as web sites become more application-like, and can increasingly work offline, saving preferences or the state of a document can make for much better usability.

My HTML5 and CSS developer tools do just this, using localStorage. That way, when someone returns to the tools, the last gradient or layout or transformation they built is waiting for them.

Best of all, using localStorage, for persistence between sessions, is almost identical to using sessionStorage.

window.localStorage

Instead of the sessionStorage object of the window, we use the localStorage object. All the methods of localStorage are the same as sessionStorage.

• we set items with setItem
• we get items with getItem

Because items on the localStorage will persist forever, we may want to delete them. We can do this with localStorage.removeItem(key), using the key for the item we want to remove.

If we want to delete the entire localStorage, we can use localStorage.clear(). But, be warned, anything your app has saved to localStorage for this user is gone for good.

sessionStorage in fact also has these methods, though we’re less likely to need them, as all of a sessionStorage is discarded by the browser when a session finishes.

Gotchas, Tips and Tricks

sessionStorage and localStorage store all data as strings

As mentioned earlier, the values stored in local and session storage are strings, which has a number of implications for developers.

In particular, when we store boolean values, integers, floating point numbers, dates, objects and other non-string values, we need to convert to and from a string when writing to and reading from storage.

There’s also a more subtle side effect of storing values as strings. JavaScript strings are UTF-16 encoded, which means each character is 2 bytes (in UTF-8 characters are one byte). This effectively halves the available storage space.

Private Browsing

Many browsers now have private (or ‘incognito’) browsing modes, where no history or other details are stored between sessions. In this situation, what happens with sessionStorage and localStorage varies widely by browser.

• Safari returns null for any item set using localStorage.setItem either before or during the private browsing session. In essence, neither sessionStorage nor localStorage are available in private browsing mode.
• Chrome and Opera return items set previous to private (“incognito”) browsing commencing, but once private browsing commences, treat localStorage like sessionStorage (only items set on the localStorage by that session will be returned) but like localStorage for other private windows and tabs
• Firefox, like Chrome will not retrieve items set on localStorage prior to a private session starting, but in private browsing treats localStorage like sessionStorage for non-private windows and tabs, but like localStorage for other private windows and tabs

Getters and Setters

In addition to using getItem and setItem we can use a key directly to get and set an item in sessionStorage and localStorage, like so (where the key is “keyName”):

var itemValue = window.localStorage.keyName;

localStorage and sessionStorage Limits

The webStorage specification recommends browsers implement a 5MB limit on the amount of data localStorage or sessionStorage can save for a given domain. If you try to exceed the limit that various browsers have in place (for some browsers users can change this allowance) setItem throws an error. There’s no way of asking localStorage for the amount of space remaining, so it’s best to set item values with a try and catch for any error:

try {
	window.localStorage.setItem(key, value);
}
catch (exception) {
	//test if this is a QUOTA_EXCEEDED_ERR	

}

If the available space for this localStorage is exceeded, the exception object will have the name "QUOTA_EXCEEDED_ERR" and a code of 22.

As mentioned, in JavaScript strings are UTF-16 encoded, which means they are 2-byte. So, when saving the string “John”, we are actually using 8 bytes, not 4. Which means instead of 5MB of storage space per storage area, we effectively have 2.5MB.

If the storage needs of your application are likely to exceed 5MB, then web databases are likely to be a better solution. However, the situation with web databases is complicated, with two different standards, one, webSQL widely supported but deprecated, and the other IndexedDB, currently supported only in Firefox, Chrome and IE10.

Storage Events

We can add an event listener to the window for storage events so that when a storage object has been changed (there’s a reason for the emphasis) then we can be notified and respond to those changes.

window.addEventListener('storage', storageChanged, false);

Here, when localStorage is changed (by setting a new item, deleting an item or changing an existing item) our function storageChanged(event) will be called. The event passed as a parameter to this function has a property storageArea, which is the window’s localStorage object.

There are two things to be aware of with storage events.

• The event only fires if the storage is changed (not if it is simply accessed and not if we set an item to the same value that it currently has!)
• In the specification, the event is not received in the window or tab where the change occurred, only in other open windows and tabs that have access to this localStorage. Some browsers have implemented storage events in such a way that the event is also received by the window or tab which causes the change, but don’t rely on this.

webStorage Performance

Of late, a number of high profile, widely read articles critical of localStorage have been published, centring on its asserted performance shortcomings. The key criticism relates to the fact that webStorage is synchronous. This means a script using sessionStorage or localStorage waits while getItem, setItem and other storage methods are invoked. In theory, this can impact both the browser’s response to user input and execution of JavaScript in a page. In practice, I’d argue that this is not likely to be a significant problem for most cases.

I recently conducted some testing across a number of devices and browsers which demonstrates that even for poorly implemented code that does a very significant number of getItem and setItem operations, the performance of webStorage is unlikely to have significant impact.

Origin restrictions

We said earlier that that sessionStorage and localStorage are restricted to windows or tabs in the same domain, but in fact, the restriction is tighter than simply the top-level domain (such as webdirections.org)

To have access to each other’s webStorage, tabs or windows must have the same top-level domain (for example webdirections.org), subdomains (for example test.webdirections.org), and protocol (https://webdirections.org has a different localStorage from http://webdirections.org).

At first glance this might seem overly restrictive but imagine john.geocities.com having access to the localStorage of fred.geocities.org?

Browser Support and Backwards Compatibility

localStorage and sessionStorage are widely supported, from IE8 upwards and in all modern browsers, including mobile devices.

There are also several polyfills that allow for webStorage in browsers which don’t support it natively.

The Wrap

webStorage solves a long standing challenge for web developers — reliably and more securely storing data between sessions entirely on the client side. While there are assertions that performance limitations make localStorage “harmful”, in the real world, services like Google and Bing are using localStorage and performance experts like Steve Souders and Nicholas Zakas defend and advocate their use. That’s not to say webStorage is perfect or ideal in all situations. The synchronous nature of the API and 5MB limit per origin do mean that in certain circumstances an alternative may be required. webStorage is however eminently usable for a great many client side data storage needs.

Further Reading

• The ever reliable HTML5 Doctors on webStorage
• Opera Developer Network, another always useful place to visit, on webStorage
• All about local storage, from Mark Pilgrim’s excellent “Dive into HTML5″
• Remy Sharp on the state of support for offline events, and some workarounds
• A couple of articles from the Mozilla Hacks blog on web databases, the Road to IndexedDB and an introduction to IndexedDB
• MSDN’s introduction to IndexedDB (supported in IE10)
• An introduction to webStorage, also from MSDN

Every year we scour the globe looking for people who are saying, and more importantly doing, really interesting work, the kind of work that is making the web a better and more fascinating place for all of us.

But we also cast our net locally: we want to get you up on that stage as well!

So, do you reckon you’ve got something that might be of interest to the Web Directions audience?

Our program

To give you some idea of how the program works, the conference has 3 tracks:

- Design: user experience, design strategy, design process, design case studies, CSS3, interaction design, interface design, content strategy, usability, ethnography.

- Development: JavaScript and HTML5, though feel free to pique our curiosity with ideas that fall outside these two very broad headings :)

- Big Picture: interesting or unique perspectives at the edges of the web. What are the possibilities that are open to us as people who work on the web? What ideas are just over the horizon? Ideas that get the audience excited about working in this awesome field at this incredible time.

Don’t feel too troubled with trying to slot your idea into one of these if that doesn’t come easily to you: this is a challenge we are used to!

Most of our sessions have traditionally been 45–50 minutes long. This year we are also exploring the idea of having shorter sessions — 15ish minutes — in the dev track. This is the perfect amount of time to show off a neat JS library, or demo some other problem solving wizardry you have mastered. So if you’ve got an idea that you don’t think is quite up for a 50 minute session, but would work in this shorter format, please do send it in!

Where do I apply?

Do you think you’ve got what it takes? We definitely want to hear from you some time in the next couple of days then. It doesn’t have to be a fully formed session proposal or anything like that, so don’t labour it, give us as much detail as you can, and just get it in! To keep all these ideas in one place, please help me out by entering it via this form.

Very much looking forward to hearing from you.

Christian’s and Tara’s positions are very similar, so I’ll address them via Christian’s post. The essence of Christian’s criticism is

we have to stop advocating localStorage as a great opportunity for storing data as it performs badly

Strong words indeed. Is there evidence to back this up?

Christian then outlines these concerns in more detail

localStorage is synchronous in nature, meaning when it loads it can block the main document from rendering

Now, this is certainly true, and in theory, we could block the main UI in most browsers (Opera threads the main browser UI separately from scripts), as well as blocking the remainder of a script which uses localStorage while it waits on a read or write. But, in practice, is this a big deal? I’m a bit of a fan of evidence and testing, rather than theory alone, so I set out to test this. You can follow along at home if you like

Reading, writing and arithmetic

Here’s what I did. The test page allows you to specify what you want to save (as everything is saved to localStorage as a string, you just input a string value), and how many times you want to save it. Then, I just save the string that number of times to localStorage in a tight loop, measuring how long it takes, and report that back. Keep in mind this is very quick and dirty, and can fall over without any warning if you over fill your localStorage.

The test is really simple, but we should get a sense of how big a performance hit localStorage is. Here are results for writing 18bytes 10,000 times inside a tight loop (results are meant to be indicative), then reading them back out again

At worst, we have a 2s wait (in, of all browsers Chrome 19, a developer preview). In many browsers the wait is less than a tenth of a second. Remember this is for reading or writing 10,000 times to localStorage.

What about for larger amounts of data? Here’s the results for reading and writing 128 bytes

So, in essence, we’re really not seeing the performance impacted even when we increase the amount of data stored by a factor of 6 or so.

Given that writing 10,000 times to localStorage in a tight loop is I’d argue an edge case (and probably not advisable), on the face of it, the assertion that localStorage, being synchronous, and for that very reason has performance issues is largely misplaced in fact.

Christian continues

localStorage does file I/O meaning it writes to your hard drive, which can take long depending on what your system does (indexing, virus scanning…)

On a developer machine these issues can look deceptively minor as the operating system cached these requests – for an end user on the web they could mean a few seconds of waiting during which the web site stalls depending on what your system does (indexing, virus scanning…)

emphasis mine

Again in theory this looks worrisome, but I think in practice browser developers will use some sort of in memory caching and then asynchronously write to disk (actually based on my tests below I suspect that’s what they are doing)

In order to appear snappy, web browsers load the data into memory on the first request – which could mean a lot of memory use if lots of tabs do it

They may do, but this is an implementation detail for browser developers, and regardless of whatever mechanism they implement for persistent client side data (IndexedDB, webSQL…) in order to appear snappy, some sort of memory caching will be required.

localStorage is persistent. If you don’t use a service or never visit a web site again, the data is still loaded when you start the browser

Again this is an implementation detail for browser developers, and I doubt browsers have to load the localStorage DBs for all pages ever stored when they launch. I can imagine all kinds of caching strategies to balance responsiveness with memory use, application developers (speaking as one for many years) do this sort of thing all the time.

But above all, in practice, are we actually seeing any of this taking place in the wild? Can Christian point to sites that are suffering because of these theoretical limitations in localStorage?

Christian then makes something of a curious leap when he argues

In essence this means that a lot of articles saying you can use localStorage for better performance are just wrong

So, when the likes of Steve Souders, and teams of very smart people at places like Google and Bing research and develop techniques using localStorage for client side resource caching, they are “just wrong” because in theory localStorage has performance issues. To be direct, I find this position baffling. Especially when not a single datapoint is quoted (the tests cases I developed here took in the order of an hour, so it’s not overly taxing to do).

None of this is to say localStorage is perfect. Its limitations as to storage size, the fact that everything stored is simply a string, and that JavaScript strings are UTF-16, thereby halving the effective storage space are all real issues. Is the synchronous nature of localStorage an issue? Synchronous APIs are easier for developers to use (no need for callback functions), and so the idea of a simple client side storage mechanism, webStorage, coupled with a more sophisticated, high performance database standard (currently mired in standardization issues, far from localStorage’s fault) makes excellent architectural sense IMO.

But frankly, Christian and Tara do developers no benefit asserting localStorage is “harmful”, or that “there is no simple solution for local storage”. localStorage is very widely supported (IE8 up), very straightforward to use, has excellent performance for most use cases, and has been implemented in a large number of real world projects, including by Bing and Google. Sure, if you’re writing an app that does 10K simultaneous writes to a database, you’ll have the chops to get around a 2s write time in the worst case, but you will doubtless have all kinds of other performance and memory use challenges. So by all means help developers understand the potential shortcomings of a given technology, outline the gotchas, and workarounds, and propose improvements. But do so based on evidence, and also with the acknowledgement that localStorage is here to stay, and that any alternative also faces many similar, as well as their own additional challenges, and in the case of client side storage is perhaps years away from the same level of support as webStorage. The perfect, as is often stated, is the enemy of the good, or adequate. Let’s by all means make the adequate good, and the good excellent, but let’s not wait around forever for the perfect. We’ll be waiting a while.

Edited to correct Taras’ name

Many hours later I had come up with the list below.

I invite you to sit back, take a little stroll down memory lane, and remember all the important things we’ve learned from these amazing women.

Rachel Andrew spoke about Core CSS3.

Relly Annett-Baker told us about content strategy for web apps as well as microcopy.

Angela Beesley — spoke on wikis and community collaboration.

Jina Bolton told us how to create sexy style sheets.

Paula Bray on connected digital initiatives and strategy.

Wendy Chisholm told us all about universal access for apps.

Esther Derby showed what happens when agile meets UI.

Hannah Donovan told stories through design, designed without the browser and then reprised designing without the browser.

Natalie Downe told us the Lanyrd story.

Kimberly Elam showed us essential composition tools for web typography.

Ruth Ellison showed us how to integrate accessibility into design.

Anne Galloway took us to a 21st century bestiary.

Cheryl Gledhill showed us how to move our organisations to web standards and also took us beyond SEO.

Kelly Goto taught us about the iterative app, and how to design for lifestyle, and then showed us workFLOW.

Lisa Herrod showed us that usability is more than skin deep, that usability is for the rest of us, and then how to recategorise WCAG 2 using a role-based approach.

Rachel Hinman talked us through some mobile prototyping essentials.

Molly Holzschlag showed us some crimes against web standards.

Tara Hunt — told us all about Government 2.0.

Suze Ingram asked us if we would like service design with that?

Leslie Jensen-Inman introduced us to the Open Web Education Alliance.

Lynne D Johnson showed us how new media means new business.

Tania Lang taught us how to use AJAX to enhance user experience.

Divya Manian got creative with CSS3 and also showed us how to be active web developers.

Hilary Mason introduced us to machine learning for web data.

Juliette Melton has taught us how to do mobile UX research and how to conduct effective remote studies.

Zoe Mickley Gillenwater showed us to create effective and efficient designs with CSS3.

Diana Mounter talked us through deciding between custom and CMS.

Jackie Moyes showed us how to convert research findings into business speak.

George Oates showed us how web apps need developers and designers to work together and also told us all about human traffic.

Laurel Papworth has shown us the business of online communities, the business of being social and taught us all about social networks and mobiles.

Silvia Pfeiffer showed us how to take HTML5 video a step further.

Veerle Pieters showed us how to find creativity in the design process.

Deborah Schultz told us that it’s the people stupid.

Teale Shapcott told us all about usability in agile environments.

Kaitlin Sherwood showed us the business and technology of mashups.

Rashmi Sinha talked us through the perils of popularity.

Kay Smoljak showed us how to start and run a successful web business.

Donna Spencer has given us an IA how-to, showed us how to get content right, how to keep it alive from cradle to grave and taught us about information seeking behaviours.

Nicole Sullivan showed us her CSS power tools.

Stephanie Sullivan-Rewis taught us all about CSS3, the web’s Swiss Army Knife.

Kerry Taylor told us about semantics and sensors.

Jenny Telford showed us how the Australian Bureau of Statistics is opening up government data.

Gina Trapani showed us how Google opened up Gmail’s web interface to any developer.

Lea Verou showed us how to master CSS gradients.

Sandi Wassmer taught us that inclusive design is for everyone.

Gian Wild showed us how she managed accessibility compliance for the Melbourne Commonwealth Games, introduced us to WCAG 2 and then showed us its hidden nuggets.

Alex Young taught us how E is for everywhere and then showed us multi-device, multi-role.

Indi Young told us all about innovation with mental models.

We’ve always loved the coding side of the web here at Web Directions, and our conferences have always had a strong developer track. But with so much going on and so much on the horizon, we’ve decided that in 2012 developers need their very own event as well. So we’re very excited to announce a brand new show, Web Directions Code, this year in Melbourne on May 23 and 24.

Read on for more details, but take note, there will of course be Web Directions South in Sydney in October, as there has been since 2006!

Why Web Directions Code?

As the web becomes increasingly application like, and the capabilities of HTML5 and other web technologies become ever more sophisticated, we thought it was time to put together an event that focusses entirely on code. Web Directions Code features a fantastic international and local lineup, including

• Paul Irish (HTML5 Rocks, ModernizR, HTML5 Boilerplate and much more)
• Divya Manian (HTML5 Please, CSS3 Please, HTML5 Boilerplate)
• Rob Hawkes (Game developer and Mozilla technical lead on gaming)
• Dave Johnson (originator of the phoneGap project)

Basically, if it’s well supported in browsers and devices, and empowers you to do awesome stuff, then we’ll be covering it

Day one is dedicated to HTML5 as an application platform, with sessions on Browser Device APIs, HTML5 Audio and Video, touch events for mobile and tablet devices, offline web apps, HTML5 messaging, the History API, HTML5 technologies for game and interactive application development and more.

Day two is Australia’s first in-depth, JavaScript focussed day, with sessions on Node.js, debugging, performance, client-side templating languages like Mustache, Dust.js, Handlebars.js, Underscore.js, event libraries and more.

If you build things with web technologies, then Web Directions Code is for you: see you there! Just don’t forget to register with that discount code before April 15. It’s just $799 before then. And if you’ve been before to a web directions event, it’s even cheaper. If you didn’t receive an email with your discount code, drop us a line now!

At its most extreme, this is used as an argument for disbanding the W3C, and even for core web technologies to become “a single source repository [with] a good owner to drive it.”

Occasionally history throws up curiously timely experiments. Right now we are seeing a very interesting one (and one with far reaching consequences) play out.

Since the middle to late 18th Century, with the enclosure of the commons, and the rise of industrial capitalism, the belief that ownership and property rights is what has largely driven advancements in our civilisation has become almost all pervasive. It lies at the heart of the, until relatively recently alien, concept of intellectual property (increasingly seen as a bane not boon for innovation).

So, what does this have to do with the clear demise of Flash on mobile? Flash has an owner. One that had, and continues to have large revenues, teams of very smart people, deep pockets.

Despite all this, Flash failed to adapt to changing technological circumstances, and withered on the vine.

In parallel, core web technologies have slowly, inexorably grown more sophisticated, organically, iteratively, cooperatively adding capabilities that, for the most part developers clearly want and need.

Let’s take an example I used in a recent presentation. The DOM, while powerful, has long been a pain for developers to really get to grips with. Recognizing this, various libraries, and most famously jQuery, came up with more developer friendly ways of accessing it. jQuery’s use of CSS selector concepts proved immediately popular, and in short order, the W3C began work on the Selectors API, while browsers also within a relatively short time frame began implementing this API.

An ownership model is different. The owner of a platform or technology makes strategic decisions, and long term bets on what will be successful. Those bets may of course pay off tremendously (as in the case of iOS). But they may not, and very often do not, as in the case of Flash.

Standards bodies are not imune to the ownership model of development. XHTML2 is a decade long demonstration of that.

Technologies with the ownership model seem less capable of adapting to change, and are very dependent on initial conditions. The cooperative, collaborative standards based approach (characterised best by the IETF’s founding principle of “rough consensus and working code”) often seems to build technologies that weather the storms of technological, social and political change far better.

It’s ironic, that the apparently “capitalist” “ownership” model is really much more like the central planned economic model of former socialist countries, while the W3 model more closely approximates how the societies of economically liberal countries work.

Neither model is going away soon. Each will have its successes and its failures. But I think it is time to put to bed the far too pervasive meme that standards are in some way an impediment to innovation. After all, take a look at the web. Built on standards (TCP/IP, http, HTML, CSS, EMCMAScript, the DOM), it’s doing ok. Better than OK I’d suggest.

According to most estimates just over 2 billion currently use the internet regularly (in ten years this has grown from around 360 million).

The growth in the web’s use, even in purely numerical terms is almost incomprehensibly dramatic. From .4% of the world’s population in 1995, to 5.9% in 2000, 13.9% in 2005, to today’s 30.4%.

When you add in the global and cultural reach of the web — currently 11% of the population of Africa, 24% of Asia, 31% of the middle east, as well as around 40% of Europe and Oceania, and nearly 80% of North America — then it’s clear we are seeing a global phenomenon the scale and breadth of which is unlike anything see in human history.

Which is in fact no accident.

The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect

Lately, there has been a lot of concern expressed, by intelligent, experienced people I have a lot of respect for, about the future of the web, about its very viability.

I’ve engaged folks like Joe Hewitt in strenuous, but I believe healthy and important debate about these issues.

As I considered in my recent Web Directions presentation, A Dao of Web Design Revisited (article, slides and audio recording coming soon), I feel that many of these concerns strangely, uncannily, echo those which prompted my original Dao of Web Design article back in 2000.

A recent tweet by Aral Balkan (once again, a passionate, intelligent contributor to the web, who has spoken at our events, and hopefully will do again), retweeted by Lea Verou (another generous contributor to the web, and again, one of our past, and I hope future speakers) really captured for me the essence of the issue

#oneversion #manifesto My websites will only support the latest versions of browsers. It’s the browser makers’ duty to get users to upgrade.

Aral Balkan, Twitter, October 20 2011

So, what does this tweet, and predictions of world population have to do with one another?

I love the shiny stuff of the web — the gradients and animations, the transforms. As someone who has developed for the web for nearly 18 years, the ever increasing sophistication of the DOM, of JavaScript, the increase in the speed of JS engines, of rendering, the arrival of mobile platforms, and micro-mobile platforms for the web excite me as a user and a developer. But, they aren’t what gets me up in the morning. They aren’t what fires me up like no other platform before (I built my first Mac apps in 1986, and have had commercially available Mac OS apps continuously since 1994, and Windows apps since 1996).

What really, at the not so tender age of 45, keeps me as passionate and excited about building stuff as I was when about 16 and got my first TRS80 clone is the potential for the web to transform our world for the better. And overarching all this is the question, the challenge, how do we get the next 2 billion online, and ultimately, the next 6 billion people online?

This might not float your boat. And that’s fine. You might consider it an ideological position. And that’s your prerogative. But I know I’m not alone in believing that the potential, the promise, and in the face of overwhelming planet-wide challenges — anthropogenic climate change, global pandemics to name just two which our generation, and particularly my children’s generation will have to increasingly confront — the necessity of bringing our planet together, and enabling all of us to collaborate, share, communicate, without the friction of borders, is something only the web can hope to achieve.

Universality is a founding principle of the web. It is the manifesto the web has been built on, and I believe one of the key drivers of the almost unimaginable success of the web over these last two decades. We ignore that at the web’s peril.

The web alone, not iOS, or Android, or Windows Phone, or any other platform can possibly connect the next 6 billion. Yes, some, many of those 6 billion will be accessing the web via iOS, some via Android devices, some Windows Phone.

But, this next six billion is children in rural India, Africa, China where access to power, and networks, may be intermittent. It’s someone in Sumatra at a decade old Wintel box. It’s people who speak hundreds of different languages, with dozens of different writing systems. It’s people who are the first in their family to be able to read and write. It’s the 20% of people worldwide who can’t read or write. Yet.

So, to say “My websites will only support the latest versions of browsers”, you are in a sense saying, “I’m going to make the fact that developing for the web is harder than it would be if I concern myself with browsers other than the latest is not my problem, and not even the browser makers problem, it’s the problem of the next 6 billion. It’s not my problem, it’s the problem of the child in rural India, Africa, China.”

The truth is, the challenge of universality is daunting. It is hard work. But to me at least, paying this forward is the quid pro quo of the enormous privilege I’ve been granted to work on the web, which has given me fascinating well paid work, connections with thousands of intelligent, passionate, generous people around the world, and the opportunity to participate, in however insignificant a way, in something genuinely extraordinary, something unique. I can pay this forward by including rather than excluding people. By, in my own small way, helping ensure that the next 6 billion will be able to share in the privilege that I, you and the first 2 billion share in.

Which is not to say we shouldn’t continue to develop the capabilities of web technologies. It is not to say we shouldn’t continually explore what these technologies enable us to do.

But to me at least, we owe it to the web to do this in a way that is generous to the web in the way the web has been generous to us.

To reformulate the now famous question Steve Jobs asked of John Sculley:

Do you want to make shiny products for the privileged for the rest of your life, or do you want to come with me and change the world?

Luckily, through photos, tweets, blog and Facebook posts and even the odd video, increasingly an event like Web Directions is captured in more than just our memory.

Web Directions South, held last week, was the sixth annual Web Directions event we’ve run in Sydney, and though likely we feel this every year, it was in our estimation, and indeed of so many we spoke to, and based on many tweets we’ve read the best we’ve done by far.

The atmosphere was incredibly positive, (reflected in everything from the twitter stream, to the crowds who refused to go home, staying long into the night at the closing night party).

A new generation of partner companies, almost all from Australia and New Zealand, and most of which did not exist when Web Directions started back in 2006 companies shouted great parties, competitions, and most importantly showcased the sophistication and breadth (as well as spirit of fun) of the Australian web industry.

Similarly, while many of the speakers, local and international, may have been unknown to our audience, they excelled–educating, inspiring and entertaining. From opening keynote to closing, never have we seen such uniformly positive responses to a program.

Every presentation we had the privilege to see, as well as all those we didn’t but which we heard amazing things about, was world class.

Whether you were there, or missed out, as the podcasts and slides from presentations come online, we’ll tweet about them (so if you don’t already, you really should follow us on Twitter. Meanwhile, you can catch up on hundreds of past presentations from our conferences around the world from the last several years.

And, as a special treat, coming soon is a slickly produced video version of James Bridle’s amazing closing keynote, thanks to Hunting with Pixels.

In the meantime, Ben Buchanan has already made available his traditional big stonking post™ with detailed live notes, links, and photos from a dozen presentations. So, if you missed it, and can bare to see what you missed out on, it’s a great place to start.

Thanks!

• A huge thanks to our wonderful sponsors, partners and exhibitors
• To our incredibly generous speakers
• To Cameron Adams for the opening sequence, setting a new standard for creativity with Web Technologies
• To the Design Computing program at the University of Sydney for their incredible exhibition of interactive technology
• To the W3C Australia Office for once again hosting the W3C South Track
• And not least to the fantastic, positive, open minded, engaged audience.

In the middle 1990s, by an accident of fate and coincidence far too tedious to go into here, I found myself teaching some of the earliest web related courses at TAFENSW. A lot of my focus was in getting students to think conceptually about document structure, linking, interactive tables of contents and so on. Very little was about HTML. Indeed, so convinced was I that WYSIWYG editing was just around the corner, I felt it a waste of their time learning markup (this was pre CSS, but that’s another story). We used Claris HomePage, a surprisingly useful application, to develop web sites as part of these courses.

Pretty much everyone at the time felt that actually working at the markup level for the web would go the way of assembly programming.

There were reasonable precedents to this belief. Desktop publishing in the middle 1980s (really only around 10 years before this time) had revolutionized traditionally markup based print page layout (and more or less given the world the term WYSIWYG).

And, around that same time (the middle and latter half of the 1980s), I was studying Computer Science at University, where for our final year major project, we were the first cohort to no longer use COBOL (I kid you not). Rather we used a 4GL named Omnis (4GLs or Fourth Generation Languages were supposed to abstract away all the messy stuff when developing complex applications). The consensus was (and this was a classic CS program with lots of C, and algorithms and stuff) that the traditional approach to programming was going the way of assembly language.

4GLs live on in systems like 4D, but you would have shocked us all back then if you said that we’d still be using 3GLs without even garbage collection today.

So, why did WYSIWYG revolutionize print design, but not application development?

Similarly, despite the success of, above all DreamWeaver, and to a lesser extent other WYSIWYG web development tools, we’re still very much working at the codeface (I don’t think I know any professional web developers who use a WYSIWYG tool these days, though many, myself included, once did).

But why is that? Will it always be the case? Or will one day someone crack the code (literally and metaphorically) and deliver a WYSIWYG tool that liberates us from the “grind” of coding.

But first, let’s consider why WYSIWYG was so successful in the world of print. I’d suggest a big part of that success was that is possible to map every aspect of a printed page onto the screen (or better, into the internal model of the software). You can even map colors onto a grayscale screen, which many desktop publishing systems were in the mid 1980s, color screens were seriously expensive. So, there was little if any need for metaphor or abstraction. This part of the page right here, will be pantone color whatever. And this little bit over here, will be white. And so on.

The problem space of printed page layout was sufficiently constrained that everything the designer might want to do, they could do with DTP software. I’d argue that a constrained “possibility space” is a key requirement for WYSIWYG systems to really work.

So, what about the web? Douglas Engelbart, one of the giants of modern computing, coined the term “WYSIAYG” (what you see is all you get), which I think captures the challenge faced in developing web development tools, and software tools more generally. With desktop publishing, all you get is all you need. As we saw, the space of possible outcomes can be mapped into a set of tools.

But with the web (even leaving aside JavaScript and the DOM), the possibility space is I’d argue, too open ended to map well onto any WYSIWYG software. So the designer of development tools needs to decide what the all is. Unlike print design, where the problem was constrained by the medium, with the web, software designers constrain the problem space.

So, if you were developing a “WYSIWYG” web development tool, how would you enable your users to make the first paragraph after a heading of level 2, in the introductory section of a document bold? You could allow users to add a class (though you may or may not use that term) to that particular page object (or do you call them elements?). Or, you could provide an interface that is familiar from word processors, where the user selects text, and a bold tool, and behind the scenes the software adds a strong element (to code the user will never see). Or you might add inline style of bold to that paragraph.

And this is about as simple a scenario as it gets. No interaction. Just styled text based on document structure.

Now, what if a document is divided into chapters. And what if the designer wants the first line in the first paragraph in each chapter to be capitalized? Or every other row in a table to have a given background color?

The best possible approach to these particular challenges at the markup and CSS level is likely to use structural selectors (e.g. nth-child). But how to intuit from the users’ actions their intent, and map that intent onto the capabilities of CSS?

And we still haven’t even introduced any interactivity to the challenge yet.

This isn’t simply a theoretical issue for me. I’ve been developing the CSS Editor Style Master for well over a decade. One of the design principles for it has been to never hide the underlying CSS concepts behind metaphors and abstractions. Rather, the goal is to make understanding, and using, these powerful features of CSS easier, more productive, and more enjoyable. This same design principle informs my work on various CSS3 tools, for example my gradient and animation tools. While other tools (and this is not a criticism, simply an observation), create metaphors (for example, provide a Photoshop-like interface for creating gradients), I emphasise a direct correspondence between the objects you manipulate, and the underlying CSS concepts.

But, to return to the question, is WYSIWYG possible for the web? I actually think that it is, but not in general. But in situations where a specific problem is sufficiently constrained, I think we can provide WYSIWYG tools. Which means the approach Adobe is taking with MUSE, will run up against the same challenges that all general purpose WYSIWYG web development tools have. That’s not to say it might not be very useful, but will be so within the constraints of its capabilities, of the problem space its designers have chosen to map.

Where I do think WYSIWYG can be, and already often is very successful is in constrained problem spaces. Blog posting is a particular example. An area I’m focussing on right now is presentations, which I believe is sufficiently constrained to make a WYWIWYG approach meaningful (but even here, I think being able to bust out an editor for markup, CSS or JS is important).
Ironically, you could create a WYSIWYG print design tool with web technologies, for precisely the same reason you can do it with other languages and development platforms.

In essence, the set of problems the web presents developers and designers is open ended in a way that no other system is. Viewport sizes can range from 128 x 128 pixels (or less) to 3840 x 2160, and no doubt beyond. Pixel density, color depth, available fonts, all of these will vary dramatically. Then there’s varying levels of support for CSS, HTML, DOM, SVG. There’s varying network speeds, drastic performance differences. Different input methods (mouse, keyboard, touch, Kinect, …) And none of these are going away. They are getting “worse” (which ironically means better — because more diversity among these characteristics means more people are using the web in more ways, which is good). The fact it causes challenges for designers and developers is essentially irrelevant. That’s our problem.

The search for an over-arching WYSIWYG solution to the challenge of developing for the web reflects our ongoing misconception of what the web is. It’s a medium unlike those which it is in some ways conceptually built on — be that paper, TV or desktop platforms like Windows. It’s understandable that we want it to be more like these. We know them well, we’re comfortable with their concepts and techniques. We’ve invested hugely in technology and expertise to design and develop for them. The web continues to make us uncomfortable. As it should. It is something new, and exciting, and challenging, and unfinished. What we see is never going to be all web get. That’s the challenge, and beauty, of the web.

To contrast with this glacial change, we’re typically pointed toward the wonders of platforms like iOS and Android, where change takes place at breakneck speed. Or we’re pointed toward any number of open source projects, for example SASS, to demonstrate that we can evolve web standards faster outside the W3C (afterall, CSS hasn’t kept place with SASS/LESS… therefor the W3C is clearly useless)

Pretty sure that already happened. SCSS, Coffeescript, etc. The W3C doesn’t have influence anymore AFAIK

And occasionally, this erupts in an orgy of revolutionary fervour, where we need to man the barricades, and

dissolve the W3C, and run the web like an open source project. No more specs, just commits. Does Linux need a standards body?

I guess Joe hasn’t taken a look at the WHATWG’s “HTML“lately. An unversioned. constantly, continuously evolving potage, wherein creeps all kinds of at the every least questionable “innovations”, whose trajectory is unburdened by such trifling challenges as the need for intellectual property policies, or getting anyone to actually implement their stuff.

But let’s step back a pace. Is there really actually such a problem here? And if so, exactly what is that problem? And, are the proposals, to the extent they do exist, likely to help, or make matters worse?

Firstly, it is fair to say, that for the best part of the first decade of this century, we saw considerable stagnation in the development of web standards. The W3C went down the rabbit hole of XHTML2, and CSS 2 almost entirely stagnated (I’l have more to say on that in a second). During that time, innovation came almost entirely from developers discovering techniques which worked with existing browser capabilities, and through JavaScript libraries.

The last three or four years have however seen an explosion of innovation at the browser level. This has included:

• Huge improvements in the performance of JavaScript engines
• Fantastic new CSS features, that have within a couple of years landed in all modern browsers
• Sophisticated new APIs across a broad range of functionality, again now widespread across most if not all modern browsers (Selectors API, Geolocation, offline, localStorage, 2D and 3D canvas to name just a few)

Before we throw the baby out with the bathwater, we should ask, is there a problem at all? Are we really seeing innovation — from standards proposal to widespread browser adoption in the order of 10 years?

Let’s take a look at an actual example. Dave Hyatt announced CSS Gradients in WebKit nightlies April 14th 2008. More or less simultaneously, Apple formally proposed these as part of CSS3. Gradients have been supported in Firefox since 3.6 (released Jan 2010), Opera since 11.1 (final release mid 2011), Internet Explorer 10 (developer releases early 2011). So, from first released and proposed as a standard to very widespread adoption in around 3 years.

We’ve seen similar timeframes for other CSS features, as well as “HTML5” features like the Selectors API, appcache, geolocation, localStorage.

So, to put it bluntly, I think the problem is overstated. We seem to have arrived at an approach that both enables the exploration and implementation of novel features in browsers, which are also widely adopted across browsers. It’s that second part that is most important, and I’ll return to it in a moment.

But I also want to quickly address why I think we seem to have thrown off the stagnation of previous years. I’d argue it comes down to stepping back from the monolithic approach of earlier W3C recommendations (CSS 1 and 2, XHTML2, SVG), toward a much more modular approach, as exemplified by CSS3, but as also seen with geolocation and the selectors API. A corollary to this is, I’d argue, that HTML5 is too monolithic and needs to be fragmented (which in some ways is happening).

But back to the real issue. The web is a different problem. It makes little if any sense to compare innovation of the web ecosystem with that of iOS, Android or other platforms. The web faces challenges far far greater (and has goals far more important). A platform such as iOS can abandon legacy applications, content and hardware, (along with their users) with little compunction. It can (and does) make developers and content creators wishing to participate jump through any number of hoops. It has a single dictatorial decision maker, beholden to no one, and nothing other than itself. And it generates extraordinary revenues, which can be reinvested into the ongoing development of the platform.

The web is different. It values interoperablity, backwards compatibility. It’s goal is to bring access to the same information to billions across the world, on all manner of devices.
Its custodians are, in my opinion, scandalously under-resourced, given just how much wealth the web has created for so many, perhaps above all Google and Apple.

Without the web, Google would not exist, and Apple’s core engine of growth, iOS devices would essentially not either.

So, rather than generally criticising the W3C, or going so far as calling for its dissolution, we should focus on how well in many ways it has done an almost impossible task — getting companies which are fierce commercial rivals to sit down, work together and agree on core technologies they will each, and all, implement, even while at the same time, these same competitors are involved in significant legal conflicts with one another.

Can the W3C be improved? Certainly. But before suggesting solutions, let’s identify, and demonstrate with evidence, genuine problems. Then, when devising solutions, it pays to ask what evidence there is that those solutions will not only solve the problems identified, but also ensure the ongoing cooperation that has given rise to the web.