Wednesday, February 10, 2010

EcmaScript 5 and Harmony

I gave a talk on EcmaScript 5 and 6. My slides are available as HTML.

My pre-talk notes below capture the gist of what I said:

I'm sure you're all aware that browsers and standards bodies have been working on new versions of JavaScript and HTML. I've been working on client side code for a while, so I'm excited about a lot of these changes. I worked on GWS, and then was the first client dev on calendar. I worked on Closure Compiler, doing optimization and type system work. Recently, I've been working with the EcmaScript committee to try and make it a better language for writing secure and robust programs.

I'd like to go over some of the languages changes in the recently release ES 5, and summarize some of the topics being discussed for the next version. Then I'm going to talk about some strategies that might help you start using new language features in code that needs to work on older interpreters.

There's a lot to cover, so I'm going to go pretty quickly, so please jump in if you see something that grabs your interest.

First, a bit of history. When I say "EcmaScript," I mean "JavaScript." Browsers are working on implementing the latest version EcmaScript 5. Most of you are probably familiar with EcmaScript 3. That leaves ES4 which never made it out of committee. ES4 went on for years without producing a standard, and it's now officially dead. You may be familiar with some of the proposals that came out of it, like the type system on which Closure Compiler's type system is based. But most of the work from that did not make it into ES5.

While ES4 was languishing, the ES3.1 group started from scratch with more modest goals, and eventually produced ES5. There are a few principles that they used to decide what to include and what not to. They wanted to standardize extensions and quirks fixes that 3 out of 4 major intepreters agreed on to move divergent implementations towards compatibility, and address some of the major sources of confusion using an opt-in mode that breaks backwards compatibility.

First major change. ES5 lets user code deal with properties the same way browser intrinsics like DOM nodes do. Many interpreters had implemented getters, setters, watchers, or some similar mechanism; but each was different. This example shows a way of defining a getter and setter using a syntax familiar to anyone who has used SpiderMonkey extensions. This is just a convenience for a richer API though.

defineProperty exposes the full power of the new property APIs. It lets you add a new property to an existing object - either a value property or a dynamic property. And you can specify property attributes -- whether the property is read-only, shows up in a loop, and whether the property can be deleted or reconfigured. The top example shows a value property, and the bottom shows a dynamic property definition.

These APIs are a bit unwieldy, so there are conveniences on top of them. Object.defineProperties lets one combine multiple property definitions. Object.create combines object creation with property definition. Object.freeze makes all properties read-only and unconfigurable, and flips a bit on the object so that new properties can't be added. This gives a shallow immutability which can help code expose objects to the outside world, while maintaining invariants without all kinds of defensive copying.

ES5 also standardizes a lot of interfaces that various libraries have proven very useful. Maybe you're familiar with Closure's goog.array. These interfaces are very similar but implemented natively, and available even without a full-scale library. There are all the usual functional operators, plus a few extra flavors of reduce. There's also a curry operator, the new bind method of function. You give bind a value for this, and optionally some positional parameters and it returns a function that delegates to the original with those parameters.

Another API that most JS libraries reinvent are ones for serializing and deserializing data. Many do this badly, and the number of JSON parsers floating around with known XSS holes is depressing. Most modern interpreters have native JSON support built in. This is a safer way to unpack data. And on calendar, we had to jump through all kinds of hoops to get data unpacking to be as fast as possible. The JSON grammar is significantly simpler than the JS grammar, so native JSON parsing is often faster than eval. It's roughly twice as fast as eval on FF 3.5, though I think eval on V8 is currently faster since they've spent more time optimizing it. One of the nice things you may not be aware of with the new JSON APIs is revivers and replacers which let you serialize/deserialize types other than Array and Object. You can also use it to get better compaction, by doing your own constant pooling. This example shows something that I did for calendar to get our messages smaller -- since instances of repeating events often contain repeated strings. You can use JSON revivers to get the same effect without needing arbitrary JSON. This code is not ideal for a slide, but it shows the basic contract of JSON revivers, and replacers are basically the dual of this.

Besides new APIs, the ES5 group tried to fix spec errors that led to unnecessary confusion. The ES3 spec specified the curly bracket object notation in terms of looking up the name Object in the current scope and calling it as a constructor. This meant that a whole slew of identifiers -- Object, Array, Function -- had special significance. That's no longer the case. Most ES3 interpreters stopped doing this because it allowed some really sneaky cross-domain attacks.

The second example shows another unintended consequence of poor spec language. Function calling is specified in terms of an abstraction called, appropriately enough, a lexical scope. The spec said that a lexical scope is an instance of Object, and so property lookup would find not just declared local variables, but any variables on Object.prototype.

Third, some interpreters pooled regular expression objects. This is problematic because regular expressions are stateful. If a regular expression is being used in a global match, or is used without supplying a new string, then its behavior depends on lastIndex. Which can be mutated by code called in between matches.

Lastly, it was very hard for interpreters to optimize local variable access because of eval. ES3 said that interpreters could raise an error any time eval called via a name other than "eval", but few did. In the above code, not only can eval be used to steal data across scope boundaries, but it show why interpreters can't do lots of interesting optimizations unless they can prove that eval is never called. Now, if eval is called by that name, and refers to the same function as the original global eval, then it reaches into the local scope. Otherwise, all free variables are interpreted in the global scope.

ES strict mode is an opt-in mode that removes a number of other sources of bugs and confusion.
You opt in by putting the string literal "use strict", by itself, at the top of your program or function body.
Any functions contained, and any code evaled inside a strict context is itself strict.
Strict mode differs from normal mode in a few ways. "this" is not coerced to an object in strict mode.

In the Boolean.prototype.not example, this would normally be coerced to an Object, so !this is always false. Not in strict mode.
this can also be null or undefined. It won't magically become the global scope, so code which inadvertently used call or apply with null won't accidentally clobber global definitions. Because of this, in strict mode, there are strictly fewer type coercions.

And, basic operations no longer fail silently. Things like deleting a non-existent property or setting a read-only property will throw an exception instead.

Where ES5 was concerned with fixing known problems, ES6 is concerned with defining a few features that let developers write programs that simply couldn't work under ES5 due to efficiency or scaling issues, or kludgey grammar.
I'm going to walk through a number of ideas the committee is toying with. Some of these are pretty likely to make it into the next version, and some are highly speculative, but I'd love to get feedback on what people like and don't like. After that, I'll talk about strategies for incorporating new language features into code that needs to also run on older interpreters.

One of the things that everyone wanted to get into ES5 but couldn't was let-scoping. If you've seen "let is the new var" t-shirts, this is what they're talking about. Right now, all variables declared in a function are visible everywhere in that function, modulo catch blocks. This leads to a lot of confusion around closures and variables apparently declared in loops. I'm sure you've all had trouble with something like this example where a closure accesses a loop counter i. That problem will go away with let scoping.

A much more controversial proposal is lambdas. If you write a lot of code in JS in a functional style, this would be useful to you. Most versions of lambdas are a reduced syntax for closures that repair the Tennent's correspondence violations that JS functions have. this, arguments, break, continue, and return, all change meaning as soon as you put them inside a function. With lambdas, that is not the case. If anyone is familiar with Neal Gafter's closures for Java proposal, this is similar.

Most everyone agreed that some support for classes is needed, but there's no concensus on exactly what that is. The general feeling is that some syntactic sugar over existing things like constructors and prototypes is probably the best way. This syntactic sugar approach would make it very easy to use classes in code, and down-compile to older versions of ES.

ES has no IdentityHashMap type, or System.identityHashCode. And it probably never will, but sometimes it's hard to write some algorithms without it -- e.g. keeping a list of visited nodes to avoid cycles. Ephemeron tables hash by identity, but without exposing indeterminism. There's no way to iterate over keys, so key order is not a source of indeterminism. And all keys are weakly held, so they don't complicate garbage collection. A value in an ephemeron table is only reachable by the garbage collector if the table and key are independently reachable.
Something like this proposal seems to have fairly broad support.

There seems to be general consensus that modules are needed, but there are active discussions of both means and goals. Some questions raised are how this ineracts with browser fetching, whether modules should be parameterizable ; basically should modules be stateful, or should the state be moved to instances ; and is an isolation mechanism needed so that code can be isolated from changes to Object.prototype and the like. Ihab Awad did a lot of spec writing on that, so can answer detailed questions afterwards.

The committee is listening to proposals around domain-specific language support in ES. DSLs like E4X and CSS selectors have been implemented as extensions or in libraries. So they're definitely useful for creating content in other languages, and for query languages. They could be used to implement new flow control constructs if done right. In the first example here, someone wants to create a regular expression to match a mime-content boundary. But the boundary string could contain special characters, so they use a DSL. The DSL desugars to a function call that gets the literal string portions, and embedded expressions as lambdas. If the re function is const, then interpreters can inline the result since all the parameters are statically known.
These DSLs can also be used to do structured string interpolation -- the syntactic flexibility of perl string interpolations without the horrible XSS problems.
And the last example shows a control structure.

There's also a lot of support for value types. Types that don't compare for equality using reference identity. IBM very much wants decimal arithmetic in, and this is one way to do this. It would also let structured string interpolation behave like real strings.

So that lets user defined code implement new value types. Proxies give user code great flexibility over reference types. I mentioned that there is a known hole in getters and setters. You can't intercept accesses to properties that you haven't defined. Properties do that in a way that preserves the semantics of Object freezing, and avoids a lot of complexity around prototype chains. Instead of defining a property handler of last resort on an existing object, proxies are new objects that delegate all property accesses to handler functions. Those functions can in turn delegate to another object.

Destructuring assignments are syntactic shorthand for unpacking data structures. They are not a general purpose pattern based decomposition as in OCAML or Scala.

Iterators and generators are another idea for which there is wide support but no definite plans.