AS/JS Virtual Machine Performance Race

This summer, I'm luckily interning at Adobe working on Tamarin. Since I haven't touched Tamarin in a while, I thought it should be worthwhile to see how Tamarin compares to all the other JavaScript virtual machines. Although Tamarin isn't a JavaScript VM, there aren't any other ActionScript VMs to compare it to. Without further adieu, the SunSpider benchmark contenders:

  • Tamarin-Central: Stable Flash VM. Runs ActionScript
  • Tamarin-Redux: The tip for Tamarin-Central. Every once in a while the Tamarin-Redux changes are pushed into the Tamarin-Central branch
  • TraceMonkey: Tracing JIT slated for shipment in Firefox 3.1
  • V8: Used by Google Chrome
  • Nitro: Previously named SquirrelFish Extreme, the VM in Safari.
A few notes:
  • Tamarin-Central/Redux AS3 - SunSpider tests written with static typing. Only available for Flash VMs.
  • All VMs are from their respective repositories and checked out 3/22/09 ~6 PM PST. Built with optimizations/release builds.
  • All tests run on a MacBook Pro 2.4 Ghz w/ 4 Gigs of ram.
  • These tests run really really fast (some less than three ms)
  • The baseline is the Tamarin-Central interpreter (JIT disabled)
  • Some SunSpider tests are omitted as Tamarin can't run them because of known issues.
  • Take these numbers with a grain of salt :)

New SquirrelFish Extreme

A fresh out of the oven Squirrelfish called SquirrelFish Extreme was announced here. They attribute their speedup to bytecode optimizations, Context Threading, a property cache, and a regular expression JIT.

I tried a context threaded interpreter in Tamarin-Tracing, which significantly sped up the interpreter, but jumping from a trace back into the interpreter was too expensive. David Mandelin has been looking at adding inline threading/context threading into SpiderMonkey. Something like a property cache is in SpiderMonkey and has given SpiderMonkey significant performance improvements. Finally, we were thinking about doing a regular expression tracing JIT, but have yet to implement it.

Cameron Zwarich already has some performance benchmarks of SFX (SquirrelFish Extreme) to TraceMonkey and Google's V8. Browser war round 2! FIGHT!

TraceMonkey vs SquirrelFish

With the announcement of TraceMonkey, we saw that SpiderMonkey got a huge improvement. However, in previous benchmarks, SpiderMonkey was still slower than SquirrelFish. So, how does Trace Monkey compare versus SquirrelFish:

Overall, if you aggregate the time for each test, TraceMonkey is about 15% faster. If you do the aggregate speedup, which is the speedup for each test divided by the number of tests, it is about 2.4 x faster. Sometimes TraceMonkey is slower, but give them more than 60 days to hash it out, and I'm sure it'll get faster.

Notes: SquirrelFish and TraceMonkey were with release builds. Tests run on a MacBook Pro 2.4 ghz, 4 gigs of ram, OS X Leopard 10.5.4. TraceMonkey was just pulled about an hour ago. SquirrelFish was last nights SVN.

Good Job Guys. More info on TraceMonkey at Andreas Gal's Blog, Mike Shaver, David Anderson, and for a real visual demo, checkout Mike Schroepfer's blog.