This summer, I'm luckily interning at Adobe working on Tamarin. Since I haven't touched Tamarin in a while, I thought it should be worthwhile to see how Tamarin compares to all the other JavaScript virtual machines. Although Tamarin isn't a JavaScript VM, there aren't any other ActionScript VMs to compare it to. Without further adieu, the SunSpider benchmark contenders:

• Tamarin-Central: Stable Flash VM. Runs ActionScript
• Tamarin-Redux: The tip for Tamarin-Central. Every once in a while the Tamarin-Redux changes are pushed into the Tamarin-Central branch
• TraceMonkey: Tracing JIT slated for shipment in Firefox 3.1
• V8: Used by Google Chrome
• Nitro: Previously named SquirrelFish Extreme, the VM in Safari.

A few notes:

• Tamarin-Central/Redux AS3 - SunSpider tests written with static typing. Only available for Flash VMs.
• All VMs are from their respective repositories and checked out 3/22/09 ~6 PM PST. Built with optimizations/release builds.
  
• All tests run on a MacBook Pro 2.4 Ghz w/ 4 Gigs of ram.
• These tests run really really fast (some less than three ms)
• The baseline is the Tamarin-Central interpreter (JIT disabled)
• Some SunSpider tests are omitted as Tamarin can't run them because of known issues.
  
• Take these numbers with a grain of salt :)

With the announcement of TraceMonkey, we saw that SpiderMonkey got a huge improvement. However, in previous benchmarks, SpiderMonkey was still slower than SquirrelFish. So, how does Trace Monkey compare versus SquirrelFish:

Overall, if you aggregate the time for each test, TraceMonkey is about 15% faster. If you do the aggregate speedup, which is the speedup for each test divided by the number of tests, it is about 2.4 x faster. Sometimes TraceMonkey is slower, but give them more than 60 days to hash it out, and I'm sure it'll get faster.

Notes: SquirrelFish and TraceMonkey were with release builds. Tests run on a MacBook Pro 2.4 ghz, 4 gigs of ram, OS X Leopard 10.5.4. TraceMonkey was just pulled about an hour ago. SquirrelFish was last nights SVN.

Good Job Guys. More info on TraceMonkey at Andreas Gal's Blog, Mike Shaver, David Anderson, and for a real visual demo, checkout Mike Schroepfer's blog.

How many awesome names can we have? LOL I love these hybrid animals. A comment on Slashdot says its like the new Web 2.0 naming scheme. Reminds me of SpiderPig! Anyway, the Webkit guys released their new JavaScript Interpreter, squirrelfish. The benchmarks show that it is pretty fast. Cameron has a post detailing the overall picture, but what about the micro picture? The following is a graph of each SunSpider test run on SquirrelFish(WebKit-r34342) and SpiderMonkey 1.8 (built on May 22 2008 at 13:31:57). The shorter the bar the better.

If you wish to rerun these tests, heres how to build WebKit. If you specifically want to build just SquirrelFish, use the following command:

1) WebKitTools/Scripts/build-testkjs
2) The binary is /WebKitBuild/Release/testkjs

Props to the webkit guys.

* Edit as Mark mentioned. All tests run on OS X 10.5.3 on a Macbook Pro 2.4 ghz w/ 2 gigs of ram. Both interpreters were built on my machine with optimizations/release build.

There was some discussion this morning about whether or not it was possible to get Tamarin's Interpreter up to speed with SpiderMonkey, and if so, how far is Tamarin? Instead of using JavaGrande, we decided to use SunSpider, since it has some tests which use Objects. If anyone's interested, you can only get SunSpider from their source repository. All benchmarks were done on a MacBook pro 2.4 ghz with 2 gigs of ram. Both were built as Release/Optimizations enabled.

Note that with Tamarin, the -interp flag has to be set to test the interpreter only. I couldn't find a way to run it through the sunspider test harness. So I wrote my own bash script which invoked Tamarin 10 times, and used the unix 'time' results. Also, some of these tests failed.

The picture below is how much slower Tamarin is compared to SpiderMonkey. The smaller the bar, the faster Tamarin executed the test. The graphic only shows tests which passed. There was no test in which Tamarin was faster than SpiderMonkey, therefore this pic only shows how much slower Tamarin was than SpiderMonkey. Below the pics are the actual numbers.

SpiderMonkey 1.8
built on May 22 2008 at 13:31:57

3d: 347.4ms +/- 0.4%
cube: 126.6ms +/- 0.5%
morph: 118.0ms +/- 0.4%
raytrace: 102.8ms +/- 0.4%

access: 415.5ms +/- 1.5%
binary-trees: 43.3ms +/- 4.0%
fannkuch: 193.3ms +/- 1.4%
nbody: 125.1ms +/- 1.4%
nsieve: 53.8ms +/- 0.6%

bitops: 397.8ms +/- 0.4%
3bit-bits-in-byte: 49.6ms +/- 1.0%
bits-in-byte: 79.1ms +/- 0.3%
bitwise-and: 178.3ms +/- 0.4%
nsieve-bits: 90.8ms +/- 0.7%

controlflow: 35.1ms +/- 0.6%
recursive: 35.1ms +/- 0.6%

crypto: 169.9ms +/- 0.3%
aes: 66.1ms +/- 0.3%
md5: 51.9ms +/- 0.8%
sha1: 51.9ms +/- 0.4%

date: 269.3ms +/- 0.4%
format-tofte: 160.3ms +/- 0.3%
format-xparb: 109.0ms +/- 0.7%

math: 288.5ms +/- 0.3%
cordic: 128.9ms +/- 0.4%
partial-sums: 102.0ms +/- 0.3%
spectral-norm: 57.6ms +/- 0.6%

regexp: 251.9ms +/- 0.2%
dna: 251.9ms +/- 0.2%

string: 594.0ms +/- 0.3%
base64: 68.7ms +/- 0.5%
fasta: 148.7ms +/- 0.2%
tagcloud: 134.2ms +/- 0.8%
unpack-code: 169.4ms +/- 0.2%
validate-input: 73.0ms +/- 0.7%

Tamarin:
3d: 2035 ms
cube: 457 ms
morph: 608 ms
raytrace: 970 ms

access: 3548 ms
binary-trees: 363 ms
fannkuch: 1819 ms
nbody: 1047 ms
nsieve: 319 ms

bitops: 3929 ms
3bit-bits-in-byte: 275 ms
bits-in-byte: 371 ms
bitwise-and: 2454 ms
nsieve-bits: 829 ms

controlflow: 282 ms
recursive: 282 ms

crypto: 1440 ms
aes: 785 ms
md5: 322 ms
sha1: 333 ms

date: Fail
format-tofte: Reference Error #1065
format-xparb: Reference Error #1065

math: 2637 ms
cordic: 616 ms
partial-sum 1584 ms
spectral-norm: 437 ms

regexp: Fail
dna: ArgumentError: Error #1063

string: 56445 ms
base64: ReferenceError: Error #1069
fasta: 837 ms
tagcloud: ReferenceError: #1065
unpack-code: 50234 ms
validate-input: 5374 ms

Tamarin is slower. The worst part has to be playing with strings. Ouch.

Andreas pointed out that the for loop example might be a bit more interesting if we add type annotations to the source. So let's take another look at Tamarin with added type annotations. The new found source:

package{
var j:Number = 0;
for (var i:Number = 0; i < 100000000; i++) {
      j++;
}
}

Here are the numbers:

Tamarin-Central Compiler
time ./shell -Dforcemir ~/Projects/tamarin-central/test/custom/forLoop.abc
real 0m0.452s
user 0m0.445s
sys 0m0.007s
Tamarin-Tracing Single Interpreter - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/forLoop.abc
real    0m5.006s
user    0m4.923s
sys    0m0.041s
Tamarin-Tracing Double Interpreter - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/forLoop.abc
real    0m6.529s
user    0m6.496s
sys    0m0.033s
SpiderMonkey 1.8 (Copied from previous post for convenience)
time ./js ~/Projects/tamarin-central/test/custom/forLoop.js
real    0m6.396s
user    0m6.385s
sys    0m0.011s

Even with type annotations, Tamarin squeaks by SpiderMonkey 1.8. Remember, the previous test was untyped code.

Finally, what if we change the type from Number to integer. The new source code:

package{
     var j:int = 0;
     for (var i:int = 0; i < 100000000; i++) {
             j++;
     }
}

And the results:

Tamarin-Tracing Single Interpreter - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/forLoop.abc
real    0m2.384s
user    0m2.331s
sys    0m0.034s
Tamarin-Tracing Double Interpreter - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/forLoop.abc
real    0m1.301s
user    0m1.273s
sys    0m0.027s
Tamarin-Central - Compiler on
time ./shell -Dforcemir ~/Projects/tamarin-central/test/custom/forLoop.abc
real 0m0.373s
user 0m0.362s
sys 0m0.007s

The resulting speed boost from using an integer type is quite nice.

*Added the flag -Dforcemir to tamarin-central to force compilation. Thx for the tip Rick.