Tamarin Benchmarks Redux - How much faster?

Tamarin-Tracing's interpreter was just totally revamped with this latest patch. More or less every post about the Interpreter is moot, as there is no longer double interpretation. Instead of having a .abc be interpreted in Forth, then interpreting Forth in C, the new patch translates .abc to the Forth intermediate language. This IL is then interpreted. Therefore, there is only one interpreter instead of two. There are also a whole slew of other changes you can read about in the patch.

I just wanted to see how much faster the new interpreter is. This is without a doubt, not a comprehensive nor complete picture. The benchmark I'll be using is heap sort, which randomly generates 1 million random numbers and sorts them using the heap sort algorithm. This is a JavaScript translation from the Java JavaGrande2 benchmark suite. If you want the source, feel free to ask.

Let's take a look at the relative numbers. All benchmarks are on a MacBook Pro 2.4 ghz w/ 2 gigs of ram. Both Tamarin old/new are built with the "Release" configuration, and run with the -interp flag so we can look purely at the interpreter. The SpiderMonkey benchmark is here simply for comparison, as is tamarin-central. Both were taken from the CVS snapshot 5 minutes before this post. SpiderMonkey was built with optimizations on. Tamarin-central was built with the Release configuration. Here are the numbers:

Tamarin with Double Interpretation - Compiler off
time ./avmplus -interp ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real    3m33.839s
user    3m33.700s
sys    0m0.072s
Tamarin with Double Interpretation - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real 0m12.201s
user 0m12.105s
sys 0m0.095s
Tamarin with Single Interpretation - Compiler off
time ./avmplus -interp ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real 3m8.205s
user 3m8.080s
sys 0m0.075s
Tamarin with Single Interpretation - Compiler on
time ./avmplus ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real 0m11.698s
user 0m11.646s
sys 0m0.048s
SpiderMonkey (JavaScript-C 1.7.0 2007-10-03)
time ./js ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.js
real 0m27.779s
user 0m27.611s
sys 0m0.123s
SpiderMonkey (JavaScript-C 1.8.0 pre-release 1 2007-10-03)
Build Date: built on Apr 26 2008 at 18:21:38
time ./js ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.js
real 0m11.671s
user 0m11.542s
sys 0m0.082s
Tamarin-Central - Compiler off
time ./shell -Dinterp ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real 2m6.523s
user 2m5.880s
sys 0m0.538s
Tamarin-Central - Compiler on
time ./shell ~/Projects/tamarin-central/test/custom/JGFHeapSortBench.abc
real 0m8.566s
user 0m8.538s
sys 0m0.021s

The scariest part about all of this, is how fast SpiderMonkey is. It's only an interpreter! Also, whatever the guys at Mozilla did between JS 1.7 to JS 1.8, bravo. There's still hope though! Tamarin-Tracing is still a work in progress. Adobe said that tamarin-tracing usually beats tamarin-central unless you get some weird looking code. Hopefully we'll start to see that soon.

Edit: Added build date to SpiderMonkey 1.8

Comments

[Andreas Gal]

You should definitively look at different benchmarks as well. Heapsort checks a very specific performance case: sorting arrays. There are several reasons while Tamarin is probably currently not very good at dealing with heapsort. First of all spidermonkey does shape lookups via a property tree, so instead of consulting a hash table for property access it usually knows the direct index. Also, spidermonkey uses dense arrays whereas this optimization is missing from Tamarin as far as I know (maybe Ed will comment on this). Run some numeric benchmark like SOR and lets see how things look there.

[shaver]

I'm curious that it reports itself as being from 2007, but the array optimizations from Feb 2008 could be a big part of the win there.

[shaver]

I forgot to mention -- Tamarin does have dense arrays, pretty similar to those now in Spidermonkey.