A Better View of Tamarin and SpiderMonkey's Interpreter

There was some discussion this morning about whether or not it was possible to get Tamarin's Interpreter up to speed with SpiderMonkey, and if so, how far is Tamarin? Instead of using JavaGrande, we decided to use SunSpider, since it has some tests which use Objects. If anyone's interested, you can only get SunSpider from their source repository. All benchmarks were done on a MacBook pro 2.4 ghz with 2 gigs of ram. Both were built as Release/Optimizations enabled.

Note that with Tamarin, the -interp flag has to be set to test the interpreter only. I couldn't find a way to run it through the sunspider test harness. So I wrote my own bash script which invoked Tamarin 10 times, and used the unix 'time' results. Also, some of these tests failed.

The picture below is how much slower Tamarin is compared to SpiderMonkey. The smaller the bar, the faster Tamarin executed the test. The graphic only shows tests which passed. There was no test in which Tamarin was faster than SpiderMonkey, therefore this pic only shows how much slower Tamarin was than SpiderMonkey. Below the pics are the actual numbers.


SpiderMonkey 1.8
built on May 22 2008 at 13:31:57

3d: 347.4ms +/- 0.4%
cube: 126.6ms +/- 0.5%
morph: 118.0ms +/- 0.4%
raytrace: 102.8ms +/- 0.4%

access: 415.5ms +/- 1.5%
binary-trees: 43.3ms +/- 4.0%
fannkuch: 193.3ms +/- 1.4%
nbody: 125.1ms +/- 1.4%
nsieve: 53.8ms +/- 0.6%

bitops: 397.8ms +/- 0.4%
3bit-bits-in-byte: 49.6ms +/- 1.0%
bits-in-byte: 79.1ms +/- 0.3%
bitwise-and: 178.3ms +/- 0.4%
nsieve-bits: 90.8ms +/- 0.7%

controlflow: 35.1ms +/- 0.6%
recursive: 35.1ms +/- 0.6%

crypto: 169.9ms +/- 0.3%
aes: 66.1ms +/- 0.3%
md5: 51.9ms +/- 0.8%
sha1: 51.9ms +/- 0.4%

date: 269.3ms +/- 0.4%
format-tofte: 160.3ms +/- 0.3%
format-xparb: 109.0ms +/- 0.7%

math: 288.5ms +/- 0.3%
cordic: 128.9ms +/- 0.4%
partial-sums: 102.0ms +/- 0.3%
spectral-norm: 57.6ms +/- 0.6%

regexp: 251.9ms +/- 0.2%
dna: 251.9ms +/- 0.2%

string: 594.0ms +/- 0.3%
base64: 68.7ms +/- 0.5%
fasta: 148.7ms +/- 0.2%
tagcloud: 134.2ms +/- 0.8%
unpack-code: 169.4ms +/- 0.2%
validate-input: 73.0ms +/- 0.7%

Tamarin:
3d: 2035 ms
cube: 457 ms
morph: 608 ms
raytrace: 970 ms

access: 3548 ms
binary-trees: 363 ms
fannkuch: 1819 ms
nbody: 1047 ms
nsieve: 319 ms

bitops: 3929 ms
3bit-bits-in-byte: 275 ms
bits-in-byte: 371 ms
bitwise-and: 2454 ms
nsieve-bits: 829 ms

controlflow: 282 ms
recursive: 282 ms

crypto: 1440 ms
aes: 785 ms
md5: 322 ms
sha1: 333 ms

date: Fail
format-tofte: Reference Error #1065
format-xparb: Reference Error #1065

math: 2637 ms
cordic: 616 ms
partial-sum 1584 ms
spectral-norm: 437 ms

regexp: Fail
dna: ArgumentError: Error #1063

string: 56445 ms
base64: ReferenceError: Error #1069
fasta: 837 ms
tagcloud: ReferenceError: #1065
unpack-code: 50234 ms
validate-input: 5374 ms

Tamarin is slower. The worst part has to be playing with strings. Ouch.