TraceMonkey vs SquirrelFish
With the announcement of TraceMonkey, we saw that SpiderMonkey got a huge improvement. However, in previous benchmarks, SpiderMonkey was still slower than SquirrelFish. So, how does Trace Monkey compare versus SquirrelFish:
Overall, if you aggregate the time for each test, TraceMonkey is about 15% faster. If you do the aggregate speedup, which is the speedup for each test divided by the number of tests, it is about 2.4 x faster. Sometimes TraceMonkey is slower, but give them more than 60 days to hash it out, and I'm sure it'll get faster.
Notes: SquirrelFish and TraceMonkey were with release builds. Tests run on a MacBook Pro 2.4 ghz, 4 gigs of ram, OS X Leopard 10.5.4. TraceMonkey was just pulled about an hour ago. SquirrelFish was last nights SVN.
Good Job Guys. More info on TraceMonkey at Andreas Gal's Blog, Mike Shaver, David Anderson, and for a real visual demo, checkout Mike Schroepfer's blog.
9 comments:
Beware that SquirrelFish doesn't compile its VM bytecodes to native code yet (hint: remember Apple sponsors LLVM), and many of the low-hanging-fruit optimizations haven't been implemented at all, while TraceMonkey is already a massive inliner of specialized, native code!! So... The game of leapfrogging has just begun ;)
Haha that is true, however TraceMonkey still has a lot of low hanging fruit as well :). Either way it is good for the web, and it is good to have competition. Should be a fun interesting ride.
Just compiling VM bytecodes for a language like JavaScript doesn't buy you much since you'll still have to emit code that handles all possible execution paths for all possible types. The trick is to reduce run-time type checking by discovering traces with stable types.
Michael, I'm pretty aware of your involvement with Andreas Gal and UC Irvine in general, so I won't dare to dispute your point.
On the other hand, you don't seem to fully realise the potential of LLVM. I suggest that you check it out, and take a peek at this page for some inspiration regarding how dynamic languages are targeting LLVM (the description of the FAUST language is particularly compelling considering the point you brought: http://llvm.org/ProjectsWithLLVM/#faust)
Cheers,
A.
I did some benchmarking with a nightly build of FF3 vs. Safari 4.0 vs. the IE 8 beta. This was in a Windows XP VM running under OSX. For benchmarks I used SunSpider and Dromaeo. With Tracemonkey enabled, FF3 hung on one of the SunSpider tests. This is a known issue. IE8 (and IE7) failed to complete Dromaeo. Safari 4 performed better on Dromaeo than did FF3 with Tracemonkey enabled. It bears noting that the Dromaeo test suite is put out by Mozilla, so if anything it should favor Firefox, as opposed to SunSpider which was built by the Webkit folks. Also, note that IE8 fixes the pathologically bad behavior seen in IE6/IE7 on certain of the string-related tests in SunSpider.
Links to my results:
IE 8 Beta 2 (8.0.6001.18241)
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B341,320,330,330,330%5D,%223d-morph%22:%5B290,260,271,261,260%5D,%223d-raytrace%22:%5B370,371,324,370,305%5D,%22access-binary-trees%22:%5B261,250,250,251,240%5D,%22access-fannkuch%22:%5B701,691,711,711,711%5D,%22access-nbody%22:%5B380,381,381,370,380%5D,%22access-nsieve%22:%5B180,190,180,180,191%5D,%22bitops-3bit-bits-in-byte%22:%5B170,180,180,181,180%5D,%22bitops-bits-in-byte%22:%5B241,251,261,250,250%5D,%22bitops-bitwise-and%22:%5B441,440,430,441,441%5D,%22bitops-nsieve-bits%22:%5B280,281,291,290,280%5D,%22controlflow-recursive%22:%5B200,190,190,180,191%5D,%22crypto-aes%22:%5B280,280,280,281,280%5D,%22crypto-md5%22:%5B150,150,151,150,160%5D,%22crypto-sha1%22:%5B160,161,160,160,160%5D,%22date-format-tofte%22:%5B360,350,350,351,351%5D,%22date-format-xparb%22:%5B311,321,311,310,310%5D,%22math-cordic%22:%5B380,380,380,381,381%5D,%22math-partial-sums%22:%5B261,250,261,250,250%5D,%22math-spectral-norm%22:%5B210,201,210,210,211%5D,%22regexp-dna%22:%5B380,390,400,391,391%5D,%22string-base64%22:%5B1352,1362,1362,1362,1392%5D,%22string-fasta%22:%5B361,351,351,351,360%5D,%22string-tagcloud%22:%5B280,281,291,281,290%5D,%22string-unpack-code%22:%5B280,290,281,290,290%5D,%22string-validate-input%22:%5B301,291,300,290,301%5D%7D
(failed on dromaeo)
Firefox 3.1a2pre (8/27) w/ JIT
(failed on sunspider)
http://dromaeo.com/?id=19754
Firefox 3.1a2pre (8/27) w/o JIT
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B131,126,128,128,129%5D,%223d-morph%22:%5B131,106,119,104,103%5D,%223d-raytrace%22:%5B155,160,120,178,158%5D,%22access-binary-trees%22:%5B49,51,52,71,50%5D,%22access-fannkuch%22:%5B173,175,180,174,175%5D,%22access-nbody%22:%5B153,147,140,140,149%5D,%22access-nsieve%22:%5B48,49,49,59,56%5D,%22bitops-3bit-bits-in-byte%22:%5B70,69,68,68,68%5D,%22bitops-bits-in-byte%22:%5B75,75,76,76,75%5D,%22bitops-bitwise-and%22:%5B92,70,69,70,69%5D,%22bitops-nsieve-bits%22:%5B92,91,92,92,93%5D,%22controlflow-recursive%22:%5B39,38,40,38,39%5D,%22crypto-aes%22:%5B66,67,70,68,71%5D,%22crypto-md5%22:%5B56,57,56,56,57%5D,%22crypto-sha1%22:%5B61,63,61,70,64%5D,%22date-format-tofte%22:%5B129,127,129,130,129%5D,%22date-format-xparb%22:%5B97,102,99,98,98%5D,%22math-cordic%22:%5B146,148,144,147,146%5D,%22math-partial-sums%22:%5B103,106,128,111,104%5D,%22math-spectral-norm%22:%5B74,75,77,74,72%5D,%22regexp-dna%22:%5B228,229,337,224,230%5D,%22string-base64%22:%5B53,56,56,56,55%5D,%22string-fasta%22:%5B131,134,131,133,133%5D,%22string-tagcloud%22:%5B125,135,128,140,133%5D,%22string-unpack-code%22:%5B237,254,245,255,241%5D,%22string-validate-input%22:%5B86,106,90,99,95%5D%7D
http://dromaeo.com/?id=19751
Safari 4.0 (528.1.1)
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B110,101,110,100,100%5D,%223d-morph%22:%5B90,91,80,100,110%5D,%223d-raytrace%22:%5B110,121,120,120,120%5D,%22access-binary-trees%22:%5B40,41,40,50,50%5D,%22access-fannkuch%22:%5B80,81,90,90,80%5D,%22access-nbody%22:%5B160,171,170,160,150%5D,%22access-nsieve%22:%5B20,21,30,30,30%5D,%22bitops-3bit-bits-in-byte%22:%5B10,10,20,10,20%5D,%22bitops-bits-in-byte%22:%5B40,40,30,30,40%5D,%22bitops-bitwise-and%22:%5B70,80,80,61,71%5D,%22bitops-nsieve-bits%22:%5B70,60,51,51,61%5D,%22controlflow-recursive%22:%5B20,20,21,20,10%5D,%22crypto-aes%22:%5B40,30,40,40,40%5D,%22crypto-md5%22:%5B50,50,40,50,50%5D,%22crypto-sha1%22:%5B50,50,40,40,40%5D,%22date-format-tofte%22:%5B120,110,110,110,111%5D,%22date-format-xparb%22:%5B80,80,80,80,80%5D,%22math-cordic%22:%5B80,80,80,80,81%5D,%22math-partial-sums%22:%5B130,130,140,140,141%5D,%22math-spectral-norm%22:%5B40,50,50,50,50%5D,%22regexp-dna%22:%5B311,300,300,310,300%5D,%22string-base64%22:%5B80,80,80,70,80%5D,%22string-fasta%22:%5B90,100,90,100,90%5D,%22string-tagcloud%22:%5B130,131,130,130,130%5D,%22string-unpack-code%22:%5B140,140,140,150,131%5D,%22string-validate-input%22:%5B131,130,130,120,120%5D%7D
http://dromaeo.com/?id=19747
IE 7 (7.0.5730.13)
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B381,381,380,381,381%5D,%223d-morph%22:%5B451,451,460,451,441%5D,%223d-raytrace%22:%5B561,551,541,541,530%5D,%22access-binary-trees%22:%5B521,631,701,741,732%5D,%22access-fannkuch%22:%5B792,781,792,781,791%5D,%22access-nbody%22:%5B390,390,390,390,390%5D,%22access-nsieve%22:%5B331,351,341,341,351%5D,%22bitops-3bit-bits-in-byte%22:%5B431,420,430,430,430%5D,%22bitops-bits-in-byte%22:%5B461,461,461,461,461%5D,%22bitops-bitwise-and%22:%5B500,511,501,501,481%5D,%22bitops-nsieve-bits%22:%5B381,370,371,370,370%5D,%22controlflow-recursive%22:%5B521,531,521,521,521%5D,%22crypto-aes%22:%5B390,401,400,391,401%5D,%22crypto-md5%22:%5B351,350,341,350,350%5D,%22crypto-sha1%22:%5B350,351,350,351,351%5D,%22date-format-tofte%22:%5B451,461,451,450,460%5D,%22date-format-xparb%22:%5B461,460,461,460,460%5D,%22math-cordic%22:%5B510,521,511,511,511%5D,%22math-partial-sums%22:%5B321,330,331,330,320%5D,%22math-spectral-norm%22:%5B410,401,401,411,401%5D,%22regexp-dna%22:%5B421,420,421,430,430%5D,%22string-base64%22:%5B12161,12288,12328,12378,13700%5D,%22string-fasta%22:%5B481,481,491,490,481%5D,%22string-tagcloud%22:%5B2404,2473,2494,2483,2514%5D,%22string-unpack-code%22:%5B530,530,531,531,530%5D,%22string-validate-input%22:%5B21822,22783,22052,22196,22006%5D%7Dhttp://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B381,381,380,381,381%5D,%223d-morph%22:%5B451,451,460,451,441%5D,%223d-raytrace%22:%5B561,551,541,541,530%5D,%22access-binary-trees%22:%5B521,631,701,741,732%5D,%22access-fannkuch%22:%5B792,781,792,781,791%5D,%22access-nbody%22:%5B390,390,390,390,390%5D,%22access-nsieve%22:%5B331,351,341,341,351%5D,%22bitops-3bit-bits-in-byte%22:%5B431,420,430,430,430%5D,%22bitops-bits-in-byte%22:%5B461,461,461,461,461%5D,%22bitops-bitwise-and%22:%5B500,511,501,501,481%5D,%22bitops-nsieve-bits%22:%5B381,370,371,370,370%5D,%22controlflow-recursive%22:%5B521,531,521,521,521%5D,%22crypto-aes%22:%5B390,401,400,391,401%5D,%22crypto-md5%22:%5B351,350,341,350,350%5D,%22crypto-sha1%22:%5B350,351,350,351,351%5D,%22date-format-tofte%22:%5B451,461,451,450,460%5D,%22date-format-xparb%22:%5B461,460,461,460,460%5D,%22math-cordic%22:%5B510,521,511,511,511%5D,%22math-partial-sums%22:%5B321,330,331,330,320%5D,%22math-spectral-norm%22:%5B410,401,401,411,401%5D,%22regexp-dna%22:%5B421,420,421,430,430%5D,%22string-base64%22:%5B12161,12288,12328,12378,13700%5D,%22string-fasta%22:%5B481,481,491,490,481%5D,%22string-tagcloud%22:%5B2404,2473,2494,2483,2514%5D,%22string-unpack-code%22:%5B530,530,531,531,530%5D,%22string-validate-input%22:%5B21822,22783,22052,22196,22006%5D%7D
(failed on dromaeo)
This has nothing to do with Javascript, but I also ran each of them through Acid3. Safari 4.0 got a perfect 100. FF 3.1 scored an 85, but it still looked pretty mangled.
Bah. The long webkit URLs didn't come through. Here are tinyurl versions:
IE 8 Beta 2 (8.0.6001.18241)
http://tinyurl.com/5lqjnx
Firefox 3.1a2pre (8/27) w/o JIT
http://tinyurl.com/6pre9d
Safari 4.0 (528.1.1)
http://tinyurl.com/5m4hpl
IE 7 (7.0.5730.13)
http://tinyurl.com/5zchhp
Aggregate speedup? Have you ever heard about geometric mean?
geometric mean? How is that at all relevant here? These opperations are added, not multiplied.
How not to lie with statistics: the correct way to summarize benchmark results
Post a Comment