Benchmarks between Clang and GCC about Ruby 1.9.3

Since Matz Ruby Implimentation ruby 1.9.3 has been released as preview release, I think that all Rubyists should abandon the ruby they are usually using, ruby 1.9.2, and should use 1.9.3 now.

Ruby 1.9.3 also supports clang instead of gcc inofficially. You can compile ruby with clang 3.0 and almost everything works(*1).

Building clang 3.0

There are two things you have to understand before you build clang.

• Clang is a subproject of LLVM. You cannot build clang independently to LLVM unless you are an LLVM specialist.
• LLVM build directly has to be not in LLVM source tree for some reason.

Otherwise it just fails building.

$ mkdir -p ~/src/llvm-git-build
$ cd ~/git
$ git clone http://llvm.org/git/llvm.git
$ cd tools
$ git clone http://llvm.org/git/clang.git
$ cd ~/src/llvm-git-build
$ ~/git/llvm/configure --prefix=`pwd`/local
$ make && make install

That will install llvm commands and clang in ~/src/llvm-git-build/local/bin directory.

$ ~/src/llvm-git-build/local/bin/clang -v
clang version 3.0 (http://llvm.org/git/clang.git d484040a5fc8d8d8a7a6a0239fc3b34486258181)
Target: x86_64-apple-darwin10.8.0
Thread model: posix

Building ruby 1.9.3 with clang

$ cd ~/git
$ git clone http://github.com/ruby/ruby.git ruby193
$ cd ruby193
$ git checkout ruby_1_9_3 origin/ruby_1_9_3

a.sh

export CC=~/src/llvm-git-build/local/bin/clang
autoconf && ./configure --with-out-ext=tk\* --with-arch=x86_64 --prefix=`pwd`/local --with-readline-dir=/usr/local --with-libyaml-include=/usr/local/include && make && make install

then

$ sh a.sh

Benchmark

$ local/bin/ruby benchmark/run.rb --ruby=local/bin/ruby

I ran on ruby made by clang and ruby made by gcc and compared them.

	gcc/clang	gcc [sec]	clang [sec]
loop_generator	1.55	1.187	0.768
vm_thread_pass_flood	1.38	0.52	0.377
vm1_const	1.23	1.449	1.175
so_random	1.15	1.191	1.04
so_k_nucleotide	1.11	0.02	0.018
vm2_case	1.10	0.338	0.307
vm2_regexp	1.07	1.8	1.683
loop_whileloop	1.07	0.677	0.634
vm_thread_pipe	1.07	1.659	1.556
so_count_words	1.06	0.442	0.418
loop_whileloop2	1.06	0.148	0.14
so_reverse_complement	1.06	0.019	0.018
vm1_swap	1.06	0.956	0.906
vm2_array	1.05	1.543	1.468
vm1_ivar	1.04	1.662	1.596
vm1_length	1.04	1.467	1.412
vm_thread_create_join	1.03	4.45	4.321
vm_thread_pass	1.02	2.308	2.258
loop_times	1.02	1.834	1.796
app_uri	1.01	1.362	1.346
io_select2	1.00	3.976	3.957
vm1_rescue	1.00	0.799	0.798
vm1_ivar_set	1.00	1.766	1.765
io_select3	1.00	0.03	0.03
vm1_ensure	1.00	0.736	0.738
io_select	1.00	3.503	3.513
vm_thread_alive_check1	1.00	0.327	0.328
so_mandelbrot	0.99	6.776	6.817
vm1_not	0.99	0.958	0.965
vm2_poly_method_ov	0.98	0.422	0.43
io_file_write	0.98	1.78	1.815
so_nbody	0.97	4.861	5.012
vm2_eval	0.97	28.817	29.772
so_partial_sums	0.97	6.003	6.211
so_concatenate	0.96	5.322	5.536
so_exception	0.96	1.653	1.723
loop_for	0.95	1.865	1.96
vm1_neq	0.95	1.206	1.269
io_file_read	0.95	3.764	3.965
so_sieve	0.94	1.06	1.13
vm3_gc	0.93	1.593	1.712
so_pidigits	0.92	0.905	0.982
so_spectralnorm	0.92	3.911	4.265
vm2_proc	0.91	0.858	0.941
so_nested_loop	0.91	1.415	1.554
so_fannkuch	0.89	2.321	2.6
vm3_clearmethodcache	0.89	0.683	0.767
so_array	0.89	1.859	2.098
vm2_defined_method	0.88	4.449	5.034
io_file_create	0.88	4.266	4.831
vm1_simplereturn	0.88	2.034	2.307
vm_thread_mutex3	0.88	1.894	2.164
vm2_send	0.87	0.494	0.567
vm2_unif1	0.87	0.419	0.484
so_matrix	0.86	1.173	1.357
so_nsieve_bits	0.86	4.002	4.637
app_tak	0.86	1.133	1.316
so_lists	0.85	1.439	1.698
app_tarai	0.85	0.9	1.062
vm1_block	0.85	2.818	3.331
so_ackermann	0.82	0.891	1.082
vm_thread_mutex1	0.82	1.145	1.393
so_nsieve	0.80	3.188	3.967
vm2_super	0.80	0.68	0.852
so_binary_trees	0.79	0.492	0.62
app_strconcat	0.79	1.93	2.444
vm2_mutex	0.79	1.546	1.968
vm2_zsuper	0.78	0.682	0.88
vm2_poly_method	0.77	2.654	3.444
app_factorial	0.76	1.317	1.734
app_erb	0.76	1.834	2.426
so_meteor_contest	0.75	4.953	6.626
so_object	0.74	1.047	1.416
vm2_method	0.71	2.174	3.049
app_pentomino	0.70	25.525	36.325
app_fib	0.70	0.812	1.168
so_fasta	0.67	2.866	4.302
app_answer	0.66	0.075	0.113
app_mandelbrot	0.65	2.422	3.743
app_raise	0.49	0.861	1.743
vm_thread_mutex2	0.41	1.228	2.977

Considering all the results gcc-made ruby is faster than clang-made ruby but not in all the benchmarks. Hopefully the results must be an interesting example for some people.

Footnote and references

• *1: make test-all shows that there is one issue in Fiber with the ruby made by clang.
• http://ujihisa.blogspot.com/2011/01/how-to-build-mri-n-times-faster.html comparison between clang and gcc about time to build ruby