Igor Alexandrov

Programming and something else…

Yet Another Ruby Shootout

| Comments

The beginning of the november brought us interesting news from Ruby World: ruby-2.0.0-preview1 was released and also rubinius-2.0rc1 was tagged on GitHub.

I decided to take these two distributions and compare them to current industry default ruby-1.9.3-p286.

I used Antonio Cangiano’s ruby-benchmark-suite. All tests run on:

  • OS: OSX 10.8.2
  • CPU: 2.2GHz i7
  • RAM: 8Gb 1333 MHz DDR3
  • HHD: Seagate Momentus XT, 2.5”, SATA 3Gb/s, 7200 rpm, hybrid

I want to try it too!

To install all distributions I used latest RVM.

1
  saturn:~$ rvm install ruby-1.9.3-p286

Don’t forget to change compilation options before installing ruby-2.0.0, or you can get errors during compilation.

1
2
  saturn:~$ export CC=clang
  saturn:~$ rvm install ruby-2.0.0-preview1
1
  saturn:~$ rvm install rbx-2.0.0rc1

After all is done, you should get something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
  saturn:~$ rvm list

  rvm rubies

      rbx-2.0.0rc1 [ x86_64 ]
      rbx-head [ x86_64 ]
      ruby-1.9.2-p320 [ x86_64 ]
  =*  ruby-1.9.3-p194 [ x86_64 ]
      ruby-1.9.3-p286 [ x86_64 ]
      ruby-2.0.0-preview1 [ x86_64 ]

  # => - current
  # =* - current && default
  #  * - default

  saturn:~$

Then to run benchmarks you should download benchmark suite from GitHub, switch RVM to ruby that you want to test, run rake task and wait for about 20 minutes.

1
2
3
  saturn:~/workspace/ruby-benchmark-suite (master)$ rake bench
  Running Ruby Benchmark Suite
    Writing report to results/rbs/RBS-ruby-121105.122343.yaml

Results

Total chart:

Mean chart:

Total table:

Benchmark FileInput Sizeruby-1.9.3-p286ruby-2.0.0-preview1rbx-2.0.0-rc1
macro-benchmarks/bm_gzip.rb1005.67295.76717.3867
macro-benchmarks/bm_hilbert_matrix.rb100.00240.00240.0183
macro-benchmarks/bm_hilbert_matrix.rb200.02980.02860.1078
macro-benchmarks/bm_hilbert_matrix.rb300.14650.13940.4026
macro-benchmarks/bm_hilbert_matrix.rb400.46960.47261.3414
macro-benchmarks/bm_hilbert_matrix.rb501.04031.01063.5903
macro-benchmarks/bm_hilbert_matrix.rb602.43132.42658.586
macro-benchmarks/bm_mpart.rb3000.33360.29770.2418
macro-benchmarks/bm_norvig_spelling.rb504.47514.828211.5449
macro-benchmarks/bm_parse_log.rb1000.59890.68370.5616
macro-benchmarks/bm_rcs.rb1000.33280.24110.8911
macro-benchmarks/bm_sudoku.rb11.74671.6043.4747
micro-benchmarks/bm_app_factorial.rb50000.01550.0154SystemStackError
micro-benchmarks/bm_app_fib.rb300.19990.16630.2051
micro-benchmarks/bm_app_fib.rb352.19881.84582.2601
micro-benchmarks/bm_app_mandelbrot.rb10.28850.20630.6621
micro-benchmarks/bm_app_pentomino.rb125.966224.224249.4342
micro-benchmarks/bm_app_tak.rb70.16120.14270.1872
micro-benchmarks/bm_app_tak.rb80.45870.40980.5389
micro-benchmarks/bm_app_tak.rb91.20811.08351.4194
micro-benchmarks/bm_app_tarai.rb30.52180.46340.6741
micro-benchmarks/bm_app_tarai.rb40.63340.56120.8111
micro-benchmarks/bm_app_tarai.rb50.76440.67830.9848
micro-benchmarks/bm_binary_trees.rb111.216410.227319.1229
micro-benchmarks/bm_cal.rb5000.05090.04950.6941
micro-benchmarks/bm_count_multithreaded.rb10.00580.00490.0027
micro-benchmarks/bm_count_multithreaded.rb20.01140.00990.0027
micro-benchmarks/bm_count_multithreaded.rb40.02350.01930.0047
micro-benchmarks/bm_count_multithreaded.rb80.0460.03880.0059
micro-benchmarks/bm_count_multithreaded.rb160.09380.07860.012
micro-benchmarks/bm_count_shared_thread.rb10.05680.04690.0242
micro-benchmarks/bm_count_shared_thread.rb20.05680.04780.0122
micro-benchmarks/bm_count_shared_thread.rb40.05730.04780.007
micro-benchmarks/bm_count_shared_thread.rb80.05760.04750.0076
micro-benchmarks/bm_count_shared_thread.rb160.05860.0490.0078
micro-benchmarks/bm_dirp.rb100001.59241.58371.0284
micro-benchmarks/bm_eval.rb10000006.72297.164338.4595
micro-benchmarks/bm_fannkuch.rb60.0020.0020.0042
micro-benchmarks/bm_fannkuch.rb80.15780.14180.0947
micro-benchmarks/bm_fannkuch.rb1020.203118.131911.7026
micro-benchmarks/bm_fasta.rb100000011.63759.289416.156
micro-benchmarks/bm_fiber_ring.rb100.00020.0002LoadError
micro-benchmarks/bm_fiber_ring.rb1000.01310.0119LoadError
micro-benchmarks/bm_fiber_ring.rb10001.29281.3856LoadError
micro-benchmarks/bm_fractal.rb51.26850.6370.7773
micro-benchmarks/bm_gc_array.rb132.945739.136211.5338
micro-benchmarks/bm_gc_mb.rb5000000.13940.14820.3714
micro-benchmarks/bm_gc_mb.rb10000000.25870.28540.8044
micro-benchmarks/bm_gc_mb.rb30000000.87850.99173.5867
micro-benchmarks/bm_gc_string.rb13.12972.88844.0832
micro-benchmarks/bm_knucleotide.rb10.64910.64071.6754
micro-benchmarks/bm_list.rb10000.02430.02440.0714
micro-benchmarks/bm_list.rb100001.77961.78277.1215
micro-benchmarks/bm_lucas_lehmer.rb96893.30633.32642.5413
micro-benchmarks/bm_lucas_lehmer.rb99413.51283.57692.7151
micro-benchmarks/bm_lucas_lehmer.rb112134.92925.02263.7476
micro-benchmarks/bm_lucas_lehmer.rb1993726.95927.340218.3166
micro-benchmarks/bm_mandelbrot.rb115.53818.243711.6549
micro-benchmarks/bm_mbari_bogus1.rb10.02690.01760.0333
micro-benchmarks/bm_mergesort.rb10.69820.67711.2037
micro-benchmarks/bm_mergesort_hongli.rb30001.08720.81531.0663
micro-benchmarks/bm_meteor_contest.rb15.35464.997616.7916
micro-benchmarks/bm_monte_carlo_pi.rb100000004.92912.88093.525
micro-benchmarks/bm_nbody.rb1000002.79181.11221.5627
micro-benchmarks/bm_nsieve.rb93.26412.65142.877
micro-benchmarks/bm_nsieve_bits.rb82.0621.6684.1123
micro-benchmarks/bm_observ.rb1000000.45090.30560.2562
micro-benchmarks/bm_open_many_files.rb500000.50240.53130.5217
micro-benchmarks/bm_partial_sums.rb25000005.96422.73965.4341
micro-benchmarks/bm_pathname.rb10045.715745.9133SignalException
micro-benchmarks/bm_pi.rb10000.0230.02420.0241
micro-benchmarks/bm_pi.rb100001.12231.15211.5111
micro-benchmarks/bm_primes.rb30000.01150.010.7166
micro-benchmarks/bm_primes.rb300000.12260.1042SignalException
micro-benchmarks/bm_primes.rb3000001.44491.3581SignalException
micro-benchmarks/bm_quicksort.rb11.32971.21292.2357
micro-benchmarks/bm_read_large.rb1001.80921.763835.9596
micro-benchmarks/bm_regex_dna.rb203.17352.50763.8788
micro-benchmarks/bm_reverse_compliment.rb13.87214.086422.1258
micro-benchmarks/bm_simple_connect.rb10.00020.00020.0013
micro-benchmarks/bm_simple_connect.rb1000.00760.00850.0163
micro-benchmarks/bm_simple_connect.rb5000.05270.06140.0961
micro-benchmarks/bm_simple_server.rb10.00080.00050.0019
micro-benchmarks/bm_simple_server.rb1000.00220.0020.0035
micro-benchmarks/bm_simple_server.rb1000001.3671.43221.9087
micro-benchmarks/bm_so_ackermann.rb70.06050.05710.067
micro-benchmarks/bm_so_ackermann.rb90.97390.913SystemStackError
micro-benchmarks/bm_so_array.rb90002.07031.44281.5398
micro-benchmarks/bm_so_count_words.rb1002.53262.62230.2373
micro-benchmarks/bm_so_exception.rb5000008.05531.02772.585
micro-benchmarks/bm_so_lists.rb10004.72054.64185.3836
micro-benchmarks/bm_so_lists_small.rb10000.96221.17811.009
micro-benchmarks/bm_so_matrix.rb600.52550.38560.2736
micro-benchmarks/bm_so_object.rb5000000.46630.35210.1415
micro-benchmarks/bm_so_object.rb10000000.92330.70750.2802
micro-benchmarks/bm_so_object.rb15000001.39961.06550.4182
micro-benchmarks/bm_so_sieve.rb40009.31547.22895.606
micro-benchmarks/bm_socket_transfer_1mb.rb100000.6190.59840.1912
micro-benchmarks/bm_socket_transfer_1mb.rb10000000.62560.60280.1928
micro-benchmarks/bm_socket_transfer_1mb_noblock.rb100000.53990.52130.2013
micro-benchmarks/bm_socket_transfer_1mb_noblock.rb10000000.52760.51950.1976
micro-benchmarks/bm_spectral_norm.rb1000.16910.12480.1698
micro-benchmarks/bm_string_concat.rb100000001.58251.6414Terminated
micro-benchmarks/bm_sum_file.rb1003.09622.808640.081
micro-benchmarks/bm_word_anagrams.rb13.06622.89513.4752
micro-benchmarks/bm_write_large.rb1000.60960.60211.3061
rdoc/bm_rdoc_against_itself_darkfish.rb15.7458.69538.2666
rdoc/bm_rdoc_against_itself_ri.rb15.014SystemExit7.6531

As you can see, ruby-2.0.0-preview1 shows really good results in most of the tests. In nearly all tests it shows results that are the same or better then in 1.9.3-p286. But some results are significantly better:

  • micro-benchmarks/bm_fractal.rb 1.9.3p286 – 1.2685 sec, 2.0.0-preview1 – 0.637 sec
  • micro-benchmarks/bm_mandelbrot.rb 1.9.3p286 – 15.5381 sec, 2.0.0-preview1 – 8.2437 sec
  • micro-benchmarks/bm_nbody.rb 1.9.3p286 – 2.7918 sec, 2.0.0-preview1 – 1.1122 sec
  • micro-benchmarks/bm_partial_sums.rb 1.9.3p286 – 5.9642 sec, 2.0.0-preview1 – 2.7396 sec (this example uses rather slow upto method, as you can see, it is much faster in 2.0.0)
  • micro-benchmarks/bm_regex_dna.rb 1.9.3p286 – 3.1735 sec, 2.0.0-preview1 – 2.5076 sec (new regex engine Onigmo does its job well)
  • micro-benchmarks/bm_so_exception.rb 1.9.3p286 – 8.0553 sec, 2.0.0-preview1 – 1.0277 sec (8x faster exception handling)

In all tests with multithreading Rubinius is much better then MRI Ruby implementations (of course this is usual situation). But the most interesting for me result is that garbage collection in Rubinius works 4x faster then in MRI Ruby:

  • micro-benchmarks/bm_gc_array.rb rbx-2.0.0-rc1 – 11.5338 sec, 2.0.0-preview1 – 39.1362 sec

Also Rubinius is better in network comminucation:

  • micro-benchmarks/bm_socket_transfer_1mb.rb rbx-2.0.0-rc1 – 0.1912 sec, 2.0.0-preview1 – 0.5984 sec
  • micro-benchmarks/bm_socket_transfer_1mb_noblock.rb rbx-2.0.0-rc1 – 0.2013 sec, 2.0.0-preview1 – 0.5213 sec

Is Ruby 2.0 Enumerator#lazy really so cool?

When I saw this feature for the first time, I was really excited. So I decided to test its performance too and now I am a bit upset.

ruby-benchmark.rb
1
2
3
4
5
6
7
8
9
10
11
  require 'benchmark'

  array = 1.upto(1000000).to_a
  Benchmark.bm do |x|
    x.report {
      array.select{ |i| i%2 == 0 }.map{ |i| i*2 }.to_a
    }
    x.report {
      array.lazy.select{ |i| i%2 == 0 }.map{ |i| i*2 }.to_a
    }
  end

And here are results.

1
2
3
4
5
  saturn:~/workspace$ ruby ruby-benchmark.rb
         user     system      total        real
     0.140000   0.010000   0.150000 (  0.149466)
     1.560000   0.000000   1.560000 (  1.575729)
  saturn:~/workspace$

Yes, it is 10x slower! I really don’t understand how this was accepted in ruby trunk and why is it so slow in preview1.

The only thing that I really understood once again – every feature should be tested before it can be used in production code, even this feature was published on www.ruby-lang.org.

PS1

Added mean values chart.

PS2

Added jruby-1.7.0 (on java 7) results.

Comments