Benchmark Driver Designed for Ruby 3x3

Project Summary

Here is a copy of the project summary which I sent to Ruby Association:

What’s benchmark_driver.gem?

benchmark_driver.gem is originally designed as a successor of benchmark/driver.rb in Ruby language repository, which can measure a Ruby script performance accurately by subtracting time taken for while loop from a looped script to measure, and which can benchmark multiple Ruby binaries at the same time and compare them.

Basic usage: Ruby interface

benchmark_driver.gem has 2 benchmark definition formats: Ruby and YAML.
Ruby interface is designed for a usual Ruby benchmark or a use case to generate a benchmark target script dynamically, while YAML format is good for minimizing effort to write benchmark definitions and for flexibly changing parameters on the fly without modifying code.

require 'benchmark/ips'class Array
alias_method :blank?, :empty?
end
Benchmark.ips do |x|
array = []
x.report('Array#empty?') { array.empty? }
x.report('Array#blank?') { array.blank? }
x.compare!
end
Warming up --------------------------------------
Array#empty? 524.659k i/100ms
Array#blank? 495.794k i/100ms
Calculating -------------------------------------
Array#empty? 15.497M (± 2.5%) i/s - 77.650M in 5.013969s
Array#blank? 14.171M (± 2.1%) i/s - 70.899M in 5.005282s
Comparison:
Array#empty?: 15497274.4 i/s
Array#blank?: 14171303.0 i/s - 1.09x slower
require 'benchmark_driver'Benchmark.driver do |x|
x.prelude %{
class Array
alias_method :blank?, :empty?
end
array = []
}
x.report 'Array#empty?', %{ array.empty? }
x.report 'Array#blank?', %{ array.blank? }
end
Warming up --------------------------------------
Array#empty? 56.340M i/s
Array#blank? 42.795M i/s
Calculating -------------------------------------
Array#empty? 181.135M i/s - 169.019M times in 0.933111s (5.52ns/i, 23clocks/i)
Array#blank? 99.275M i/s - 128.386M times in 1.293235s (10.07ns/i, 44clocks/i)
Comparison:
Array#empty?: 181134991.8 i/s
Array#blank?: 99275008.9 i/s - 1.82x slower
  • Overhead of calling a block has large overhead, compared to just running a part of while loop. benchmark_driver.gem takes a benchmark definition as string to dynamically generate such a loop script, instead of taking a script as a block.
  • As I said before, it subtracts while loop overhead.

Comparing multiple Ruby binaries

Another reason why benchmark_driver.gem is good for measuring Ruby’s performance is that it can run any Ruby binaries at the same time and compare the benchmark results.

require 'benchmark_driver'Benchmark.driver do |x|
x.prelude %{
def script
i = 0
while i < 1000_000
i += 1
end
i
end
}
x.report 'while', %{ script }
x.loop_count 2000
x.rbenv(
'2.0.0::2.0.0-p0',
'2.5.0',
'2.6.0-dev',
'2.6.0-dev+JIT::2.6.0-dev,--jit',
)
x.verbose
end
2.0.0: ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
2.5.0: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
2.6.0-dev+JIT: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
Calculating -------------------------------------
2.0.0 2.5.0 2.6.0-dev 2.6.0-dev+JIT
while 77.952 80.325 87.239 491.907 i/s - 2.000k times in 25.656691s 24.898879s 22.925498s 4.065807s
Comparison:
while
2.6.0-dev+JIT: 491.9 i/s
2.6.0-dev: 87.2 i/s - 5.64x slower
2.5.0: 80.3 i/s - 6.12x slower
2.0.0: 78.0 i/s - 6.31x slower

Advanced usage: YAML and CLI

benchmark_driver.gem has CLI like this (descriptions are omitted because the width of medium is too small...):

$ benchmark-driver -h
Usage: benchmark-driver [options] [YAML]
-r, --runner [TYPE]
-o, --output [TYPE]
-e, --executables [EXECS]
--rbenv [VERSIONS]
--repeat-count [NUM]
--bundler
--filter [REGEXP]
--verbose [LEVEL]
--run-duration [SECONDS]
prelude: |
large_a = "Hellooooooooooooooooooooooooooooooooooooooooooooooooooo"
large_b = "Wooooooooooooooooooooooooooooooooooooooooooooooooooorld"
small_a = "Hello"
small_b = "World"
benchmark:
large: '"#{large_a}, #{large_b}!"'
small: '"#{small_a}, #{small_b}!"'
$ benchmark-driver benchmark.yml --rbenv '2.4.3;2.5.0'
Warming up --------------------------------------
large 3.693M i/s
small 9.913M i/s
Calculating -------------------------------------
2.4.3 2.5.0
large 3.895M 5.485M i/s - 11.079M times in 2.844249s 2.019943s
small 11.755M 11.103M i/s - 29.740M times in 2.529966s 2.678612s
Comparison:
large
2.5.0: 5484705.8 i/s
2.4.3: 3895155.8 i/s - 1.41x slower
small
2.4.3: 11755264.9 i/s
2.5.0: 11102923.0 i/s - 1.06x slower

Plugin System

You can customize metrics to be measured and a format for output as you like,
and build your own versions if you want.

benchmark-driver.github.io

Ruby Core

Ruby Method

MJIT

Optcarrot

Future works

Outside the scope which was originally proposed for this project, we have several things to work on to make the best use of the outcome.

Sophisticating plugin interface

Right now the interface between Runner and Output is like this:

module BenchmarkDriver
Metrics = ::BenchmarkDriver::Struct.new(
:value, # @param [Float]
:executable, # @param [BenchmarkDriver::Config::Executable]
:duration, # @param [Float,nil]
)
Metrics::Type = ::BenchmarkDriver::Struct.new(
:unit, # @param [String]
:larger_better, # @param [TrueClass,FalseClass]
:worse_word, # @param [String]
defaults: { larger_better: true, worse_word: 'slower' },
)
end

Add more benchmarks to benchmark-driver.github.io

Only measuring micro benchmarks and NES emulator performance sounds not enough to achieve Ruby 3x3. But adding benchmarks is a little challenging because it’s already taking a lot of time to finish running all benchmarks for one revision. Also, installing many gems for each revision will trigger a disk consumption problem if we keep many Ruby revisions for future benchmark addition.

Integrating derailed_benchmarks.gem

You may want to use your Rails application for a benchmark test case for Ruby 3x3. If I integrate derailed_benchmarks.gem with benchmark_driver.gem, we may be able to compare Ruby’s performance using the Rails application easily.

Conclusion

benchmark_driver.gem is good for many use cases of Ruby's benchmark.
Please try it and give me a feedback to achieve Ruby 3x3.

Acknowledgements

The success of this project would be impossible without help from Ruby Association. Especially Koichi Sasada, a mentor of this project, gave me many good ideas to improve it. Thank you so much.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store