Benchmark Driver Designed for Ruby 3x3

Project Summary

What’s benchmark_driver.gem?

Basic usage: Ruby interface

require 'benchmark/ips'class Array
alias_method :blank?, :empty?
end
Benchmark.ips do |x|
array = []
x.report('Array#empty?') { array.empty? }
x.report('Array#blank?') { array.blank? }
x.compare!
end
Warming up --------------------------------------
Array#empty? 524.659k i/100ms
Array#blank? 495.794k i/100ms
Calculating -------------------------------------
Array#empty? 15.497M (± 2.5%) i/s - 77.650M in 5.013969s
Array#blank? 14.171M (± 2.1%) i/s - 70.899M in 5.005282s
Comparison:
Array#empty?: 15497274.4 i/s
Array#blank?: 14171303.0 i/s - 1.09x slower
require 'benchmark_driver'Benchmark.driver do |x|
x.prelude %{
class Array
alias_method :blank?, :empty?
end
array = []
}
x.report 'Array#empty?', %{ array.empty? }
x.report 'Array#blank?', %{ array.blank? }
end
Warming up --------------------------------------
Array#empty? 56.340M i/s
Array#blank? 42.795M i/s
Calculating -------------------------------------
Array#empty? 181.135M i/s - 169.019M times in 0.933111s (5.52ns/i, 23clocks/i)
Array#blank? 99.275M i/s - 128.386M times in 1.293235s (10.07ns/i, 44clocks/i)
Comparison:
Array#empty?: 181134991.8 i/s
Array#blank?: 99275008.9 i/s - 1.82x slower
  • Overhead of calling a block has large overhead, compared to just running a part of while loop. benchmark_driver.gem takes a benchmark definition as string to dynamically generate such a loop script, instead of taking a script as a block.
  • As I said before, it subtracts while loop overhead.

Comparing multiple Ruby binaries

require 'benchmark_driver'Benchmark.driver do |x|
x.prelude %{
def script
i = 0
while i < 1000_000
i += 1
end
i
end
}
x.report 'while', %{ script }
x.loop_count 2000
x.rbenv(
'2.0.0::2.0.0-p0',
'2.5.0',
'2.6.0-dev',
'2.6.0-dev+JIT::2.6.0-dev,--jit',
)
x.verbose
end
2.0.0: ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
2.5.0: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
2.6.0-dev+JIT: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
Calculating -------------------------------------
2.0.0 2.5.0 2.6.0-dev 2.6.0-dev+JIT
while 77.952 80.325 87.239 491.907 i/s - 2.000k times in 25.656691s 24.898879s 22.925498s 4.065807s
Comparison:
while
2.6.0-dev+JIT: 491.9 i/s
2.6.0-dev: 87.2 i/s - 5.64x slower
2.5.0: 80.3 i/s - 6.12x slower
2.0.0: 78.0 i/s - 6.31x slower

Advanced usage: YAML and CLI

$ benchmark-driver -h
Usage: benchmark-driver [options] [YAML]
-r, --runner [TYPE]
-o, --output [TYPE]
-e, --executables [EXECS]
--rbenv [VERSIONS]
--repeat-count [NUM]
--bundler
--filter [REGEXP]
--verbose [LEVEL]
--run-duration [SECONDS]
prelude: |
large_a = "Hellooooooooooooooooooooooooooooooooooooooooooooooooooo"
large_b = "Wooooooooooooooooooooooooooooooooooooooooooooooooooorld"
small_a = "Hello"
small_b = "World"
benchmark:
large: '"#{large_a}, #{large_b}!"'
small: '"#{small_a}, #{small_b}!"'
$ benchmark-driver benchmark.yml --rbenv '2.4.3;2.5.0'
Warming up --------------------------------------
large 3.693M i/s
small 9.913M i/s
Calculating -------------------------------------
2.4.3 2.5.0
large 3.895M 5.485M i/s - 11.079M times in 2.844249s 2.019943s
small 11.755M 11.103M i/s - 29.740M times in 2.529966s 2.678612s
Comparison:
large
2.5.0: 5484705.8 i/s
2.4.3: 3895155.8 i/s - 1.41x slower
small
2.4.3: 11755264.9 i/s
2.5.0: 11102923.0 i/s - 1.06x slower

Plugin System

benchmark-driver.github.io

Ruby Core

Ruby Method

MJIT

Optcarrot

Future works

Sophisticating plugin interface

module BenchmarkDriver
Metrics = ::BenchmarkDriver::Struct.new(
:value, # @param [Float]
:executable, # @param [BenchmarkDriver::Config::Executable]
:duration, # @param [Float,nil]
)
Metrics::Type = ::BenchmarkDriver::Struct.new(
:unit, # @param [String]
:larger_better, # @param [TrueClass,FalseClass]
:worse_word, # @param [String]
defaults: { larger_better: true, worse_word: 'slower' },
)
end

Add more benchmarks to benchmark-driver.github.io

Integrating derailed_benchmarks.gem

Conclusion

Acknowledgements

--

--

--

Software engineer at Treasure Data. https://twitter.com/k0kubun https://github.com/k0kubun https://speakerdeck.com/k0kubun

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Our little story

A quick view of Data Snapshot

Running tcpdump inside Docker Swarm

I’m Not So Sure About Chrome OS 90

Build your own website on GitHub for FREE

DLithe_BC_NFS_T_Task30_C#

Ninja Skills writeup

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
k0kubun

k0kubun

Software engineer at Treasure Data. https://twitter.com/k0kubun https://github.com/k0kubun https://speakerdeck.com/k0kubun

More from Medium

Ruby on Rails — Phase 4 Project

Ruby on Rails Project: Devhub

Ruby on Rails Console

Upgrade Ruby on Rails Project 101 — Phase 2 (Webpack)