Bug#840572: ruby-benchmark-suite: FTBFS (failing tests)

[Santiago Vila]

Package: src:ruby-benchmark-suite
Version: 1.0.0+git.20130122.5bded6-2
Severity: serious

Dear maintainer:

I tried to build this package in stretch with "dpkg-buildpackage -A"
(which is what the "Arch: all" autobuilder would do to build it)
but it failed:

--------------------------------------------------------------------------------
[...]
debian/rules build-indep
dh build-indep --buildsystem=ruby --with ruby
dh_testdir -i -O--buildsystem=ruby
dh_update_autotools_config -i -O--buildsystem=ruby
dh_auto_configure -i -O--buildsystem=ruby
dh_ruby --configure
debian/rules override_dh_auto_build
make[1]: Entering directory '/<<BUILDDIR>>/ruby-benchmark-suite-1.0.0+git.20130122.5bded6'
dh_auto_build
dh_ruby --build
dh_ruby --build
# build documentation
rdoc --main=README.md lib

[... snipped ...]

/usr/bin/ruby2.3 /usr/bin/gem2deb-test-runner

┌──────────────────────────────────────────────────────────────────────────────┐
│ Run tests for ruby2.3 from debian/ruby-tests.rb │
└──────────────────────────────────────────────────────────────────────────────┘

RUBYLIB=/<<BUILDDIR>>/ruby-benchmark-suite-1.0.0+git.20130122.5bded6/debian/ruby-benchmark-suite/usr/lib/ruby/vendor_ruby:. GEM_PATH=/var/lib/gems/2.3.0:/usr/lib/x86_64-linux-gnu/rubygems-integration/2.3.0:/usr/share/rubygems-integration/2.3.0:/usr/share/rubygems-integration/all ruby2.3 debian/ruby-tests.rb
Loaded suite debian/ruby-tests
Started
Calculating -------------------------------------
sleep 1 i/100ms
-------------------------------------------------
sleep 3.7 (±27.1%) i/s - 4 in 1.110849s
F
===============================================================================
Failure:
<4.0> -/+ <0.2> expected to include
<3.692020327852676>.

Relation:
<<3.692020327852676> < <4.0>-<0.2>[3.8] <= <4.0>+<0.2>[4.2]>
test_ips(TestBenchmarkSuite)
/<<BUILDDIR>>/ruby-benchmark-suite-1.0.0+git.20130122.5bded6/test/test_benchmark_suite.rb:18:in `test_ips'
15:
16: assert_equal "sleep", rep.label
17: assert_equal 4, rep.iterations
=> 18: assert_in_delta 4.0, rep.ips, 0.2
19: end
20: end
===============================================================================


Finished in 2.12274063 seconds.
------
1 tests, 3 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
------
0.47 tests/s, 1.41 assertions/s
ERROR: Test "ruby2.3" failed. Exiting.
dh_auto_install: dh_ruby --install /<<BUILDDIR>>/ruby-benchmark-suite-1.0.0+git.20130122.5bded6/debian/tmp returned exit code 1
debian/rules:27: recipe for target 'override_dh_auto_install' failed
make[1]: *** [override_dh_auto_install] Error 1
make[1]: Leaving directory '/<<BUILDDIR>>/ruby-benchmark-suite-1.0.0+git.20130122.5bded6'
debian/rules:15: recipe for target 'binary-indep' failed
make: *** [binary-indep] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary-indep gave error exit status 2
--------------------------------------------------------------------------------

The relevant part of the build log is included above.

There is apparently a test which tries to ensure that a certain task is completed
in a certain amount of time (an interval, really). This is usually not a good
idea for a Debian source package, because there is not any guarantee anywhere
that the CPU performance will remain constant during the package build.
The same machine could be doing lots of other things in parallel.

If this is really a bug in one of the build-depends, please use reassign and affects,
so that this is still visible in the page for this package.

Thanks.

[Santiago Vila]

retitle 840572 ruby-benchmark-suite: FTBFS randomly (failing tests)
severity 840572 important
thanks

I built this package a lot of times today. So far I have one failure
and several hundred successful builds, so the probability of failure
may well be below 1% and this is really a "FTBFS randomly" bug.

I asked Release Managers what to do with bugs like this but it's still
undecided. The only thing I have is a guideline by Julien Cristau:

[...] if the failure rate is low enough I think a lower severity
can make sense.

(See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844264 for details)

Based on the above, it does not make sense to keep this as "serious".
I'm adding this bug to the collection of "FTBFS randomly" bugs:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=***@debian.org;tag=ftbfs-randomly

Thanks.

[Debian Bug Tracking System]

Post by Santiago Vila
retitle 840572 ruby-benchmark-suite: FTBFS randomly (failing tests)

Bug #840572 [src:ruby-benchmark-suite] ruby-benchmark-suite: FTBFS (failing tests)
Changed Bug title to 'ruby-benchmark-suite: FTBFS randomly (failing tests)' from 'ruby-benchmark-suite: FTBFS (failing tests)'.

Post by Santiago Vila
severity 840572 important

Bug #840572 [src:ruby-benchmark-suite] ruby-benchmark-suite: FTBFS randomly (failing tests)
Severity set to 'important' from 'serious'

Post by Santiago Vila
thanks

Stopping processing here.

Please contact me if you need assistance.