Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h

[Paul Gevers]

Hi Nilesh,

Hi,

Your package has an autopkgtest, great. However, I looked into
the history of your autopkgtest [1] and I noticed version 1.5.3-2 fails
regularly on amd64, while sporadically a rerun passes. I copied some of
the output at the bottom of this report. It hits the autopkgtest time
out after 2hours and 47 minutes. Successful runs pass in less than a minute.
Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

That makes sense - do you think marking this test as flaky can be
solution?

No, because when a test times out, that restriction doesn't work. You'll
need to keep the test below 2:47, and as it normally takes less than a
minute, it may point at something seriously hanging.

Paul

[Debian Bug Tracking System]

forwarded -1 https://github.com/GATB/simka/issues/20

Bug #986256 [src:simka] simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Set Bug forwarded-to-address to 'https://github.com/GATB/simka/issues/20'.

[Étienne Mollier]

Control: forwarded -1 https://github.com/GATB/simka/issues/20

Hi Paul, Hi Nilesh,
feel also free to grant dm rights if you feel
appropriate.

Done

Thanks Nilesh, forwarded upstream and uploaded! :)

Kind Regards,

[Debian Bug Tracking System]

Your message dated Mon, 05 Apr 2021 20:20:33 +0000
with message-id <E1lTVi5-000Gxf-***@fasolo.debian.org>
and subject line Bug#986256: fixed in simka 1.5.3-3
has caused the Debian Bug report #986256,
regarding simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)

[Graham Inggs]

Control: reopen -1

This is still occurring with simka 1.5.3-3, see:
https://ci.debian.net/packages/s/simka/testing/amd64/

[Debian Bug Tracking System]

Post by Graham Inggs
reopen -1

Bug #986256 {Done: Étienne Mollier <***@mailoo.org>} [src:simka] simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
'reopen' may be inappropriate when a bug has been closed with a version;
all fixed versions will be cleared, and you may need to re-add them.
Bug reopened
No longer marked as fixed in versions simka/1.5.3-3.

[Étienne Mollier]

Hi Graham,

Post by Graham Inggs
https://ci.debian.net/packages/s/simka/testing/amd64/

Thanks for having noticed it. It looks like I got caught by the
Python script not taking the -nb-cores argument, contrary to the
shell script. While I couldn't reproduce the hang on emulator,
it seems I can just reproduce the problem by enforcing the cores
count to 48 within the Python script. The test is precisely
hanging at this location:

[...]
Command for visualizing results with metadata annotations:
python ../scripts/visualization/run-visualization.py -in ./simka_results/ -out ./simka_results/ -pca -heatmap -tree -metadata-in ../example/dataset_metadata.csv -metadata-variable VARIABLE_1
TESTING k=31 t=0
simka -in simka_input.txt -out ./__results__/results_k31_t0 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 31 -abundance-min 0 -verbose 0 -nb-cores 48
OK
TESTING k=21 t=0
simka -in simka_input.txt -out ./__results__/results_k21_t0 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 21 -abundance-min 0 -verbose 0 -nb-cores 48
OK
TESTING k=31 t=2
simka -in simka_input.txt -out ./__results__/results_k31_t2 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 31 -abundance-min 2 -verbose 0 -nb-cores 48
_

The output on CI infrastructure is a bit misleading, because the
standard output is not flushed on each output line by default.
I enforced flushed output manually in the Python script to have
a cleared view. But I /believe/ I could reproduce precisely the
deadlock this time. Hopefully a further upload with controlled
cores count at this level should do the trick.

Have a nice day, :)

[Debian Bug Tracking System]

Your message dated Wed, 07 Apr 2021 21:48:59 +0000
with message-id <E1lUG2l-0006Er-***@fasolo.debian.org>
and subject line Bug#986256: fixed in simka 1.5.3-4
has caused the Debian Bug report #986256,
regarding simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)

[Étienne Mollier]

Hi, just to confirm, tests are now passing on ci-worker13[1].

[1]: https://ci.debian.net/data/autopkgtest/testing/amd64/s/simka/11550136/log.gz

Hope it will last, thanks for having kept track of this!

Have a nice day, :)