Discussion:
Bug#1081027: src:sssd: flaky autopkgtest: spawn id exp3 not open
Add Reply
Paul Gevers
2024-09-07 05:20:01 UTC
Reply
Permalink
Source: sssd
Version: 2.9.5-3
Severity: serious
User: debian-***@lists.debian.org
Usertags: flaky

Dear maintainer(s),

I looked at the results of the autopkgtest of your package. I noticed
that it regularly fails.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

https://ci.debian.net/packages/s/sssd/testing/amd64/51295873/

41s The LDAP user can login on a terminal
41s + /usr/bin/expect -f debian/tests/login.exp testuser1 testuser1secret
41s spawn login
41s send: spawn id exp3 not open
41s while executing
41s "send "$user\r""
41s (file "debian/tests/login.exp" line 21)
Andreas Hasenack
2024-11-05 18:10:02 UTC
Reply
Permalink
It passed[1] in salsa:

+ /usr/bin/expect -f debian/tests/login.exp testuser1 testuser1secret
The LDAP user can login on a terminal
spawn login
ldap login: testuser1
Password:
Linux ldap.example.com 5.10.0-33-cloud-amd64 #1 SMP Debian 5.10.226-1
(2024-10-03) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Creating directory '/home/testuser1'.
***@ldap:~$ id -un
testuser1
***@ldap:~$ + cleanup
+ result=0
+ set +e
+ [ 0 -ne 0 ]
+ echo ## All tests passed, phew
## All tests passed, phew
autopkgtest [18:02:28]: test ldap-user-group-ldap-auth: -----------------------]

Perhaps we could remove the "set -x"? I wonder if it could interfere sometimes.

1. https://salsa.debian.org/ahasenack/sssd/-/jobs/6539986/viewer#L801
I wrote that test initially, and it has been passing in Ubuntu. Sounds
like it's some difference in the infrastructure.
I tried locally in a debian LXD container, and it passed just fine. Is
it also failing in salsa pipeline runners, or just in the migration by
britney?
105s + /usr/bin/expect -f debian/tests/login.exp testuser1
105s spawn login
105s send: spawn id exp3 not open
105s while executing
105s "send "$user\r""
105s (file "debian/tests/login.exp" line 21)
-----------------------]
These timestamps don't seem to indicate when it happened, but when the
log dump happened, right? Otherwise it all happened at 105s basically.
If this failure also happens in salsa, then we can inject some
debugging into the test in the case of failures. I don't know if I
have permissions to trigger a pipeline run in salsa, I'll try with an
MP.
Hi,
I looked at the results of the autopkgtest of your package. I noticed that
it regularly fails.
Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.
Don't hesitate to reach out if you need help and some more information
from our infrastructure.
Paul
https://ci.debian.net/packages/s/sssd/testing/amd64/51295873/
41s The LDAP user can login on a terminal
41s + /usr/bin/expect -f debian/tests/login.exp testuser1 testuser1secret
41s spawn login
41s send: spawn id exp3 not open
41s while executing
41s "send "$user\r""
41s (file "debian/tests/login.exp" line 21)
I'm aware of folks looking into that, but AFAICT so far nobody was
able to proberly reproduce the issue - so it feels like really being
a flaky test? Should the ldap-user-group-ldap-auth test get marked
as flaky for now, until someone managed to properly take care of
this?
IMO sssd needs to make its way into trixie.
regards
-mika-
Michael Prokop
2024-11-06 06:50:01 UTC
Reply
Permalink
Hi!
Post by Andreas Hasenack
I wrote that test initially, and it has been passing in Ubuntu. Sounds
like it's some difference in the infrastructure.
I tried locally in a debian LXD container, and it passed just fine. Is
it also failing in salsa pipeline runners, or just in the migration by
britney?
[...]
[...]
Post by Andreas Hasenack
autopkgtest [18:02:28]: test ldap-user-group-ldap-auth: -----------------------]
Thanks for looking into it, Andreas.
Post by Andreas Hasenack
Perhaps we could remove the "set -x"? I wonder if it could interfere sometimes.
The `set -x` *should* only influence the output of the executed
script, needed/useful for debugging usually. :)
Post by Andreas Hasenack
1. https://salsa.debian.org/ahasenack/sssd/-/jobs/6539986/viewer#L801
Paul, do you know what could be the best option to reproduce the
behavior of https://ci.debian.net/ (locally)? Because the problem
seems to be environment specific, no one seems to have been able to
reproduce it on salsa so far. :-/

Or what would be the best option to ignore this for now until it has
been tracked down, mark the test as flaky?

regards
-mika-
Paul Gevers
2024-11-06 19:10:02 UTC
Reply
Permalink
Hi
Post by Michael Prokop
Paul, do you know what could be the best option to reproduce the
behavior of https://ci.debian.net/ (locally)? Because the problem
seems to be environment specific, no one seems to have been able to
reproduce it on salsa so far. :-/
As reported, it's flaky. Which means it might very well be only
occurring under heavy load, or when specific other things are happening
on the system. E.g. on i386, where only one debci worker runs per host,
it seems to be much less flaky than on the other hosts where we run
multiple (up to 18 on amd64) debci workers per host. You could try to
spot patterns by matching timestamps of passing and failing tests to the
historical performance [1].
Post by Michael Prokop
Or what would be the best option to ignore this for now until it has
been tracked down, mark the test as flaky?
It looks like each autopkgtest stanza has only one test, so yes, marking
it flaky will resolve the problem (but also make the test close to
worthless). (If on the contrary it's part of a whole test suite, you'd
rather want to only mark the particular test as flaky or disable it, and
not mark the autopkgtest stanza as flaky).

Paul

[1] https://ci.debian.net/munin/

Loading...