Processed: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x

[Debian Bug Tracking System]

found -1 rmatrix/1.3-0-1

Bug #980809 [src:rmatrix, src:r-cran-glmmtmb] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Marked as found in versions rmatrix/1.3-0-1.

found -1 r-cran-glmmtmb/1.0.2.1-1

Bug #980809 [src:rmatrix, src:r-cran-glmmtmb] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Marked as found in versions r-cran-glmmtmb/1.0.2.1-1.

[Dirk Eddelbuettel]

Hi Graham,

On 23 January 2021 at 00:21, Graham Inggs wrote:
| On Fri, 22 Jan 2021 at 23:38, Dirk Eddelbuettel <***@debian.org> wrote:
| > Can you still please talk to the corresponding Debian maintainer for glmmTMB
| > so that he/she can walk it upstream?
|
| I have assigned this bug to both packages, until we can figure out
| where the fix needs to happen.
|
| > The package is clean where it matters:
| > https://cloud.r-project.org/web/checks/check_results_glmmTMB.html
|
| Are they doing any tests on big-endian architectures? Remember s390x
| is our only big-endian release architecture, and r-cran-glltmb's
| autopkgtests have green lights on all our little-endian architectures,
| so this hints to an endianness bug to me.

No CRAN does not test on s390x or comparable. But the change in Matrix was
also relatively small.

But I will contact its main maintainer and also CC the glmmTMB maintainer.

Dirk

[Dirk Eddelbuettel]

Graham,

Any update here? It still points at Matrix aka rmatrix aka r-cran-matrix,
and I still think wrongly.

Best, Dirk

[Graham Inggs]

Hi Dirk

Post by Dirk Eddelbuettel
Any update here? It still points at Matrix aka rmatrix aka r-cran-matrix,
and I still think wrongly.

I'm still waiting for the results of the autopkgtest against the
rebuilt r-cran-tmb on s390x.
Unfortunately, it went into the queue behind a bunch of perl tests.

Right now, I see results from 2020-12-11 and 2021-01-22, and two queued tests:
https://ci.debian.net/user/britney/jobs?package=r-cran-glmmtmb&suite[]=testing&arch[]=s390x

Regards
Graham

[Dirk Eddelbuettel]

Hi Graham,

On 28 January 2021 at 08:19, Graham Inggs wrote:
| On Thu, 28 Jan 2021 at 01:12, Dirk Eddelbuettel <***@debian.org> wrote:
| > Any update here? It still points at Matrix aka rmatrix aka r-cran-matrix,
| > and I still think wrongly.
|
| I'm still waiting for the results of the autopkgtest against the
| rebuilt r-cran-tmb on s390x.
| Unfortunately, it went into the queue behind a bunch of perl tests.
|
| Right now, I see results from 2020-12-11 and 2021-01-22, and two queued tests:
| https://ci.debian.net/user/britney/jobs?package=r-cran-glmmtmb&suite[]=testing&arch[]=s390x

Yes, sorry, realized that too as I still had a browser tab open for
https://ci.debian.net/packages/r/r-cran-glmmtmb/unstable/s390x/
which appears stuck at Jan 22 for now.

Should have checked there first, my bad. Thanks for confirming though.

Dirk

[Paul Gevers]

On request of Graham I ran the autopkgtest on 390x manually to speed up
the results.

Please find the output attached.

Paul

[Debian Bug Tracking System]

Your message dated Sat, 30 Jan 2021 08:57:52 -0600
with message-id <***@rob.eddelbuettel.com>
and subject line Re: Bug#980809: log on s390x
has caused the Debian Bug report #980809,
regarding rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)

[Graham Inggs]

Control: reopen -1

Hi Dirk

Paul's test results showed that the run-unit-test autopkgtest still
fails with the same errors (
Error: gradient function must return a numeric vector of length...) as
in my original report.

The "Package version inconsistency detected" warning is now gone (due
to #980902).

You can now see the same result on ci.debian.net [1] in the test run
at 2021-01-31 02:58:21 UTC.

Regards
Graham


[1] https://ci.debian.net/packages/r/r-cran-glmmtmb/unstable/s390x/

[Debian Bug Tracking System]

Post by Graham Inggs
reopen -1

Bug #980809 {Done: Dirk Eddelbuettel <***@debian.org>} [src:rmatrix, src:r-cran-glmmtmb] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Bug reopened
Ignoring request to alter fixed versions of bug #980809 to the same values previously set

[Dirk Eddelbuettel]

On 31 January 2021 at 09:20, Graham Inggs wrote:
| Control: reopen -1
|
| Hi Dirk
|
| Paul's test results showed that the run-unit-test autopkgtest still
| fails with the same errors (
| Error: gradient function must return a numeric vector of length...) as
| in my original report.
|
|
| The "Package version inconsistency detected" warning is now gone (due
| to #980902).
|
| You can now see the same result on ci.debian.net [1] in the test run
| at 2021-01-31 02:58:21 UTC.

¯\_(ツ)_/¯

When I look at the log I only see as relevant this tail part at the end:

autopkgtest [21:58:17]: test pkg-r-autopkgtest: -----------------------]
pkg-r-autopkgtest PASS
autopkgtest [21:58:17]: test pkg-r-autopkgtest: - - - - - - - - - - results - - - - - - - - - -
autopkgtest [21:58:17]: @@@@@@@@@@@@@@@@@@@@ summary
run-unit-test FAIL non-zero exit status 1
pkg-r-autopkgtest PASS

which shows no influence of Matrix. Matrix is a core R package (part of the
"recommended" set) and has been there since forever. This is a bug in
glmmTMB. All the 'gradient function must ...' errors are from its tests and
points to me to a recent change in R (where boolean comparisons can now check
if the compared vector is corre. The Matrix change may simply be the trigger.

In any event, all these errors also have backtrace:

-- 1. Error (test-VarCorr.R:11:1): (code run outside of `test_that()`) ---------
Error: gradient function must return a numeric vector of length 6
Backtrace:
1. glmmTMB::glmmTMB(distance ~ age + (age | Subject), data = Orthodont) test-VarCorr.R:11:0
2. glmmTMB::fitTMB(TMBStruc)
4. glmmTMB:::optfun()
6. base::with.default(...)
7. [ base::eval(...) ] with 1 more call

No Matrix here.

Can you please talk to glmmTMB? Their GH seems active
(https://github.com/glmmTMB/glmmTMB/commits/master)

Dirk

| Regards
| Graham
|
|
| [1] https://ci.debian.net/packages/r/r-cran-glmmtmb/unstable/s390x/

[Graham Inggs]

Control: forwarded -1 https://github.com/glmmTMB/glmmTMB/issues/665

After another detour via #981623, r-cran-glmmtmb is now also rebuilt.
Even with the rebuilt r-cran-glmmtmb, I still see the "gradient
function must return a numeric vector of length" errors on s390x and
ppc64, which are both big-endian architectures.

[Debian Bug Tracking System]

Post by Graham Inggs
forwarded -1 https://github.com/glmmTMB/glmmTMB/issues/665

Bug #980809 [src:rmatrix, src:r-cran-glmmtmb] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Set Bug forwarded-to-address to 'https://github.com/glmmTMB/glmmTMB/issues/665'.

[Dirk Eddelbuettel]

Hi Graham,

On 2 February 2021 at 11:32, Graham Inggs wrote:
| Control: forwarded -1 https://github.com/glmmTMB/glmmTMB/issues/665
|
| After another detour via #981623, r-cran-glmmtmb is now also rebuilt.
| Even with the rebuilt r-cran-glmmtmb, I still see the "gradient
| function must return a numeric vector of length" errors on s390x and
| ppc64, which are both big-endian architectures.

I still suspect it is coming from somewhere else. Upstream package "TMB" just
had a new release 1.7.19 on Feb 5, it also provides headers for glmmTMB so
that is another change vector. (As I still don't think it is coming from
Matrix ubut we'd need a traceback() to know more...).

Dirk

[Graham Inggs]

Hi Dirk

Sure. Just fire up R on the box, don't install anything beyond r-base-core
(and maybe r-recommended accounting for a few of the ones above) and then say

Perfect, thank you!

I installed only r-base-core and r-base-dev (gfortran, blas and lapack
were needed for compilation).

install.packages("glmmTMB")

... much downloading and compiling ...
* DONE (glmmTMB)

The downloaded source packages are in
'/tmp/RtmpzKTGsP/downloaded_packages'

require("glmmTMB")

Loading required package: glmmTMB

require("lme4")

Loading required package: lme4
Loading required package: Matrix

data("Orthodont", package="nlme")
fm1 <- glmmTMB(distance ~ age + (age|Subject), data = Orthodont)

Error in (function (start, objective, gradient = NULL, hessian = NULL, :
gradient function must return a numeric vector of length 6
In addition: Warning messages:
1: In Matrix::sparseMatrix(dims = c(0, 0), i = integer(0), j = integer(0), :
'giveCsparse' has been deprecated; setting 'repr = "T"' for you
2: In Matrix::sparseMatrix(dims = c(0, 0), i = integer(0), j = integer(0), :
'giveCsparse' has been deprecated; setting 'repr = "T"' for you
3: In Matrix::sparseMatrix(dims = c(0, 0), i = integer(0), j = integer(0), :
'giveCsparse' has been deprecated; setting 'repr = "T"' for you
4: In (function (start, objective, gradient = NULL, hessian = NULL, :
NA/NaN function evaluation
Timing stopped at: 0.244 0.001 0.246

Upstream says those warnings can be ignored. I also tried the above
with r-recommended installed as well and got the same result.

I do not have access to any weird or exotic hardware.

As a DD, you already have access to all our porterboxes[1]! :-)

Regards
Graham


[1] https://wiki.debian.org/PorterBoxHowToUse

[Graham Inggs]

Hi Dirk

Graham: What does that bigendian box say for Sys.info()[["machine"]] ?
Sys.info()[["machine"]]

[1] "ppc64"

Should we test for endianness instead? And then maybe try to roll up to the
cause of the difference?

I've just tested on 32-bit big-endian and the glmmTMB problem does not
occur there, so at this stage I would say test for big-endian **and**
64 bits.

Just to be clear, is the fix you are proposing in Matrix going to fix
the glmmTMB error?

If a bug shows up on 64-bit big-endian, but not on little-endian or
32-bit big-endian, then it's usually a sign that somewhere a 64-bit
variable is incorrectly being read or written to as a 32-bit variable.

Regards
Graham


[*] https://db.debian.org/machines.cgi?sortby=purpose&sortorder=dsc

[Dirk Eddelbuettel]

On 17 February 2021 at 16:58, Graham Inggs wrote:
| Hi Dirk
|
| On Wed, 17 Feb 2021 at 15:10, Dirk Eddelbuettel <***@debian.org> wrote:
| > Graham: What does that bigendian box say for Sys.info()[["machine"]] ?
|
| The other big-endian box I tested was perotto.debian.net [*], it says:
|
| > Sys.info()[["machine"]]
| [1] "ppc64"
|
| > Should we test for endianness instead? And then maybe try to roll up to the
| > cause of the difference?
|
| I've just tested on 32-bit big-endian and the glmmTMB problem does not
| occur there, so at this stage I would say test for big-endian **and**
| 64 bits.
|
| Just to be clear, is the fix you are proposing in Matrix going to fix
| the glmmTMB error?

No. That may well be something else.

| If a bug shows up on 64-bit big-endian, but not on little-endian or
| 32-bit big-endian, then it's usually a sign that somewhere a 64-bit
| variable is incorrectly being read or written to as a 32-bit variable.

CRAN uses a Solaris machine for some endianness variability but I believe
that machine runs only 32bit.

Dirk

| Regards
| Graham
|
|
| [*] https://db.debian.org/machines.cgi?sortby=purpose&sortorder=dcs

[Dirk Eddelbuettel]

Graham,

Thanks for the bug tracker follow-up which made me aware of the ongoing
discussion in #665 at glmmTMB. It's frustrating to have the run around but it
really looks like as I argued all along: not an issue in Matrix. Now, TMB is
of course a complex package too.. Appreciate you chasing this so thoroughly.

Dirk

PS What is bts-link? Is that a new service we have to tie our BTS to GH and others?

[Graham Inggs]

Hi Dirk

If it were me I would turn off the autopkgtest in r-cran-glmmtmb (maybe just
on s390x) and move on.

That would just be hiding the problem.

If it is the kind of bug I described previously, it shows up more
often on big-endian systems, but it is still present on little-endian
systems, and will show up in the right (wrong) conditions.

Regards
Graham

[Martin Maechler]

Post by Graham Inggs
Hi Dirk

If it were me I would turn off the autopkgtest in r-cran-glmmtmb (maybe just
on s390x) and move on.

That would just be hiding the problem.
If it is the kind of bug I described previously, it shows up more
often on big-endian systems, but it is still present on little-endian
systems, and will show up in the right (wrong) conditions.

You may be right on spot, Graham, or not..
Currently I don't have time to investigate ... also I *have* been a
bit puzzled by the rankMatrix(*, "qr") -- Matrix-only --
discrepancy that Dirk found.
I agree it *would* be good to dig further, but that needs a chunk of
dedicated time (and "free head") which I don't have for now.

Apropos 'rabbit hole' : Isn't the origin -- long before the Matrix
movie -- in Lewis Carrol's "Alice in Wonderland" (1865 !!):
https://en.wikipedia.org/wiki/White_Rabbit ??

Best, Martin

[Dirk Eddelbuettel]

Graham, Martin,

So I logged back onto the s390x box we can access. Matrix 1.13-2 does fail
the factorization test when doing

_R_CHECK_FORCE_SUGGESTS_=false R CMD check --no-manual \
--no-vignettes Matrix_1.3-2.tar.gz

and Matrix 1.2-18 passes it. So Graham was right all-along that there was
something in the 1.3-0 transition. Now, is that ultimately Matrix or the
underlying QR code it uses? I don't know. But it reasonably easy for me to
get access (and install the required packages) so once Martin has time to
think through an approach or two I can likely test it.

Best, Dirk

[Debian Bug Tracking System]

reassign -1 src:rmatrix 1.3-0-1

Bug #980809 [src:rmatrix, src:r-cran-glmmtmb] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Bug reassigned from package 'src:rmatrix, src:r-cran-glmmtmb' to 'src:rmatrix'.
No longer marked as found in versions rmatrix/1.3-0-1 and r-cran-glmmtmb/1.0.2.1-1.
Ignoring request to alter fixed versions of bug #980809 to the same values previously set
Bug #980809 [src:rmatrix] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Marked as found in versions rmatrix/1.3-0-1.

tags -1 - fixed-upstream

Bug #980809 [src:rmatrix] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Removed tag(s) fixed-upstream.

forwarded -1 https://github.com/kaskr/adcomp/issues/340

Bug #980809 [src:rmatrix] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Changed Bug forwarded-to-address to 'https://github.com/kaskr/adcomp/issues/340' from 'https://github.com/glmmTMB/glmmTMB/issues/665'.

[Debian Bug Tracking System]

affects -1 src:r-cran-glmmtmb

Bug #980809 [src:rmatrix] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Added indication that 980809 affects src:r-cran-glmmtmb

affects -1 src:r-cran-tmb

Bug #980809 [src:rmatrix] rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x
Added indication that 980809 affects src:r-cran-tmb