Bug#978353: serf: FTBFS: tests failed

[Lucas Nussbaum]

Source: serf
Version: 1.3.9-8
Severity: serious
Justification: FTBFS on amd64
Tags: bullseye sid ftbfs
Usertags: ftbfs-20201226 ftbfs-bullseye

Hi,

During a rebuild of all packages in sid, your package failed to build
on amd64.

make[2]: Entering directory '/<<PKGBUILDDIR>>'
if ! [ -d debian/distcerts ]; then \
mkdir -p debian/testcerts/private debian/distcerts; \
cp test/server/*.pem test/server/serfclientcert.p12 debian/distcerts/; \
(cd debian/testcerts && python3 ../create_certs.py); \
cp debian/testcerts/*.pem debian/testcerts/private/serfserverkey.pem debian/testcerts/serfclientcert.p12 test/server/; \
fi
scons check CFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" LINKFLAGS="-Wl,-z,relro -Wl,-z,now -Wl,-z,defs -Wl,--as-needed"
scons: Reading SConscript files ...
Checking for GNU-compatible C compiler...(cached) yes
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test/serf_get.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/serf_get.c
test/serf_get.c:423:21: warning: variable 'request' set but not used [-Wunused-but-set-variable]
423 | serf_request_t *request;
| ^~~~~~~
gcc -o test/serf_get -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/serf_get.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
gcc -o test/serf_response.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/serf_response.c
gcc -o test/serf_response -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/serf_response.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
gcc -o test/serf_request.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/serf_request.c
gcc -o test/serf_request -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/serf_request.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
gcc -o test/serf_spider.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/serf_spider.c
test/serf_spider.c:332:30: warning: variable 'req_alloc' set but not used [-Wunused-but-set-variable]
332 | serf_bucket_alloc_t *req_alloc;
| ^~~~~~~~~
test/serf_spider.c:636:9: warning: variable 'count' set but not used [-Wunused-but-set-variable]
636 | int count;
| ^~~~~
test/serf_spider.c:635:27: warning: variable 'method' set but not used [-Wunused-but-set-variable]
635 | const char *raw_url, *method;
| ^~~~~~
gcc -o test/serf_spider -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/serf_spider.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
gcc -o test/test_all.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_all.c
gcc -o test/CuTest.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/CuTest.c
gcc -o test/test_util.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_util.c
gcc -o test/test_context.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_context.c
test/test_context.c:654:23: warning: unused variable 'pb' [-Wunused-variable]
654 | progress_baton_t *pb;
| ^~
gcc -o test/test_buckets.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_buckets.c
test/test_buckets.c:1112:35: warning: format '%d' expects argument of type 'int', but argument 3 has type 'apr_size_t' {aka 'long unsigned int'} [-Wformat=]
1112 | "Read more data than expected, EAGAIN"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1113 | " inserted at pos: %d, remainder: \"%s\"",
1114 | cut, fullmsg + cut);
| ~~~
| |
| apr_size_t {aka long unsigned int}
test/test_buckets.c:1113:55: note: format string is defined here
1113 | " inserted at pos: %d, remainder: \"%s\"",
| ~^
| |
| int
| %ld
test/test_buckets.c:1117:35: warning: format '%d' expects argument of type 'int', but argument 3 has type 'apr_size_t' {aka 'long unsigned int'} [-Wformat=]
1117 | "Read data is not equal to expected, EAGAIN"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1118 | " inserted at pos: %d, remainder: \"%s\"",
1119 | cut, fullmsg + cut);
| ~~~
| |
| apr_size_t {aka long unsigned int}
test/test_buckets.c:1118:55: note: format string is defined here
1118 | " inserted at pos: %d, remainder: \"%s\"",
| ~^
| |
| int
| %ld
test/test_buckets.c:1495:13: warning: 'test_deflate_4GBplus_buckets' defined but not used [-Wunused-function]
1495 | static void test_deflate_4GBplus_buckets(CuTest *tc)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
gcc -o test/test_auth.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_auth.c
gcc -o test/mock_buckets.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/mock_buckets.c
gcc -o test/test_ssl.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/test_ssl.c
gcc -o test/server/test_server.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/server/test_server.c
test/server/test_server.c:342:43: warning: passing argument 3 of 'servctx->clientstream->type->read' from incompatible pointer type [-Wincompatible-pointer-types]
342 | &buf, &readlen);
| ^~~~
| |
| char **
./serf.h:912:29: note: in definition of macro 'SERF__RECREAD'
912 | #define SERF__RECREAD(b,s) (s)
| ^
test/server/test_server.c:341:26: note: in expansion of macro 'serf_bucket_read'
341 | status = serf_bucket_read(servctx->clientstream, readlen,
| ^~~~~~~~~~~~~~~~
test/server/test_server.c:342:43: note: expected 'const char **' but argument is of type 'char **'
342 | &buf, &readlen);
| ^~~~
./serf.h:912:29: note: in definition of macro 'SERF__RECREAD'
912 | #define SERF__RECREAD(b,s) (s)
| ^
test/server/test_server.c:341:26: note: in expansion of macro 'serf_bucket_read'
341 | status = serf_bucket_read(servctx->clientstream, readlen,
| ^~~~~~~~~~~~~~~~
test/server/test_server.c:423:39: warning: passing argument 3 of 'servctx->servstream->type->read' from incompatible pointer type [-Wincompatible-pointer-types]
423 | &buf, &readlen);
| ^~~~
| |
| char **
./serf.h:912:29: note: in definition of macro 'SERF__RECREAD'
912 | #define SERF__RECREAD(b,s) (s)
| ^
test/server/test_server.c:422:22: note: in expansion of macro 'serf_bucket_read'
422 | status = serf_bucket_read(servctx->servstream, BUFSIZE,
| ^~~~~~~~~~~~~~~~
test/server/test_server.c:423:39: note: expected 'const char **' but argument is of type 'char **'
423 | &buf, &readlen);
| ^~~~
./serf.h:912:29: note: in definition of macro 'SERF__RECREAD'
912 | #define SERF__RECREAD(b,s) (s)
| ^
test/server/test_server.c:422:22: note: in expansion of macro 'serf_bucket_read'
422 | status = serf_bucket_read(servctx->servstream, BUFSIZE,
| ^~~~~~~~~~~~~~~~
gcc -o test/server/test_sslserver.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/server/test_sslserver.c
gcc -o test/test_all -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/test_all.o test/CuTest.o test/test_util.o test/test_context.o test/test_buckets.o test/test_auth.o test/mock_buckets.o test/test_ssl.o test/server/test_server.o test/server/test_sslserver.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
gcc -o test/serf_bwtp.o -c -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -std=c99 -Wdeclaration-after-statement -Wmissing-prototypes -Wall -O2 -pthread -isystem /usr/include/mit-krb5 -Wdate-time -D_FORTIFY_SOURCE=2 -DNDEBUG -DLINUX -D_REENTRANT -D_GNU_SOURCE -DSERF_HAVE_GSSAPI -DMOCKHTTP_OPENSSL -I. -I/usr/include/apr-1.0 -I/usr/include -I/usr/include test/serf_bwtp.c
test/serf_bwtp.c:158:20: warning: variable 'body_bkt' set but not used [-Wunused-but-set-variable]
158 | serf_bucket_t *body_bkt;
| ^~~~~~~~
test/serf_bwtp.c:374:29: warning: variable 'new_req' set but not used [-Wunused-but-set-variable]
374 | serf_request_t *new_req;
| ^~~~~~~
test/serf_bwtp.c:448:33: warning: variable 'new_req' set but not used [-Wunused-but-set-variable]
448 | serf_request_t *new_req;
| ^~~~~~~
test/serf_bwtp.c:485:21: warning: variable 'request' set but not used [-Wunused-but-set-variable]
485 | serf_request_t *request;
| ^~~~~~~
gcc -o test/serf_bwtp -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -pthread -Wl,-z,relro -Wl,-rpath=/<<PKGBUILDDIR>> test/serf_bwtp.o -L. -L/usr/lib/x86_64-linux-gnu -L/usr/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -lserf-1 -lssl -lcrypto -lz -lapr-1 -laprutil-1 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
/usr/bin/python3 build/check.py test test
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Test of gzip Content-Encoding</title>
</head>
<body>
<h1>This is a test</h1>
<p>This file was created with mod_deflate on the server side.</p>
<pre>curl -i --output gzip.response -H "Accept-Encoding: gzip" http://localhost:8080/1.html</pre>
<hr />
<address>Apache</address>
</body></html>
this is 1 test.
i am a test.this is a test.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>scotch.ics.uci.edu</title>
<!--base href="http://scotch.ics.uci.edu/" /-->
<link href="default.css" rel="stylesheet" type="text/css" />
</head>
<body>
<p>More to come!</p>
<p><a href="manual/">Apache httpd 2.0 manual</a></p>
<p><a href="CA.cert.pem">Trust our CA!</a></p>
<p><img src="apache_pb.gif" alt="Powered by Apache!" /></p>
</body>
</html>
this is 1 test.
i am a test.this is a test.
Trailer-Test: f
...........F......................................................
1) test_ssl_handshake: test/test_util.c:456: expected <0> but was <120199>
!!!FAILURES!!!
Runs: 66 Passes: 65 Fails: 1
== Testing test/testcases/deflate.response ==
== Testing test/testcases/chunked.response ==
== Testing test/testcases/simple.response ==
== Testing test/testcases/chunked-trailers.response ==
== Testing test/testcases/chunked-empty.response ==
== Running the unit tests ==
ERROR: test(s) failed in test_all
scons: *** [check] Error 1
scons: building terminated because of errors.
make[2]: *** [debian/rules:33: override_dh_auto_test] Error 2

The full build log is available from:
http://qa-logs.debian.net/2020/12/26/serf_1.3.9-8_unstable.log

A list of current common problems and possible solutions is available at
http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!

If you reassign this bug to another package, please marking it as 'affects'-ing
this package. See https://www.debian.org/Bugs/server-control#affects

If you fail to reproduce this, please provide a build log and diff it with me
so that we can identify if something relevant changed in the meantime.

About the archive rebuild: The rebuild was done on EC2 VM instances from
Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
failed build was retried once to eliminate random failures.

[James McCoy]

Post by Lucas Nussbaum
Source: serf
Version: 1.3.9-8
[...]

Trailer-Test: f
...........F......................................................
1) test_ssl_handshake: test/test_util.c:456: expected <0> but was <120199>

It looks like the change from libssl1.1 version 1.1.1h to 1.1.1i
regressed this test.

The documented changes between these two releases are:

Changes between 1.1.1h and 1.1.1i [8 Dec 2020]

*) Fixed NULL pointer deref in the GENERAL_NAME_cmp function
This function could crash if both GENERAL_NAMEs contain an EDIPARTYNAME.
If an attacker can control both items being compared then this could lead
to a possible denial of service attack. OpenSSL itself uses the
GENERAL_NAME_cmp function for two purposes:
1) Comparing CRL distribution point names between an available CRL and a
CRL distribution point embedded in an X509 certificate
2) When verifying that a timestamp response token signer matches the
timestamp authority name (exposed via the API functions
TS_RESP_verify_response and TS_RESP_verify_token)
(CVE-2020-1971)
[Matt Caswell]

*) Add support for Apple Silicon M1 Macs with the darwin64-arm64-cc target.
[Stuart Carnie]

*) The security callback, which can be customised by application code, supports
the security operation SSL_SECOP_TMP_DH. This is defined to take an EVP_PKEY
in the "other" parameter. In most places this is what is passed. All these
places occur server side. However there was one client side call of this
security operation and it passed a DH object instead. This is incorrect
according to the definition of SSL_SECOP_TMP_DH, and is inconsistent with all
of the other locations. Therefore this client side call has been changed to
pass an EVP_PKEY instead.
[Matt Caswell]

*) In 1.1.1h, an expired trusted (root) certificate was not anymore rejected
when validating a certificate path. This check is restored in 1.1.1i.
[David von Oheimb]

The full diff is at https://github.com/openssl/openssl/compare/OpenSSL_1_1_1h...OpenSSL_1_1_1i

Cheers,

[Justin Erenkrantz]

Thanks. I expect that this might be due to the last change - erroring out
on an expired self-signed root cert. Though I thought we didn’t check in a
root cert for our test chain...could Debian’s packaging be including a cert
for testing?

I will try to take a look this week with Debian sid...I assume it has
1.1.1i already? — justin

Post by Lucas Nussbaum
Source: serf
Version: 1.3.9-8
[...]

Trailer-Test: f
...........F......................................................
1) test_ssl_handshake: test/test_util.c:456: expected <0> but was

<120199>
It looks like the change from libssl1.1 version 1.1.1h to 1.1.1i
regressed this test.
Changes between 1.1.1h and 1.1.1i [8 Dec 2020]
*) Fixed NULL pointer deref in the GENERAL_NAME_cmp function
This function could crash if both GENERAL_NAMEs contain an EDIPARTYNAME.
If an attacker can control both items being compared then this could lead
to a possible denial of service attack. OpenSSL itself uses the
1) Comparing CRL distribution point names between an available CRL and a
CRL distribution point embedded in an X509 certificate
2) When verifying that a timestamp response token signer matches the
timestamp authority name (exposed via the API functions
TS_RESP_verify_response and TS_RESP_verify_token)
(CVE-2020-1971)
[Matt Caswell]
*) Add support for Apple Silicon M1 Macs with the darwin64-arm64-cc target.
[Stuart Carnie]
*) The security callback, which can be customised by application code, supports
the security operation SSL_SECOP_TMP_DH. This is defined to take an EVP_PKEY
in the "other" parameter. In most places this is what is passed. All these
places occur server side. However there was one client side call of this
security operation and it passed a DH object instead. This is incorrect
according to the definition of SSL_SECOP_TMP_DH, and is inconsistent with all
of the other locations. Therefore this client side call has been changed to
pass an EVP_PKEY instead.
[Matt Caswell]
*) In 1.1.1h, an expired trusted (root) certificate was not anymore rejected
when validating a certificate path. This check is restored in 1.1.1i.
[David von Oheimb]
The full diff is at
https://github.com/openssl/openssl/compare/OpenSSL_1_1_1h...OpenSSL_1_1_1i
Cheers,
--
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB

[James McCoy]

Thanks.  I expect that this might be due to the last change - erroring out on
an expired self-signed root cert.  Though I thought we didn’t check in a root
cert for our test chain...could Debian’s packaging be including a cert for
testing?

I use create_certs.py from trunk to re-generate the test certificates
every build, otherwise I was running into time bombs with the certs
expiring. Other than that, I don't do anything different than the
normal test process.

The Debian packaging does have some local patches[0] applied to address
issues that have been fixed upstream but not yet released.

[0]: https://sources.debian.org/patches/serf/1.3.9-8/

I will try to take a look this week with Debian sid...I assume it has 1.1.1i
already?  — justin 

Yes, it does.

Cheers,

[Justin Erenkrantz]

As an update, I've been able to triage this a bit further.

It's definitely that last noted change (erroring out on expired self-signed
root) that broke it. OpenSSL 1.1.1{g,h} are fine, but {i,-stable} are
not. Reverting just x509_vfy.c to what is in 1.1.1h causes the test to
pass.

In this test case, Serf receives 2 verify callbacks in test_ssl_handshake.
The first failing test case is to not have a known CA - so, we are
intentionally trying to trigger a verify failure. One of the app
callback received has the expected failure, the other doesn't. Serf
basically flags the second (success) as an unexpected pass.

2020-12-28T16:51:34.045142-05 test/test_ssl.c: Cert failure received: 4 ;
expected failure mask: 4

2020-12-28T16:51:34.045159-05 test/test_ssl.c: Cert failure received: 0 ;
expected failure mask: 4

The upstream issues/commits appear to be:

https://github.com/openssl/openssl/issues/13427
https://github.com/openssl/openssl/commit/3bed88a3970605a2ff817065f93b08e965d89e5f#diff-2a76d0a7ddc5ae2646a6c183270a7b4d5302d8491acb0af0dfbd70643efdf431

The key difference is almost certainly this change:

https://github.com/openssl/openssl/blob/OpenSSL_1_1_1h/crypto/x509/x509_vfy.c#L1754

---
return verify_cb_cert(ctx, xi, 0,

X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE);
---

https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/x509/x509_vfy.c#L1755

---
if (!verify_cb_cert(ctx, xi, 0,
X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE))
return 0;

xs = xi;
goto check_cert_time;
---

Up to 1.1.1h, OpenSSL would stop processing the certificate after sending
along the leaf error to the app callback. In -stable (1.1.1i+ and master),
if the app's callback doesn't return a failure, it will then proceed to the
date check portion (check_cert_time) - which then receives a successful
verification callback.

It's not clear to me if OpenSSL authors intended to make this breaking
change. On the serf side, we would need to think through what it would
mean to have our app callback return false upon failure in order to
short-circuit the check.

I probably won't get a chance to open an upstream OpenSSL issue today (or
even tomorrow)...

Cheers. -- justin

Index: test/test_ssl.c

===================================================================

--- test/test_ssl.c (revision 1884847)

+++ test/test_ssl.c (working copy)

@@ -465,6 +465,7 @@



tb->result_flags |= TEST_RESULT_SERVERCERTCB_CALLED;



+ test__log(TEST_VERBOSE, __FILE__, "Cert failure received: %d ;
expected failure mask: %d\n", failures, expected_failures);

/* We expect an error from the certificate validation function. */

if (failures & expected_failures)

return APR_SUCCESS;

Post by Justin Erenkrantz
Thanks. I expect that this might be due to the last change - erroring

out on

Post by Justin Erenkrantz
an expired self-signed root cert. Though I thought we didn’t check in a

root

Post by Justin Erenkrantz
cert for our test chain...could Debian’s packaging be including a cert

for

Post by Justin Erenkrantz
testing?

I use create_certs.py from trunk to re-generate the test certificates
every build, otherwise I was running into time bombs with the certs
expiring. Other than that, I don't do anything different than the
normal test process.
The Debian packaging does have some local patches[0] applied to address
issues that have been fixed upstream but not yet released.
[0]: https://sources.debian.org/patches/serf/1.3.9-8/

Post by Justin Erenkrantz
I will try to take a look this week with Debian sid...I assume it has

1.1.1i

Post by Justin Erenkrantz
already? — justin

Yes, it does.
Cheers,
--
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB

[Justin Erenkrantz]

Post by Justin Erenkrantz
It's not clear to me if OpenSSL authors intended to make this breaking
change. On the serf side, we would need to think through what it would
mean to have our app callback return false upon failure in order to
short-circuit the check.
I probably won't get a chance to open an upstream OpenSSL issue today (or
even tomorrow)...

I found the original issue where they changed the behavior and added a
comment there:

https://github.com/openssl/openssl/issues/11297

Cheers. -- justin

[Justin Erenkrantz]

The OpenSSL devs intended this to be a breaking change - but it's not
documented anywhere. Sigh.

I've got a WIP patch against trunk that causes test_ssl to pass - see
below. It also seems to work with OpenSSL 1.1.1h as well as OpenSSL 1.1.1i
/ 1.1.1-stable, AFAICT.

James: can you please give it a try as well?

We've been on the threshold of releasing serf 1.4 for quite some time
now...maybe we should just do that... If this looks reasonable, I'll try
to clean this up and get it into trunk and 1.4.x.

Cheers. -- justin

Index: test/test_serf.h
===================================================================
--- test/test_serf.h (revision 1884847)
+++ test/test_serf.h (working copy)
@@ -296,6 +296,14 @@
handler_baton_t handler_ctx[],
apr_pool_t *pool);

+/* Helper function, runs the client and server context loops and validates
+ that errors were encountered. */
+void
+run_client_and_mock_servers_loops_expect_fail(CuTest *tc, test_baton_t *tb,
+ int num_requests,
+ handler_baton_t
handler_ctx[],
+ apr_pool_t *pool);
+
/* Logs a test suite error with its code location, and return status
SERF_ERROR_ISSUE_IN_TESTSUITE. */
#define REPORT_TEST_SUITE_ERROR()\
Index: test/test_ssl.c
===================================================================
--- test/test_ssl.c (revision 1884847)
+++ test/test_ssl.c (working copy)
@@ -465,9 +465,10 @@

tb->result_flags |= TEST_RESULT_SERVERCERTCB_CALLED;

+ test__log(TEST_VERBOSE, __FILE__, "Cert failure received: %d ;
expected failure mask: %d\n", failures, expected_failures);
/* We expect an error from the certificate validation function. */
if (failures & expected_failures)
- return APR_SUCCESS;
+ return APR_EGENERAL;
else
return REPORT_TEST_SUITE_ERROR();
}
@@ -532,8 +533,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
}

/* Validate that connecting to a SSLv2 only server fails. */
@@ -1121,8 +1122,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
CuAssertTrue(tc, tb->result_flags & TEST_RESULT_SERVERCERTCB_CALLED);

}
@@ -1165,8 +1166,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
CuAssertTrue(tc, tb->result_flags & TEST_RESULT_SERVERCERTCB_CALLED);
}

@@ -2095,8 +2096,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
CuAssertTrue(tc, tb->result_flags & TEST_RESULT_SERVERCERTCB_CALLED);
}

@@ -2138,8 +2139,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
CuAssertTrue(tc, tb->result_flags & TEST_RESULT_SERVERCERTCB_CALLED);
}

@@ -2181,8 +2182,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
CuAssertTrue(tc, tb->result_flags & TEST_RESULT_SERVERCERTCB_CALLED);
}

@@ -2310,8 +2311,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- run_client_and_mock_servers_loops_expect_ok(tc, tb, num_requests,
- handler_ctx, tb->pool);
+ run_client_and_mock_servers_loops_expect_fail(tc, tb, num_requests,
+ handler_ctx, tb->pool);
#endif /* OPENSSL_NO_TLSEXT */
}

Index: test/test_util.c
===================================================================
--- test/test_util.c (revision 1884847)
+++ test/test_util.c (working copy)
@@ -561,6 +561,19 @@
CuAssertIntEquals(tc, num_requests, tb->handled_requests->nelts);
}

+void
+run_client_and_mock_servers_loops_expect_fail(CuTest *tc, test_baton_t *tb,
+ int num_requests,
+ handler_baton_t
handler_ctx[],
+ apr_pool_t *pool)
+{
+ apr_status_t status;
+
+ status = run_client_and_mock_servers_loops(tb, num_requests,
handler_ctx,
+ pool);
+ CuAssertIntEquals_Msg(tc, serf_error_string(status), APR_EGENERAL,
status);
+}
+
void setup_test_mock_server(test_baton_t *tb)
{
if (!tb->mh) /* TODO: move this to test_setup */

Post by Justin Erenkrantz

Post by Justin Erenkrantz
It's not clear to me if OpenSSL authors intended to make this breaking
change. On the serf side, we would need to think through what it would
mean to have our app callback return false upon failure in order to
short-circuit the check.
I probably won't get a chance to open an upstream OpenSSL issue today (or
even tomorrow)...

I found the original issue where they changed the behavior and added a
https://github.com/openssl/openssl/issues/11297
Cheers. -- justin

[James McCoy]

Happy New Year!

Post by Justin Erenkrantz
The OpenSSL devs intended this to be a breaking change - but it's not
documented anywhere.  Sigh.
I've got a WIP patch against trunk that causes test_ssl to pass - see below. 
It also seems to work with OpenSSL 1.1.1h as well as OpenSSL 1.1.1i /
1.1.1-stable, AFAICT.
James: can you please give it a try as well?

Yes, I can confirm this fixes test_ssl_handshake on trunk. There's
enough difference between trunk and branches/1.3.x that it doesn't apply
cleanly there.

Any chance you would be able to make a patch for 1.3.x? Although a 1.4
release would be nice, it's a bit late in the Debian release cycle to
upload a major new version.

A targeted fix for the test suite would address the immediate issue,
though.

Cheers,

[Justin Erenkrantz]

Sadly, my Debian sid box ran into other issues and is currently
inaccessible.

I *think* that this would address the 1.3.x test issues, but 1.3.x doesn't
build on Mac OS for me for other reasons...so, let me know how it goes?
=) -- justin

Index: test/test_context.c
===================================================================
--- test/test_context.c (revision 1885525)
+++ test/test_context.c (working copy)
@@ -1138,7 +1138,7 @@

/* We expect an error from the certificate validation function. */
if (failures & expected_failures)
- return APR_SUCCESS;
+ return APR_EGENERAL;
else
return SERF_ERROR_ISSUE_IN_TESTSUITE;
}
@@ -1206,8 +1206,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- test_helper_run_requests_expect_ok(tc, tb, num_requests, handler_ctx,
- test_pool);
+ test_helper_run_requests_expect_fail(tc, tb, num_requests, handler_ctx,
+ test_pool);
}

/* Set up the ssl context with the CA and root CA certificates needed for
@@ -1774,8 +1774,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- test_helper_run_requests_expect_ok(tc, tb, num_requests, handler_ctx,
- test_pool);
+ test_helper_run_requests_expect_fail(tc, tb, num_requests, handler_ctx,
+ test_pool);
}

/* Validate that the expired certificate is reported as failure in the
@@ -1820,8 +1820,8 @@

create_new_request(tb, &handler_ctx[0], "GET", "/", 1);

- test_helper_run_requests_expect_ok(tc, tb, num_requests, handler_ctx,
- test_pool);
+ test_helper_run_requests_expect_fail(tc, tb, num_requests, handler_ctx,
+ test_pool);
}


Index: test/test_serf.h
===================================================================
--- test/test_serf.h (revision 1885525)
+++ test/test_serf.h (working copy)
@@ -239,6 +239,12 @@
int num_requests,
handler_baton_t handler_ctx[],
apr_pool_t *pool);
+void
+test_helper_run_requests_expect_fail(CuTest *tc, test_baton_t *tb,
+ int num_requests,
+ handler_baton_t handler_ctx[],
+ apr_pool_t *pool);
+
serf_bucket_t* accept_response(serf_request_t *request,
serf_bucket_t *stream,
void *acceptor_baton,
Index: test/test_util.c
===================================================================
--- test/test_util.c (revision 1885525)
+++ test/test_util.c (working copy)
@@ -461,6 +461,19 @@
CuAssertIntEquals(tc, num_requests, tb->handled_requests->nelts);
}

+void
+test_helper_run_requests_expect_fail(CuTest *tc, test_baton_t *tb,
+ int num_requests,
+ handler_baton_t handler_ctx[],
+ apr_pool_t *pool)
+{
+ apr_status_t status;
+
+ status = test_helper_run_requests_no_check(tc, tb, num_requests,
+ handler_ctx, pool);
+ CuAssertIntEquals(tc, APR_EGENERAL, status);
+}
+
serf_bucket_t* accept_response(serf_request_t *request,
serf_bucket_t *stream,
void *acceptor_baton,

Post by James McCoy
Happy New Year!

Post by Justin Erenkrantz
The OpenSSL devs intended this to be a breaking change - but it's not
documented anywhere. Sigh.
I've got a WIP patch against trunk that causes test_ssl to pass - see

below.

Post by Justin Erenkrantz
It also seems to work with OpenSSL 1.1.1h as well as OpenSSL 1.1.1i /
1.1.1-stable, AFAICT.
James: can you please give it a try as well?

Yes, I can confirm this fixes test_ssl_handshake on trunk. There's
enough difference between trunk and branches/1.3.x that it doesn't apply
cleanly there.

Any chance you would be able to make a patch for 1.3.x? Although a 1.4
release would be nice, it's a bit late in the Debian release cycle to
upload a major new version.
A targeted fix for the test suite would address the immediate issue,
though.
Cheers,
--
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB

[James McCoy]

Sadly, my Debian sid box ran into other issues and is currently inaccessible.
I *think* that this would address the 1.3.x test issues, but 1.3.x doesn't
build on Mac OS for me for other reasons...so, let me know how it goes?  =)  --

Success!

Thanks,

[Debian Bug Tracking System]

tag -1 pending

Bug #978353 [src:serf] serf: FTBFS: tests failed with OpenSSL 1.1.1i
Added tag(s) pending.

[James McCoy]

Control: tag -1 pending

Hello,

Bug #978353 in serf reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/jamessan/serf/-/commit/17baa8f922362010416a91dadbcada79c2c2d447

------------------------------------------------------------------------
Include upstream patch to fix OpenSSL 1.1.1i compat

Closes: #978353
Signed-off-by: James McCoy <***@debian.org>
------------------------------------------------------------------------

(this message was generated automatically)

[Debian Bug Tracking System]

Your message dated Sat, 16 Jan 2021 17:35:18 +0000
with message-id <E1l0pTq-0004JW-***@fasolo.debian.org>
and subject line Bug#978353: fixed in serf 1.3.9-9
has caused the Debian Bug report #978353,
regarding serf: FTBFS: tests failed with OpenSSL 1.1.1i
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)