Bug#1079329: systemd(?) still creates /lib64 on arm64

[Helmut Grohne]

Package: systemd
Severity: serious
Justification: causes base-files.preinst to fail
User: ***@debian.org
Usertags: dep17
Control: block 1077866 by -1
X-Debbugs-Cc: ***@debian.org

Hi,

we're not 100% sure yet that this is systemd at cause, but it seems so
likely that I'm filing the bug here initially for further investigation.

Jochen observed that an arm64 machine with dracut would have a /lib64 ->
usr/lib symbolic link. The presence of such a link makes
base-files.preinst fail. debvm can be used to reproduce the problem:

$ debvm-create -- --architecture=arm64 --include=dracut
# creates a ./rootfs.ext4
$ debvm-run --append "rd.shell rd.break=pre-pivot"

That --append should drop you into a shell right before dracut does
pivot_root and hands off to systemd. At that time, now /sysroot/lib64
exists. Once the machine has booted /lib64 exists. At this time, the
most plausible hypothesis is that systemd still creates it despite
having been improved earlier.

Would you happen to know a good way to validate this hypothesis?

A possible approach could be enabling linux-level tracing and hooking on
the symlinkat syscall to see which processes issue it, but it could be
many. Any idea for narrowing down the component?

Thanks in advance

Helmut

[Debian Bug Tracking System]

Post by Helmut Grohne
block 1077866 by -1

Bug #1077866 [base-files] base-files: need workaround for incorrect /lib64 symlink
1077866 was not blocked by any bugs.
1077866 was not blocking any bugs.
Added blocking bug(s) of 1077866: 1079329

[Debian Bug Tracking System]

tags -1 moreinfo

Bug #1079329 [systemd] systemd(?) still creates /lib64 on arm64
Added tag(s) moreinfo.

[Luca Boccassi]

Control: tags -1 moreinfo

Post by Helmut Grohne
Package: systemd
Severity: serious
Justification: causes base-files.preinst to fail
Usertags: dep17
Control: block 1077866 by -1
Hi,
we're not 100% sure yet that this is systemd at cause, but it seems so
likely that I'm filing the bug here initially for further

investigation.

Post by Helmut Grohne
Jochen observed that an arm64 machine with dracut would have a /lib64 ->
usr/lib symbolic link. The presence of such a link makes
$ debvm-create -- --architecture=arm64 --include=dracut
# creates a ./rootfs.ext4
$ debvm-run --append "rd.shell rd.break=pre-pivot"
That --append should drop you into a shell right before dracut does
pivot_root and hands off to systemd. At that time, now /sysroot/lib64
exists. Once the machine has booted /lib64 exists. At this time, the
most plausible hypothesis is that systemd still creates it despite
having been improved earlier.
Would you happen to know a good way to validate this hypothesis?
A possible approach could be enabling linux-level tracing and hooking on
the symlinkat syscall to see which processes issue it, but it could be
many. Any idea for narrowing down the component?
Thanks in advance
Helmut

Hi,

That's correct, as the dynamic loader is at /usr/lib/ld-linux-
aarch64.so.1 and the symlink will be created to the first directory
between /usr/lib64 and /usr/lib that exists and has the loader, in that
order.

So it seems to me either base-files needs to be able to deal with this,
or the loader should be symlinked in /usr/lib64 (like Fedora does)?

[Helmut Grohne]

Control: tags -1 - moreinfo

What further information do you require exactly?

Post by Luca Boccassi
That's correct, as the dynamic loader is at /usr/lib/ld-linux-
aarch64.so.1 and the symlink will be created to the first directory
between /usr/lib64 and /usr/lib that exists and has the loader, in that
order.
So it seems to me either base-files needs to be able to deal with this,
or the loader should be symlinked in /usr/lib64 (like Fedora does)?

I see no technical reason for base-files to deal with packages messing
with /lib64 symlinks. The pending policy change forbids doing so and you
seconded it.

I also see no reason to add the loader to /usr/lib64 as nothing
references that path nor any reason for having /lib64 on arm64 in the
first place.

Please clarify what kind of problem is being solved on the systemd side
in attempting to manage a /lib64 symlink. I can partially answer this
question myself: systemd supports "hermetic /usr" which amounts to
bootstrapping an installation from a bare /usr tree. As such, it
contains code to manage locations outside /usr and that happens to
include aliasing symlinks. What I do not see is why it has to manage
/lib64 on arm64. It could simply stop doing so and all would be good as
far as I can see. Let us pretend systemd upstream were to drop the code
for creating /lib64 on arm64. Which users would complain?

Helmut

[Debian Bug Tracking System]

tags -1 - moreinfo

Bug #1079329 [systemd] systemd(?) still creates /lib64 on arm64
Removed tag(s) moreinfo.

[Helmut Grohne]

Hi Luca,

Yes that's the use case, just as you defined it. The problem is that
these links need to exist, and they need to be created at runtime in
an ephemeral root, and the target needs to be figured out <somehow>.
The best we have so far, that has been working fine over the years, is
to target it to where the loader is located. If you think there's a
better way, that works everywhere, please feel free to propose such a
change upstream, via a pull request ideally, or at least via an issue
(but please do note that PRs are better). In the end I think that's
the right place to discuss such details, rather than a downstream bug.

Thanks for confirming. Please go ahead and remove the creation of /lib64
from systemd then.

Sorry, I do not follow. As mentioned, this cannot just be removed.

You appeared to agree with my general description of the mechanism. You
detailed that systemd needs to create these links when needed at
runtime, but for arm64 there is no known nor documented need for /lib64,
so I read that as you agreeing the /lib64 link on arm64 should be
removed upstream. If you believe that it cannot be removed, please give
details why you think so.

If you insist on cargo culting this code, there is another way. We know
for a fact that in the hermetic-/usr case, there is no package manager
on whose toes systemd could step. In particular, there is no
/var/lib/dpkg. So systemd could simply check that location and skip
bothering with aliasing links if it exists. Thus we would turn the
broken code in systemd into dead code on Debian systems until you
transform the installation hermetic and then it would return.

Helmut

[Helmut Grohne]

Hi Luca,

Sorry, but I am still not following. I do not understand, why would
lib64 not be needed? It's a 64bit architecture, no? If I download a
https://fedoraproject.org/server/download
total 16
dr-xr-xr-x. 2 root root 6 Jan 24 2024 afs
lrwxrwxrwx. 1 root root 7 Jan 24 2024 bin -> usr/bin
drwxr-xr-x. 2 root root 6 Apr 15 00:02 boot
drwxr-xr-x. 2 root root 6 Apr 15 00:02 dev
drwxr-xr-x. 121 root root 8192 Apr 15 00:26 etc
drwxr-xr-x. 2 root root 6 Jan 24 2024 home
lrwxrwxrwx. 1 root root 7 Jan 24 2024 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Jan 24 2024 lib64 -> usr/lib64
drwxr-xr-x. 2 root root 6 Jan 24 2024 media
drwxr-xr-x. 2 root root 6 Jan 24 2024 mnt
drwxr-xr-x. 2 root root 6 Jan 24 2024 opt
drwxr-xr-x. 2 root root 6 Apr 15 00:02 proc
dr-xr-x---. 3 root root 126 Apr 15 00:27 root
drwxr-xr-x. 2 root root 6 Apr 15 00:02 run
lrwxrwxrwx. 1 root root 8 Jan 24 2024 sbin -> usr/sbin
drwxr-xr-x. 2 root root 6 Jan 24 2024 srv
drwxr-xr-x. 2 root root 6 Apr 15 00:02 sys
drwxrwxrwt. 2 root root 6 Apr 15 00:02 tmp
drwxr-xr-x. 12 root root 144 Apr 15 00:03 usr
drwxr-xr-x. 20 root root 4096 Apr 15 00:12 var

Sorry, but I am still not following. I do not understand, why would
lib64 be needed. You detail how it is being created, but there is no
implication from creation to need. We discover unused files all the
time. If I create a Debian arm64 sid chroot using mmdebstrap, it is not
there. Quite obviously, there is no universal need for /lib64 on 64bit
architectures. There still may be a conditional need whose nature we do
not understand yet.

There is one obvious way in which multilib aliasing links are required.
That's when the path of the dynamic loader requires them. We have a wiki
page documenting this: https://wiki.debian.org/ArchitectureSpecificsMemo

* amd64 -> lib64
* loong64 -> lib64
* mips64el -> lib64
* ppc64 -> lib64
* ppc64el -> lib64
* s390x -> lib64
* sparc64 -> lib64
* x32 -> libx32

These are the cases where systemd really should check for aliasing links
of multilib directories and create them if missing. In particular, it
should only ever point them at the corresponding location. Even your
Fedora example has /lib64 point at usr/lib64 and not usr/lib. It is the
/lib64 -> usr/lib link that this bug takes issue with and that systemd
must not create.

Generally speaking, the use of multilib paths for dynamic loaders is
deprecated and new architectures should use plain /lib and use a unique
name therein. loong64 is recent, but they utterly failed at
communicating and hence followed the bad example. In particular, arm64
is not on the list.

As a consequence, I think systemd needs to change in two ways:
* It should never create non-matching aliasing links (such as /lib64 ->
usr/lib).
* When the link target (e.g /usr/lib64) does not exist, it should not
create the aliasing symlink.

If it follows these two rules, it should be good even creating /lib64 on
arm64 even on Debian with no matching on the package manager. It'll also
continue creating the aliasing link on Fedora whose existence (though
not need) you demonstrated.

Helmut

[Luca Boccassi]

Post by Helmut Grohne
Hi Luca,

Sorry, but I am still not following. I do not understand, why would
lib64 not be needed? It's a 64bit architecture, no? If I download a
https://fedoraproject.org/server/download
total 16
dr-xr-xr-x. 2 root root 6 Jan 24 2024 afs
lrwxrwxrwx. 1 root root 7 Jan 24 2024 bin -> usr/bin
drwxr-xr-x. 2 root root 6 Apr 15 00:02 boot
drwxr-xr-x. 2 root root 6 Apr 15 00:02 dev
drwxr-xr-x. 121 root root 8192 Apr 15 00:26 etc
drwxr-xr-x. 2 root root 6 Jan 24 2024 home
lrwxrwxrwx. 1 root root 7 Jan 24 2024 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Jan 24 2024 lib64 -> usr/lib64
drwxr-xr-x. 2 root root 6 Jan 24 2024 media
drwxr-xr-x. 2 root root 6 Jan 24 2024 mnt
drwxr-xr-x. 2 root root 6 Jan 24 2024 opt
drwxr-xr-x. 2 root root 6 Apr 15 00:02 proc
dr-xr-x---. 3 root root 126 Apr 15 00:27 root
drwxr-xr-x. 2 root root 6 Apr 15 00:02 run
lrwxrwxrwx. 1 root root 8 Jan 24 2024 sbin -> usr/sbin
drwxr-xr-x. 2 root root 6 Jan 24 2024 srv
drwxr-xr-x. 2 root root 6 Apr 15 00:02 sys
drwxrwxrwt. 2 root root 6 Apr 15 00:02 tmp
drwxr-xr-x. 12 root root 144 Apr 15 00:03 usr
drwxr-xr-x. 20 root root 4096 Apr 15 00:12 var

Sorry, but I am still not following. I do not understand, why would
lib64 be needed. You detail how it is being created, but there is no
implication from creation to need. We discover unused files all the
time. If I create a Debian arm64 sid chroot using mmdebstrap, it is not
there. Quite obviously, there is no universal need for /lib64 on 64bit
architectures. There still may be a conditional need whose nature we do
not understand yet.

But how can you be sure that nothing depends on it? How have you
checked that it's not needed? This logic has been there for many years

Post by Helmut Grohne
There is one obvious way in which multilib aliasing links are required.
That's when the path of the dynamic loader requires them. We have a wiki
page documenting this: https://wiki.debian.org/ArchitectureSpecificsMemo
* amd64 -> lib64
* loong64 -> lib64
* mips64el -> lib64
* ppc64 -> lib64
* ppc64el -> lib64
* s390x -> lib64
* sparc64 -> lib64
* x32 -> libx32
These are the cases where systemd really should check for aliasing links
of multilib directories and create them if missing. In particular, it
should only ever point them at the corresponding location. Even your
Fedora example has /lib64 point at usr/lib64 and not usr/lib. It is the
/lib64 -> usr/lib link that this bug takes issue with and that systemd
must not create.
Generally speaking, the use of multilib paths for dynamic loaders is
deprecated and new architectures should use plain /lib and use a unique
name therein. loong64 is recent, but they utterly failed at
communicating and hence followed the bad example. In particular, arm64
is not on the list.
* It should never create non-matching aliasing links (such as /lib64 ->
usr/lib).

As above

Post by Helmut Grohne
* When the link target (e.g /usr/lib64) does not exist, it should not
create the aliasing symlink.

This is already the case