Debugging a coredump with gdb-multiarch

This post is a small reminder on how to use the gdb-multiarch on my Ubuntu machine to debug coredumps from other arches/machines. I need to write it because the information I found in the internet does not always lead to the whole stack trace being available even when all the symbols and external sysroot is available.

One of the suggestions is to use:

$ gdb-multiarch -e <cross-executable> -c<core>

[...]License[...]
[New LWP 5246]

warning: `/lib/libgcc_s.so.1': Shared library architecture i386:x86-64 is not compatible with target architecture armv7.

warning: .dynamic section for "/lib/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)

warning: Could not load shared library symbols for 11 libraries, e.g. /lib/libpthread.so.0.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/usr/bin/fluent-bit --config=/etc/fluent-bit/fluent-bit.conf'.
Program terminated with signal SIGABRT, Aborted.
#0  0xac9d10a6 in ?? ()

As can be seen it is pretty useless and gdb-multiarch gets confused. Unfortunately without running set sysroot <path to sysroot>, gdb will be unable to get a whole stacktrace. But it is worse. Even if you set the sysroot properly, it will not for some reason reload the debug symbols for the crossexecutable. Instead it will only load for the shared libraries, which is often quite useless and leaves the backtrace like the following:

# continuation of the previous session
(gdb) set sysroot /tmp/tmp/my-root/
Reading symbols from /tmp/tmp/my-root/lib/libpthread.so.0...
Reading symbols from /tmp/tmp/my-root/lib/.debug/libpthread-2.31.so...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /tmp/tmp/my-root/lib/libm.so.6...
Reading symbols from /tmp/tmp/my-root/lib/.debug/libm-2.31.so...
[...]
(gdb) bt
#0  __libc_do_syscall () at libc-do-syscall.S:49
#1  0xac9df676 in __libc_signal_restore_set (set=0xbeea6d78) at ../sysdeps/unix/sysv/linux/internal-signals.h:86
#2  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3  0xac9d0be2 in __GI_abort () at abort.c:79
#4  0x0750a49e in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Pretty useless as well, and even misleading.

The correct way to do it is:

gdb-multiarch <cross-executable>
(gdb) set sysroot <path to rootfs+debugsymbols>
(gdb) core <path to core>
[...]propper backtracke shows[...]

Only then will the whole stack trace be available.

PS: Often gdb will only show the stack trace of the thread that crashed by default, but there might be more threads worth inspecting. To show the stack trace of all the threads just do the following in gdb:

(gdb) thread apply all bt