[Bug 229222] 11.2-PRERELEASE panic-General Protection Fault,

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

Bug ID: 229222
Summary: 11.2-PRERELEASE panic-General Protection Fault,
aesni_encrypt_cbc implicated
Product: Base System
Version: 11.2-STABLE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: ***@FreeBSD.org
Reporter: ***@heuristicsystems.com.au

Platform:CPU: Intel(R) Xeon(R) CPU E3-1230L v3 @ 1.80GHz (1795.88-MHz K8-class
CPU)
Filesystems: ufs2
OS: FreeBSD 11.2-PRERELEASE r335442M amd64 hqdev-amd64-smp-vga 1102501
1102500
Relevant apps:
openssh-portable-7.7.p1_2,1
libressl-2.6.4_1
all ports are built using binutils-2.30_3,1 which is why its installed

Observation:
- System boots, runs multiple sambas, sendmail, squid, ss5,...
- Unpredictable crashes, nothing in logs.
- seems related to ssh'ing outbound, unsure because computer triggers
connections/commands to firewalls
- sshd connections use chacha20.
- crashes have been occuring since May, we usually rebuild stable fortnightly.
- after a failed test we revert to FreeBSD 11.1-STABLE r329008M amd64
hqdev-amd64-smp-vga 1101509 1102500

- this most recent crash uses revision 335442, however we reverted parts of
llvm back to 333070 as we wrongly suspected that as the cause. Using:
svnlite update /usr/src
svnlite update -r 333070 /usr/src/contrib/llvm
/usr/src/lib/clang/freebsd_cc_version.h /usr/src/lib/clang/libllvm/Makefile
- the systems were built using

~# "/usr/bin/cc" -v
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM
6.0.0)
Target: x86_64-unknown-freebsd11.2
Thread model: posix
InstalledDir: /usr/bin

~# "/usr/bin/ld" -v
GNU ld 2.17.50 [FreeBSD] 2007-07-03

- The kernel was placed in the normal location, /boot/kernel,
but as this was a temporary/test build it was destined for /pd2/tmp/destP.
/smallblocks is where we place src and ports, ie /usr/src equivalent to
/smallblocks/src. The minidump follows:

# kgdb /pd2/tmp/destP/usr/lib/debug/boot/kernel/kernel.debug
/var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[21419]
[21419]
[21419] Fatal trap 9: general protection fault while in kernel mode
[21419] cpuid = 7; apic id = 07
[21419] instruction pointer = 0x20:0xffffffff80df76ee
[21419] stack pointer = 0x0:0xfffffe0688f57df8
[21419] frame pointer = 0x0:0xfffffe0688f57e88
[21419] code segment = base 0x0, limit 0xfffff, type 0x1b
[21419] = DPL 0, pres 1, long 1, def32 0, gran 1
[21419] processor eflags = interrupt enabled, resume, IOPL = 0
[21419] current process = 47820 (ssh)
[21419] trap number = 9
[21419] panic: general protection fault
[21419] cpuid = 7
[21419] Uptime: 5h56m59s
[21419] Dumping 2135 out of 24501
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from
/pd2/tmp/destP/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug...done.
Loaded symbols for /pd2/tmp/destP/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug
Reading symbols from
/pd2/tmp/destP/usr/lib/debug/boot/kernel/cpufreq.ko.debug...done.
Loaded symbols for /pd2/tmp/destP/usr/lib/debug/boot/kernel/cpufreq.ko.debug
Reading symbols from
/pd2/tmp/destP/usr/lib/debug/boot/kernel/coretemp.ko.debug...done.
Loaded symbols for /pd2/tmp/destP/usr/lib/debug/boot/kernel/coretemp.ko.debug
Reading symbols from
/pd2/tmp/destP/usr/lib/debug/boot/kernel/uplcom.ko.debug...done.
Loaded symbols for /pd2/tmp/destP/usr/lib/debug/boot/kernel/uplcom.ko.debug
Reading symbols from
/pd2/tmp/destP/usr/lib/debug/boot/kernel/ucom.ko.debug...done.
Loaded symbols for /pd2/tmp/destP/usr/lib/debug/boot/kernel/ucom.ko.debug
#0 doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
315 dumptid = curthread->td_tid;
(kgdb) list 0xffffffff80df76ee
Function "0xffffffff80df76ee" not defined.
(kgdb) list *0xffffffff80df76ee
0xffffffff80df76ee is in aesni_encrypt_cbc
(/smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64).
59 {
60 __m128i tot, ivreg;
61 size_t i;
62
63 len /= AES_BLOCK_LEN;
64 ivreg = _mm_loadu_si128((const __m128i *)iv);
65 for (i = 0; i < len; i++) {
66 tot = aesni_enc(rounds - 1, key_schedule,
67 _mm_loadu_si128((const __m128i *)from) ^ ivreg);
68 ivreg = tot;
Current language: auto; currently minimal
(kgdb)
69 _mm_storeu_si128((__m128i *)to, tot);
70 from += AES_BLOCK_LEN;
71 to += AES_BLOCK_LEN;
72 }
73 }
74
75 void
76 aesni_decrypt_cbc(int rounds, const void *key_schedule, size_t len,
77 uint8_t *buf, const uint8_t iv[static AES_BLOCK_LEN])
78 {
(kgdb) Quit
(kgdb) backtrace
#0 doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
#1 0xffffffff8080973b in kern_reboot (howto=260) at
/smallblocks/src/sys/kern/kern_shutdown.c:383
#2 0xffffffff80809f7c in vpanic (fmt=0xffffffff80e8993a "%s",
ap=0xfffffe0688f57900)
at /smallblocks/src/sys/kern/kern_shutdown.c:776
#3 0xffffffff80809d60 in panic (fmt=0xffffffff80e8993a "%s")
at /smallblocks/src/sys/kern/kern_shutdown.c:707
#4 0xffffffff80de9597 in trap_fatal (frame=0xfffffe0688f57d30, eva=0)
at /smallblocks/src/sys/amd64/amd64/trap.c:875
#5 0xffffffff80de8c1d in trap (frame=0xfffffe0688f57d30) at
/smallblocks/src/sys/amd64/amd64/trap.c:608
#6 0xffffffff80de99e5 in trap_check (frame=0xfffffe0688f57d30)
at /smallblocks/src/sys/amd64/amd64/trap.c:657
#7 0xffffffff80dbe7de in calltrap () at
/smallblocks/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff80df76ee in aesni_encrypt_cbc (rounds=10,
key_schedule=0xfffff8001446cc00, len=2,
from=0xfffff8001408bb80 "\016\005", to=0xfffff8001408bb80 "\016\005",
iv=0xfffffe0688f57fa8 "\003\024|Dc▒\0200a%▒\032!▒\016")
at /smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64
#9 0xffffffff80defb0b in aesni_cipher_process (ses=0xfffff8001446cc00,
enccrd=0xfffff803905493f0,
authcrd=0x0, crp=0xfffff80390258bb0) at
/smallblocks/src/sys/crypto/aesni/aesni.c:606
#10 0xffffffff80def050 in aesni_process (dev=0xfffff8000567de00,
crp=0xfffff80390258bb0, hint=0)
at /smallblocks/src/sys/crypto/aesni/aesni.c:413
#11 0xffffffff80c79eb5 in CRYPTODEV_PROCESS (dev=0xfffff8000567de00,
op=0xfffff80390258bb0, flags=0)
at cryptodev_if.h:53
#12 0xffffffff80c78894 in crypto_invoke (cap=0xfffff80005023078,
crp=0xfffff80390258bb0, hint=0)
at /smallblocks/src/sys/opencrypto/crypto.c:1083
#13 0xffffffff80c7861a in crypto_dispatch (crp=0xfffff80390258bb0)
at /smallblocks/src/sys/opencrypto/crypto.c:844
#14 0xffffffff80c7c9fa in cryptodev_op (cse=0xfffff800c4669900,
cop=0xfffffe0688f583f0,
active_cred=0xfffff805bc031700, td=0xfffff802b9250620)
at /smallblocks/src/sys/opencrypto/cryptodev.c:849
#15 0xffffffff80c7ba9f in cryptof_ioctl (fp=0xfffff8001434b0f0, cmd=3223085927,
data=0xfffffe0688f58720, active_cred=0xfffff805bc031700,
td=0xfffff802b9250620)
at /smallblocks/src/sys/opencrypto/cryptodev.c:633
#16 0xffffffff80898cec in fo_ioctl (fp=0xfffff8001434b0f0, com=3223085927,
data=0xfffffe0688f58720,
active_cred=0xfffff805bc031700, td=0xfffff802b9250620) at file.h:323
#17 0xffffffff80898ae4 in kern_ioctl (td=0xfffff802b9250620, fd=5,
com=3223085927,
data=0xfffffe0688f58720 "") at /smallblocks/src/sys/kern/sys_generic.c:836
#18 0xffffffff808986aa in sys_ioctl (td=0xfffff802b9250620,
uap=0xfffffe0688f58878)
at /smallblocks/src/sys/kern/sys_generic.c:745
#19 0xffffffff8030e321 in freebsd32_ioctl (td=0xfffff802b9250620,
uap=0xfffff802b9250b58)
at /smallblocks/src/sys/compat/freebsd32/freebsd32_ioctl.c:470
#20 0xffffffff80e315be in syscallenter (td=0xfffff802b9250620) at
subr_syscall.c:132
#21 0xffffffff80e30f3f in ia32_syscall (frame=0xfffffe0688f58ab8)
at /smallblocks/src/sys/amd64/ia32/ia32_syscall.c:218
#22 0xffffffff80dbf796 in int0x80_syscall_common () at ia32_exception.S:76
#23 0x0000000000000000 in ?? ()
(kgdb) quit

I have retained the core dump and debug symbols if further information
required, or compressed to 208MB upon request.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

Mark Linimon <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Keywords| |regression
CC| |***@FreeBSD.org

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

8 0xffffffff80df76ee in aesni_encrypt_cbc (rounds=10, key_schedule=0xfffff8001446cc00, len=2,

If this is accurate (may not be), this is a bogus invocation of AES-CBC (and
probably should be validated higher in OCF). AES-CBC is a block cipher; it
operates in AES block unit sizes (multiples of 16 bytes).

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

Konstantin Belousov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #2 from Konstantin Belousov <***@FreeBSD.org> ---
I agree that the length is weird.

Disassemble the faulting instruction, and show the registers content.
From kgdb, perhaps you need one installed from ports,
(kgdb) disassemble 0xffffffff80df76ee
(kgdb) p/x *(struct frame *)0xfffffe0688f57d30

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #3 from ***@heuristicsystems.com.au ---
(In reply to Konstantin Belousov from comment #2)
Thank-you for looking into this. I should clarify an earlier observation. It
appears that the ssh connection was initiated from within an i386 jail that
resides on the amd64 base system. i386 jails run openssl and not libressl.

To you request Konstantin
(kgdb) disassemble 0xffffffff80df76ee
Dump of assembler code for function aesni_encrypt_cbc:
0xffffffff80df76b0 <aesni_encrypt_cbc+0>: push %rbp
0xffffffff80df76b1 <aesni_encrypt_cbc+1>: mov %rsp,%rbp
0xffffffff80df76b4 <aesni_encrypt_cbc+4>: sub $0x90,%rsp
0xffffffff80df76bb <aesni_encrypt_cbc+11>: mov %edi,-0x2c(%rbp)
0xffffffff80df76be <aesni_encrypt_cbc+14>: mov %rsi,-0x38(%rbp)
0xffffffff80df76c2 <aesni_encrypt_cbc+18>: mov %rdx,-0x40(%rbp)
0xffffffff80df76c6 <aesni_encrypt_cbc+22>: mov %rcx,-0x48(%rbp)
0xffffffff80df76ca <aesni_encrypt_cbc+26>: mov %r8,-0x50(%rbp)
0xffffffff80df76ce <aesni_encrypt_cbc+30>: mov %r9,-0x58(%rbp)
0xffffffff80df76d2 <aesni_encrypt_cbc+34>: mov -0x40(%rbp),%rcx
0xffffffff80df76d6 <aesni_encrypt_cbc+38>: shr $0x4,%rcx
0xffffffff80df76da <aesni_encrypt_cbc+42>: mov %rcx,-0x40(%rbp)
0xffffffff80df76de <aesni_encrypt_cbc+46>: mov -0x58(%rbp),%rcx
0xffffffff80df76e2 <aesni_encrypt_cbc+50>: mov %rcx,-0x28(%rbp)
0xffffffff80df76e6 <aesni_encrypt_cbc+54>: mov -0x28(%rbp),%rcx
0xffffffff80df76ea <aesni_encrypt_cbc+58>: movdqu (%rcx),%xmm0
0xffffffff80df76ee <aesni_encrypt_cbc+62>: movdqa %xmm0,-0x80(%rbp)
0xffffffff80df76f3 <aesni_encrypt_cbc+67>: movq $0x0,-0x88(%rbp)
0xffffffff80df76fe <aesni_encrypt_cbc+78>: mov -0x88(%rbp),%rax
0xffffffff80df7705 <aesni_encrypt_cbc+85>: cmp -0x40(%rbp),%rax
0xffffffff80df7709 <aesni_encrypt_cbc+89>: jae 0xffffffff80df7795
<aesni_encrypt_cbc+229>
0xffffffff80df770f <aesni_encrypt_cbc+95>: mov -0x2c(%rbp),%eax
0xffffffff80df7712 <aesni_encrypt_cbc+98>: sub $0x1,%eax
0xffffffff80df7715 <aesni_encrypt_cbc+101>: mov -0x38(%rbp),%rcx
0xffffffff80df7719 <aesni_encrypt_cbc+105>: mov -0x48(%rbp),%rdx
0xffffffff80df771d <aesni_encrypt_cbc+109>: mov %rdx,-0x8(%rbp)
0xffffffff80df7721 <aesni_encrypt_cbc+113>: mov -0x8(%rbp),%rdx
0xffffffff80df7725 <aesni_encrypt_cbc+117>: movdqu (%rdx),%xmm0
0xffffffff80df7729 <aesni_encrypt_cbc+121>: pxor -0x80(%rbp),%xmm0
0xffffffff80df772e <aesni_encrypt_cbc+126>: mov %eax,%edi
0xffffffff80df7730 <aesni_encrypt_cbc+128>: mov %rcx,%rsi
0xffffffff80df7733 <aesni_encrypt_cbc+131>: callq 0xffffffff80df77a0
<aesni_enc>
0xffffffff80df7738 <aesni_encrypt_cbc+136>: movdqa %xmm0,-0x70(%rbp)
0xffffffff80df773d <aesni_encrypt_cbc+141>: movdqa -0x70(%rbp),%xmm0
0xffffffff80df7742 <aesni_encrypt_cbc+146>: movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7747 <aesni_encrypt_cbc+151>: mov -0x50(%rbp),%rcx
---Type <return> to continue, or q <return> to quit---
0xffffffff80df774b <aesni_encrypt_cbc+155>: movdqa -0x70(%rbp),%xmm0
0xffffffff80df7750 <aesni_encrypt_cbc+160>: mov %rcx,-0x10(%rbp)
0xffffffff80df7754 <aesni_encrypt_cbc+164>: movdqa %xmm0,-0x20(%rbp)
0xffffffff80df7759 <aesni_encrypt_cbc+169>: movdqa -0x20(%rbp),%xmm0
0xffffffff80df775e <aesni_encrypt_cbc+174>: mov -0x10(%rbp),%rcx
0xffffffff80df7762 <aesni_encrypt_cbc+178>: movdqu %xmm0,(%rcx)
0xffffffff80df7766 <aesni_encrypt_cbc+182>: mov -0x48(%rbp),%rcx
0xffffffff80df776a <aesni_encrypt_cbc+186>: add $0x10,%rcx
0xffffffff80df776e <aesni_encrypt_cbc+190>: mov %rcx,-0x48(%rbp)
0xffffffff80df7772 <aesni_encrypt_cbc+194>: mov -0x50(%rbp),%rcx
0xffffffff80df7776 <aesni_encrypt_cbc+198>: add $0x10,%rcx
0xffffffff80df777a <aesni_encrypt_cbc+202>: mov %rcx,-0x50(%rbp)
0xffffffff80df777e <aesni_encrypt_cbc+206>: mov -0x88(%rbp),%rax
0xffffffff80df7785 <aesni_encrypt_cbc+213>: add $0x1,%rax
0xffffffff80df7789 <aesni_encrypt_cbc+217>: mov %rax,-0x88(%rbp)
0xffffffff80df7790 <aesni_encrypt_cbc+224>: jmpq 0xffffffff80df76fe
<aesni_encrypt_cbc+78>
0xffffffff80df7795 <aesni_encrypt_cbc+229>: add $0x90,%rsp
0xffffffff80df779c <aesni_encrypt_cbc+236>: pop %rbp
0xffffffff80df779d <aesni_encrypt_cbc+237>: retq
End of assembler dump.
Current language: auto; currently minimal
(kgdb) p/x *(struct frame *)0xfffffe0688f57d30
No struct type named frame.

Happy to provide any information that will help. I'm GMT+10, 23:18 local, so
may take awhile.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #4 from Konstantin Belousov <***@FreeBSD.org> ---
(In reply to dewayne from comment #3)
The structure is called trapframe.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #5 from ***@heuristicsystems.com.au ---
(In reply to Konstantin Belousov from comment #4)
(kgdb) p/x *(struct trapframe *)0xfffffe0688f57d30
$1 = {tf_rdi = 0xa, tf_rsi = 0xfffff8001446cc00, tf_rdx = 0x20, tf_rcx =
0xfffffe0688f57fa8,
tf_r8 = 0xfffff8001408bb80, tf_r9 = 0xfffffe0688f57fa8, tf_rax =
0xfffff803905493f0,
tf_rbx = 0xfffffe0688f58698, tf_rbp = 0xfffffe0688f57e88, tf_r10 = 0x28,
tf_r11 = 0x0,
tf_r12 = 0x0, tf_r13 = 0x0, tf_r14 = 0x400000000000080, tf_r15 = 0x0,
tf_trapno = 0x9,
tf_fs = 0x13, tf_gs = 0x1b, tf_addr = 0x0, tf_flags = 0x1, tf_es = 0x3b,
tf_ds = 0x3b,
tf_err = 0x0, tf_rip = 0xffffffff80df76ee, tf_cs = 0x20, tf_rflags = 0x10202,
tf_rsp = 0xfffffe0688f57df8, tf_ss = 0x0}

Thanks Kib, I appreciate your patience, as I'm really just your fingers, its
been 28 years since I had to use a C debugger.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #6 from Konstantin Belousov <***@FreeBSD.org> ---
Created attachment 194505
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=194505&action=edit
MFC r333059

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #7 from ***@heuristicsystems.com.au ---
(In reply to Konstantin Belousov from comment #6)
Unfortunately not a solution - performed svn update, then successfully applied
the patch attachment; rebuilt,installed and running the non-debugging kernel
which has crashed. Need to fsck and build fresh debugging kernel.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #8 from Konstantin Belousov <***@FreeBSD.org> ---
(In reply to dewayne from comment #7)
Crashed at the same place, I expect ? I.e. same instruction in the disassembly
?

The issue is that the stack is not aligned to 16b. The patch I attached should
fixed some case where the stack could be unaligned (but usually not) on the
first C function entry from the exception asm handler. Since it did not
helped, my highest suspect is the compiler.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #9 from ***@heuristicsystems.com.au ---
(In reply to Konstantin Belousov from comment #8)
After performing buildworld, buildkernel & installkernel - I'm trying to induce
a crash which may take a few hours. Yes, I thought that the compiler had a
part to play but I did a partial revert (mentioned in initial PR); which didn't
prevent crashing.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #10 from ***@heuristicsystems.com.au ---
(In reply to Konstantin Belousov from comment #8)
Just crashed, results being:
kgdb /pd2/tmp/destQ/usr/lib/debug/boot/kernel/kernel.debug /var/crash/vmcore.10
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[9024]
[9024]
[9024] Fatal trap 9: general protection fault while in kernel mode
[9024] cpuid = 2; apic id = 02
[9024] instruction pointer = 0x20:0xffffffff80df7abe
[9024] stack pointer = 0x0:0xfffffe0688b39df8
[9024] frame pointer = 0x0:0xfffffe0688b39e88
[9024] code segment = base 0x0, limit 0xfffff, type 0x1b
[9024] = DPL 0, pres 1, long 1, def32 0, gran 1
[9024] processor eflags = interrupt enabled, resume, IOPL = 0
[9024] current process = 53068 (ssh)
[9024] trap number = 9
[9024] panic: general protection fault
[9024] cpuid = 2
[9024] Uptime: 2h30m24s
[9024] Dumping 1861 out of 24501
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from
/pd2/tmp/destQ/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/mac_ifoff.ko.debug
Reading symbols from
/pd2/tmp/destQ/usr/lib/debug/boot/kernel/cpufreq.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/cpufreq.ko.debug
Reading symbols from
/pd2/tmp/destQ/usr/lib/debug/boot/kernel/coretemp.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/coretemp.ko.debug
Reading symbols from
/pd2/tmp/destQ/usr/lib/debug/boot/kernel/uplcom.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/uplcom.ko.debug
Reading symbols from
/pd2/tmp/destQ/usr/lib/debug/boot/kernel/ucom.ko.debug...done.
Loaded symbols for /pd2/tmp/destQ/usr/lib/debug/boot/kernel/ucom.ko.debug
#0 doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
315 dumptid = curthread->td_tid;
(kgdb) list *0xffffffff80df7abe
0xffffffff80df7abe is in aesni_encrypt_cbc
(/smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64).
59 {
60 __m128i tot, ivreg;
61 size_t i;
62
63 len /= AES_BLOCK_LEN;
64 ivreg = _mm_loadu_si128((const __m128i *)iv);
65 for (i = 0; i < len; i++) {
66 tot = aesni_enc(rounds - 1, key_schedule,
67 _mm_loadu_si128((const __m128i *)from) ^ ivreg);
68 ivreg = tot;
Current language: auto; currently minimal
(kgdb) disassemble 0xffffffff80df7abe
Dump of assembler code for function aesni_encrypt_cbc:
0xffffffff80df7a80 <aesni_encrypt_cbc+0>: push %rbp
0xffffffff80df7a81 <aesni_encrypt_cbc+1>: mov %rsp,%rbp
0xffffffff80df7a84 <aesni_encrypt_cbc+4>: sub $0x90,%rsp
0xffffffff80df7a8b <aesni_encrypt_cbc+11>: mov %edi,-0x2c(%rbp)
0xffffffff80df7a8e <aesni_encrypt_cbc+14>: mov %rsi,-0x38(%rbp)
0xffffffff80df7a92 <aesni_encrypt_cbc+18>: mov %rdx,-0x40(%rbp)
0xffffffff80df7a96 <aesni_encrypt_cbc+22>: mov %rcx,-0x48(%rbp)
0xffffffff80df7a9a <aesni_encrypt_cbc+26>: mov %r8,-0x50(%rbp)
0xffffffff80df7a9e <aesni_encrypt_cbc+30>: mov %r9,-0x58(%rbp)
0xffffffff80df7aa2 <aesni_encrypt_cbc+34>: mov -0x40(%rbp),%rcx
0xffffffff80df7aa6 <aesni_encrypt_cbc+38>: shr $0x4,%rcx
0xffffffff80df7aaa <aesni_encrypt_cbc+42>: mov %rcx,-0x40(%rbp)
0xffffffff80df7aae <aesni_encrypt_cbc+46>: mov -0x58(%rbp),%rcx
0xffffffff80df7ab2 <aesni_encrypt_cbc+50>: mov %rcx,-0x28(%rbp)
0xffffffff80df7ab6 <aesni_encrypt_cbc+54>: mov -0x28(%rbp),%rcx
0xffffffff80df7aba <aesni_encrypt_cbc+58>: movdqu (%rcx),%xmm0
0xffffffff80df7abe <aesni_encrypt_cbc+62>: movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7ac3 <aesni_encrypt_cbc+67>: movq $0x0,-0x88(%rbp)
0xffffffff80df7ace <aesni_encrypt_cbc+78>: mov -0x88(%rbp),%rax
0xffffffff80df7ad5 <aesni_encrypt_cbc+85>: cmp -0x40(%rbp),%rax
0xffffffff80df7ad9 <aesni_encrypt_cbc+89>: jae 0xffffffff80df7b65
<aesni_encrypt_cbc+229>
0xffffffff80df7adf <aesni_encrypt_cbc+95>: mov -0x2c(%rbp),%eax
0xffffffff80df7ae2 <aesni_encrypt_cbc+98>: sub $0x1,%eax
0xffffffff80df7ae5 <aesni_encrypt_cbc+101>: mov -0x38(%rbp),%rcx
0xffffffff80df7ae9 <aesni_encrypt_cbc+105>: mov -0x48(%rbp),%rdx
0xffffffff80df7aed <aesni_encrypt_cbc+109>: mov %rdx,-0x8(%rbp)
0xffffffff80df7af1 <aesni_encrypt_cbc+113>: mov -0x8(%rbp),%rdx
0xffffffff80df7af5 <aesni_encrypt_cbc+117>: movdqu (%rdx),%xmm0
0xffffffff80df7af9 <aesni_encrypt_cbc+121>: pxor -0x80(%rbp),%xmm0
0xffffffff80df7afe <aesni_encrypt_cbc+126>: mov %eax,%edi
0xffffffff80df7b00 <aesni_encrypt_cbc+128>: mov %rcx,%rsi
0xffffffff80df7b03 <aesni_encrypt_cbc+131>: callq 0xffffffff80df7b70
<aesni_enc>
0xffffffff80df7b08 <aesni_encrypt_cbc+136>: movdqa %xmm0,-0x70(%rbp)
0xffffffff80df7b0d <aesni_encrypt_cbc+141>: movdqa -0x70(%rbp),%xmm0
0xffffffff80df7b12 <aesni_encrypt_cbc+146>: movdqa %xmm0,-0x80(%rbp)
0xffffffff80df7b17 <aesni_encrypt_cbc+151>: mov -0x50(%rbp),%rcx
0xffffffff80df7b1b <aesni_encrypt_cbc+155>: movdqa -0x70(%rbp),%xmm0
0xffffffff80df7b20 <aesni_encrypt_cbc+160>: mov %rcx,-0x10(%rbp)
0xffffffff80df7b24 <aesni_encrypt_cbc+164>: movdqa %xmm0,-0x20(%rbp)
0xffffffff80df7b29 <aesni_encrypt_cbc+169>: movdqa -0x20(%rbp),%xmm0
0xffffffff80df7b2e <aesni_encrypt_cbc+174>: mov -0x10(%rbp),%rcx
0xffffffff80df7b32 <aesni_encrypt_cbc+178>: movdqu %xmm0,(%rcx)
0xffffffff80df7b36 <aesni_encrypt_cbc+182>: mov -0x48(%rbp),%rcx
0xffffffff80df7b3a <aesni_encrypt_cbc+186>: add $0x10,%rcx
---Type <return> to continue, or q <return> to quit---
0xffffffff80df7b3e <aesni_encrypt_cbc+190>: mov %rcx,-0x48(%rbp)
0xffffffff80df7b42 <aesni_encrypt_cbc+194>: mov -0x50(%rbp),%rcx
0xffffffff80df7b46 <aesni_encrypt_cbc+198>: add $0x10,%rcx
0xffffffff80df7b4a <aesni_encrypt_cbc+202>: mov %rcx,-0x50(%rbp)
0xffffffff80df7b4e <aesni_encrypt_cbc+206>: mov -0x88(%rbp),%rax
0xffffffff80df7b55 <aesni_encrypt_cbc+213>: add $0x1,%rax
0xffffffff80df7b59 <aesni_encrypt_cbc+217>: mov %rax,-0x88(%rbp)
0xffffffff80df7b60 <aesni_encrypt_cbc+224>: jmpq 0xffffffff80df7ace
<aesni_encrypt_cbc+78>
0xffffffff80df7b65 <aesni_encrypt_cbc+229>: add $0x90,%rsp
0xffffffff80df7b6c <aesni_encrypt_cbc+236>: pop %rbp
0xffffffff80df7b6d <aesni_encrypt_cbc+237>: retq
End of assembler dump.
(kgdb) backtrace
#0 doadump (textdump=1) at /smallblocks/src/sys/kern/kern_shutdown.c:315
#1 0xffffffff8080984b in kern_reboot (howto=260)
at /smallblocks/src/sys/kern/kern_shutdown.c:383
#2 0xffffffff8080a08c in vpanic (fmt=0xffffffff80e89d2a "%s",
ap=0xfffffe0688b398e0)
at /smallblocks/src/sys/kern/kern_shutdown.c:776
#3 0xffffffff80809e70 in panic (fmt=0xffffffff80e89d2a "%s")
at /smallblocks/src/sys/kern/kern_shutdown.c:707
#4 0xffffffff80de9967 in trap_fatal (frame=0xfffffe0688b39d30, eva=0)
at /smallblocks/src/sys/amd64/amd64/trap.c:877
#5 0xffffffff80de8ff2 in trap (frame=0xfffffe0688b39d30)
at /smallblocks/src/sys/amd64/amd64/trap.c:610
#6 0xffffffff80de9db5 in trap_check (frame=0xfffffe0688b39d30)
at /smallblocks/src/sys/amd64/amd64/trap.c:659
#7 0xffffffff80dbe8be in calltrap () at
/smallblocks/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff80df7abe in aesni_encrypt_cbc (rounds=10,
key_schedule=0xfffff8001420a800, len=2,
from=0xfffff8001203ede0 "\016\005", to=0xfffff8001203ede0 "\016\005",
iv=0xfffffe0688b39fa8 "▒:▒▒\214:d▒w}4o▒\232\235\214")
at /smallblocks/src/sys/crypto/aesni/aesni_wrap.c:64
#9 0xffffffff80defedb in aesni_cipher_process (ses=0xfffff8001420a800,
enccrd=0xfffff802de6533f0, authcrd=0x0, crp=0xfffff801c28d8bb0)
at /smallblocks/src/sys/crypto/aesni/aesni.c:606
#10 0xffffffff80def420 in aesni_process (dev=0xfffff8000567de00,
crp=0xfffff801c28d8bb0, hint=0)
at /smallblocks/src/sys/crypto/aesni/aesni.c:413
#11 0xffffffff80c7a085 in CRYPTODEV_PROCESS (dev=0xfffff8000567de00,
op=0xfffff801c28d8bb0,
flags=0) at cryptodev_if.h:53
#12 0xffffffff80c78a64 in crypto_invoke (cap=0xfffff80005023078,
crp=0xfffff801c28d8bb0, hint=0)
at /smallblocks/src/sys/opencrypto/crypto.c:1083
#13 0xffffffff80c787ea in crypto_dispatch (crp=0xfffff801c28d8bb0)
at /smallblocks/src/sys/opencrypto/crypto.c:844
#14 0xffffffff80c7cbca in cryptodev_op (cse=0xfffff8028fb70e00,
cop=0xfffffe0688b3a3f0,
active_cred=0xfffff8028fb71000, td=0xfffff80146055620)
at /smallblocks/src/sys/opencrypto/cryptodev.c:849
#15 0xffffffff80c7bc6f in cryptof_ioctl (fp=0xfffff802de7e2690, cmd=3223085927,
data=0xfffffe0688b3a720, active_cred=0xfffff8028fb71000,
td=0xfffff80146055620)
at /smallblocks/src/sys/opencrypto/cryptodev.c:633
#16 0xffffffff80898dfc in fo_ioctl (fp=0xfffff802de7e2690, com=3223085927,
data=0xfffffe0688b3a720, active_cred=0xfffff8028fb71000,
td=0xfffff80146055620)
at file.h:323
#17 0xffffffff80898bf4 in kern_ioctl (td=0xfffff80146055620, fd=5,
com=3223085927,
data=0xfffffe0688b3a720 "") at /smallblocks/src/sys/kern/sys_generic.c:836
#18 0xffffffff808987ba in sys_ioctl (td=0xfffff80146055620,
uap=0xfffffe0688b3a878)
at /smallblocks/src/sys/kern/sys_generic.c:745
#19 0xffffffff8030e351 in freebsd32_ioctl (td=0xfffff80146055620,
uap=0xfffff80146055b58)
at /smallblocks/src/sys/compat/freebsd32/freebsd32_ioctl.c:470
#20 0xffffffff80e3198e in syscallenter (td=0xfffff80146055620) at
subr_syscall.c:132
#21 0xffffffff80e3130f in ia32_syscall (frame=0xfffffe0688b3aab8)
---Type <return> to continue, or q <return> to quit---
at /smallblocks/src/sys/amd64/ia32/ia32_syscall.c:218
#22 0xffffffff80dbf906 in int0x80_syscall_common () at ia32_exception.S:76
#23 0x0000000000000000 in ?? ()
(kgdb) p/x *(struct trapframe *)0xfffffe0688b39d30
$1 = {tf_rdi = 0xa, tf_rsi = 0xfffff8001420a800, tf_rdx = 0x20, tf_rcx =
0xfffffe0688b39fa8,
tf_r8 = 0xfffff8001203ede0, tf_r9 = 0xfffffe0688b39fa8, tf_rax =
0xfffff802de6533f0,
tf_rbx = 0xfffffe0688b3a698, tf_rbp = 0xfffffe0688b39e88, tf_r10 = 0x28,
tf_r11 = 0x0,
tf_r12 = 0x0, tf_r13 = 0x0, tf_r14 = 0x400000000000080, tf_r15 = 0x0,
tf_trapno = 0x9,
tf_fs = 0x13, tf_gs = 0x1b, tf_addr = 0x0, tf_flags = 0x1, tf_es = 0x3b,
tf_ds = 0x3b,
tf_err = 0x0, tf_rip = 0xffffffff80df7abe, tf_cs = 0x20, tf_rflags = 0x10202,
tf_rsp = 0xfffffe0688b39df8, tf_ss = 0x0}

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #11 from ***@heuristicsystems.com.au ---
CFLAGS, COPTFLAGS, LDFLAGS are as follows:
# make -VCFLAGS -Vsp -VCOPTFLAGS -Vsp -VLDFLAGS
-pipe -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -DUSB_HAVE_DISABLE_ENUM -O3
-fno-math-errno -fomit-frame-pointer -Wno-error=unused-command-line-argument
-Wl,--hash-style=sysv -march=haswell

-g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -DUSB_HAVE_DISABLE_ENUM
-fno-math-errno -fomit-frame-pointer -Wno-error=unused-command-line-argument
-Wl,--hash-style=sysv -O2


# ie no LDFLAGS.

Differences in make.conf files between the working kernel/world builds (prior
to May) and now is that we:
- removed -O3 from COPTFLAGS, we explicitly state -O2
- Added TARGET=amd64

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #12 from Konstantin Belousov <***@FreeBSD.org> ---
(In reply to dewayne from comment #11)
Build with *all default* settings and reproduce the issue first.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229222

--- Comment #14 from Konstantin Belousov <***@FreeBSD.org> ---
(In reply to dewayne from comment #13)
Ok.

With this kernel and core, please do the manual unwind:
starting from the tf_rbp = 0xfffffe06892d9138 value,
dereference *tf_rbp, i.e. the long value at the address contained
in the rbp register. It points to the next frame address, and tf_rbp+8
is the address of the caller, it should match the gdb backtrace.
Do that with the next frame as well, up to the end of the trace.
The end result should be the frame address and source line number for
each frame in the backtrace.

What I am looking for is the oldest frame where the frame is only 8-bytes
aligned.