r328085 bapt Jan. 17, 2018, 1:25 p.m.
MFC after:	3 days
r328084 dim Jan. 17, 2018, 1:19 p.m.

Reported by:	ohartmann@walstatt.org,david@catwhisker.org
Pointy hat to:	mckusick
r328083 kib Jan. 17, 2018, 11:44 a.m.
The implementation of the Kernel Page Table Isolation (KPTI) for
amd64, first version. It provides a workaround for the 'meltdown'
vulnerability.  PTI is turned off by default for now, enable with the
loader tunable vm.pmap.pti=1.

The pmap page table is split into kernel-mode table and user-mode
table. Kernel-mode table is identical to the non-PTI table, while
usermode table is obtained from kernel table by leaving userspace
mappings intact, but only leaving the following parts of the kernel

    kernel text (but not modules text)
    GDT/IDT/user LDT/task structures
    IST stacks for NMI and doublefault handlers.

Kernel switches to user page table before returning to usermode, and
restores full kernel page table on the entry. Initial kernel-mode
stack for PTI trampoline is allocated in PCPU, it is only 16
qwords.  Kernel entry trampoline switches page tables. then the
hardware trap frame is copied to the normal kstack, and execution

IST stacks are kept mapped and no trampoline is needed for
NMI/doublefault, but of course page table switch is performed.

On return to usermode, the trampoline is used again, iret frame is
copied to the trampoline stack, page tables are switched and iretq is
executed.  The case of iretq faulting due to the invalid usermode
context is tricky, since the frame for fault is appended to the
trampoline frame.  Besides copying the fault frame and original
(corrupted) frame to kstack, the fault frame must be patched to make
it look as if the fault occured on the kstack, see the comment in
doret_iret detection code in trap().

Currently kernel pages which are mapped during trampoline operation
are identical for all pmaps.  They are registered using
pmap_pti_add_kva().  Besides initial registrations done during boot,
LDT and non-common TSS segments are registered if user requested their
use.  In principle, they can be installed into kernel page table per
pmap with some work.  Similarly, PCPU can be hidden from userspace
mapping using trampoline PCPU page, but again I do not see much
benefits besides complexity.

PDPE pages for the kernel half of the user page tables are
pre-allocated during boot because we need to know pml4 entries which
are copied to the top-level paging structure page, in advance on a new
pmap creation.  I enforce this to avoid iterating over the all
existing pmaps if a new PDPE page is needed for PTI kernel mappings.
The iteration is a known problematic operation on i386.

The need to flush hidden kernel translations on the switch to user
mode make global tables (PG_G) meaningless and even harming, so PG_G
use is disabled for PTI case.  Our existing use of PCID is
incompatible with PTI and is automatically disabled if PTI is
enabled.  PCID can be forced on only for developer's benefit.

MCE is known to be broken, it requires IST stack to operate completely
correctly even for non-PTI case, and absolutely needs dedicated IST
stack because MCE delivery while trampoline did not switched from PTI
stack is fatal.  The fix is pending.

Reviewed by:	markj (partially)
Tested by:	pho (previous version)
Discussed with:	jeff, jhb
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
r328082 kib Jan. 17, 2018, 11:21 a.m.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
r328081 tuexen Jan. 17, 2018, 10:30 a.m.
r328080 wma Jan. 17, 2018, 9:45 a.m.
Add missing little-endian 64-bit read and write. Since there
is no direct ASM opcode for this, perform byte swap if

Created by:            Wojciech Macek <wma@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          QCM Technologies
r328079 wma Jan. 17, 2018, 9:36 a.m.
Use current userspace address for segment mapping. Previously,
there was a bug which made the funciton constantly using the userspace
base address which could cause data integrity issues.

Created by:            Wojciech Macek <wma@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          QCM Technologies
r328078 wma Jan. 17, 2018, 9:33 a.m.
Add CXGBE driver which is required for PowerNV system.
Also, remove AHCI which does not work in BigEndian.

Created by:            Wojciech Macek <wma@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          QCM Technologies
r328077 wma Jan. 17, 2018, 8:01 a.m.
FreeBSD prints text char-by-char, which is not what OPAL
is designed to. Poll events more frequently to avoid buffer
overflow and loosing data.

Created by:            Wojciech Macek <wma@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          QCM Technologies
r328076 wma Jan. 17, 2018, 7:39 a.m.
- map all devices to PE0
- use 1:1 TCE mapping
- provide the same TCE mapping for all PEs (not only PE0)
- add TCE reset and alignment (required by OPAL)

Created by:            Wojciech Macek <wma@semihalf.com>
Obtained from:         Semihalf
Sponsored by:          QCM Technologies
r328075 mckusick Jan. 17, 2018, 6:31 a.m.
No functional change.
r328071 landonf Jan. 17, 2018, 3:34 a.m.
On a SPROM-less device, the PCI(e) bridge core will be initialized with its
power-on-reset defaults; this can leave the SPROM-derived BHND_PCI_SRSH_PI
value pointing to the wrong backplane address. This value is used by the
PCI core when performing address translation between the static register
windows in BAR0 that map the PCI core's register block, and backplane
address space.

Previously, bhndb_pci(4) incorrectly used the potentially invalid static
BAR0 PCI register windows when attempting to correct the BHND_PCI_SRSH_PI
value in the PCI core's SPROM shadow.

Instead, we now read/update BHND_PCI_SRSH_PI by fetching the PCI core's
backplane address from the core enumeration table, and then using a dynamic
register window to explicitly map the PCI core's register block into BAR0.

Sponsored by:	The FreeBSD Foundation
r328070 pfg Jan. 16, 2018, 11:19 p.m.
r328069 ian Jan. 16, 2018, 11:18 p.m.
by these calls is now protected by a spin mutex (obscured within the

Reported by:	bde@
r328068 ian Jan. 16, 2018, 11:14 p.m.
The static atrtc_set() function was called only from clock_settime(), so
just move its contents entirely into clock_settime() and delete atrtc_set().

Rename the struct bcd_clocktime variables from 'ct' to 'bct'.  I had
originally wanted to emphasize how identical the clocktime and bcd_clocktime
structs were, but things evolved to the point where the structs are not at
all identical anymore, so now emphasizing the difference seems better.