42c7760b mw Sept. 1, 2021, 11:06 p.m.
Some of the changes in this release:
* Hardware RSS hash key reconfiguration and indirection table
reconfiguration support.
* Full kernel RSS support.
* Extra statistic counters.
* Netmap support for ENAv3.
* Locking assertions.
* Extra log messages.
* Reset handling fixes.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
6d1ef2ab mw Sept. 1, 2021, 11:06 p.m.
Bind RX/TX queues and MSI-X vectors to matching CPUs based on the RSS
bucket entries.

Introduce sysctls for the following RSS functionality:
- rss.indir_table:      indirection table mapping
- rss.indir_table_size: indirection table size
- rss.key:              RSS hash key (if Toeplitz used)

Said sysctls are only available when compiled without `option RSS`, as
kernel-side RSS support currently doesn't offer RSS reconfiguration.

Migrate the hash algorithm from CRC32 to Toeplitz and change the initial
hash value to 0x0 in order to match the standard Toeplitz implementation.
Provide helpers for hash key inversion required for HW operations.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
223c8cb1 mw Sept. 1, 2021, 11:06 p.m.
Provide the following sysctl statistics in order to stay aligned with
the Linux driver:
* rx_ring.csum_good
* tx_ring.unmask_interrupt_num

Also rename the 'bad_csum' statistic name to 'csum_bad' for alignment.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
07aff471 mw Sept. 1, 2021, 11:06 p.m.
In order to use `ena_global_lock` in sysctl context, it must be kept
outside the driver instance's software context, as sysctls can be called
before attach and after detach, leading to lock use before sx_init and
after sx_destroy otherwise.
Solve this issue by turning `ena_global_lock` into a file scope
variable, shared between all instances of the driver and associated
sysctl context, and in turn initialized/destroyed in dedicated
SYSINIT/SYSUNINIT functions.
As a side effect, this change also fixes existing race in the reset
routine, when simultaneously accessing sysctl exposed properties.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
a8314668 mw Sept. 1, 2021, 11:06 p.m.
If LLQ is being used, `ena_tx_ctx.meta_valid` must stay enabled. This
fixes netmap support on latest generation ENA HW and aligns it with the
core driver behavior.

As netmap doesn't support any csum offloads, the
`adapter->disable_meta_caching` value can be simply passed to the HW.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
986e7b92 mw Sept. 1, 2021, 11:06 p.m.
Delegate RSS related functionality into separate .c/.h files in
preparation for the full RSS support.

While at it, reorder functions and remove prototypes for ones with
internal linkage.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
cb98c439 mw Sept. 1, 2021, 11:06 p.m.
ENA silently assumed that ena_up, ena_down and ena_start_xmit routines
should be called within locked context. Driver's logic heavily assumes
on concurrent access to those routines, so for safety and better
documentation about this assumption, the locking assertions were added
to the above functions.

The assertion was added only for the main steps (skipping the helper
functions) which can be called from multiple places including the kernel
and the driver itself.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
77160654 mw Sept. 1, 2021, 11:06 p.m.
Stay aligned with the Linux driver by adding the following logs:
* inform the user about retrying queue creation
* warn on non-empty ena_tx_buffer.mbuf prior to ena_tx_map_mbuf

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
433ab9b6 mw Sept. 1, 2021, 11:06 p.m.
Check for ENA_FLAG_TRIGGER_RESET inside a locked context in order to
avoid potential race conditions with ena_destroy_device. This aligns the
reset task logic with the Linux driver.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
36130d29 mw Sept. 1, 2021, 11:05 p.m.
All ena_com_prepare_tx errors other than ENA_COM_NO_MEM are fatal and
require device reset.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
c81f8c26 mw Sept. 1, 2021, 11:05 p.m.
In case of Low-latency Queue, one small enough descriptor can be pushed
directly to the ENA hw, thus saving one fragment. Check for this
condition before performing collapse.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
cgit
6bc90e8a brooks Sept. 1, 2021, 8:58 p.m.
Reviewed by:	kevans, emaste
MFC after:	1 week
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D31351
cgit
df501bac brooks Sept. 1, 2021, 8:58 p.m.
Switch the main syscall table to use CAPENABLED flags rather than
capabilities.conf.  This avoid synchronization issues between
syscalls.master and capabilities.conf (e.g. when renaming a syscall
during development).

For now, move capabilities.conf to sys/compat/freebsd32 and use it
there.  Use of sys/compat/freebsd32/syscalls.master should be replaced
by makesyscalls.lua enhancements to allow the main one to be used.

This change results in no changes to generated files after running
`make sysent`.

Reviewed by:	kevans, emaste
MFC after:	1 week
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D31350
cgit
6945df3f brooks Sept. 1, 2021, 8:58 p.m.
The CAPENABLED flag indicates that the syscall can be used in capsicum
capability mode.  It is intended to replace capabilities.conf.

Reviewed by:	kevans, emaste
MFC after:	1 week
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D31349
cgit
5647f85a kbowling Sept. 1, 2021, 8:13 p.m.
This reverts commit b684d812fcb04b2997fd755405a92c36b9f6e30e.

It causes an issue on a pfsense routing workload where memory
fragmentation prevents the necessary consecutive pages from being
readily available.

Reported by:	pfsense (mjg, scottl)
Approved by:	ian
MFC after:	1 day
Differential Revision:	https://reviews.freebsd.org/D31244
cgit