.\" Copyright (c) 2013 Peter Grehan
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd December 30, 2024
.Dt VMM 4
.Os
.Sh NAME
.Nm vmm.ko
.Nd "bhyve virtual machine monitor"
.Sh SYNOPSIS
To load the driver as a module at boot, add this line to
.Xr loader.conf 5 :
.Bd -literal -offset indent
.Cd vmm_load="YES"
.Ed
.Pp
The module can also be loaded manually with
.Xr kldload 8 :
.Bd -literal -offset indent
kldload vmm
.Ed
.Sh DESCRIPTION
.Nm
provides the kernel portion of the
.Xr bhyve 4
hypervisor.
The following platforms are supported:
.Bl -bullet -compat
.It
amd64: An Intel CPU with VT-x/EPT or AMD CPU with SVM support is required.
.It
arm64: The boot CPU must start in EL2 and the system must have a GICv3 interrupt
controller.
VHE support will be used if available.
.It
riscv: The CPUs must implement the H (hypervisor) RISC-V ISA extension.
.El
.Pp
PCI device passthrough to a virtual machine requires
hardware with VT-d support and is available only on amd64.
.Sh ACCESS CONTROL
Only the super-user and processes with write access to the
.Pa /dev/vmmctl
device file may create and destroy virtual machines.
By default, members of the
.Va vmm
group have such access.
Once created, a virtual machine may be destroyed only by that user or
the super-user.
.Pp
Unprivileged users must use
.Dq monitor mode
to run the virtual machine; in this mode, the virtual machine is automatically
destroyed when its device file is closed.
When running
.Xr bhyve 8 ,
this mode can be selected by specifying the
.Fl M
flag.
.Pp
Virtual machines can be created in a jail if the jail has the
.Va allow.vmm
flag set.
.Sh PCI PASSTHROUGH
On amd64 where the hardware supports VT-d,
PCI devices can be reserved for use by the hypervisor.
Entries consisting of the PCI
.Ar bus Ns / Ns Ar slot Ns / Ns Ar function
are added to the
.Va pptdevs
.Xr loader.conf 5
variable.
Additional entries are separated by spaces.
Host PCI devices that match an entry will be assigned to the hypervisor
and will not be probed by
.Fx
device drivers.
See the
.Sx EXAMPLES
section below for sample usage.
.Pp
Note that
.Nm vmm
must be given first the right of refusal to all
.Xr pci 4
devices it may need to claim.
As a result, the
.Nm vmm
kernel module almost certainly needs to be loaded from
.Xr loader.conf 5
rather than by adding it to
.Va kld_list in
.Xr rc.conf 5 .
.Pp
A large number of PCI device entries may require a string longer than the
128-character limit of
.Xr loader.conf 5
variables.
The
.Va pptdevs2
and
.Va pptdevs3
variables can be used for additional entries.
.Pp
In general, PCI passthrough cannot be used when running
.Xr bhyve 8
as an unprivileged user or in a jail, as this feature requires write
access to
.Pa /dev/pci .
.Sh LOADER TUNABLES
Tunables can be set at the
.Xr loader 8
prompt before booting the kernel or stored in
.Xr loader.conf 5 .
.Bl -tag -width indent
.It Va hw.vmm.maxcpu
Maximum number of virtual CPUs.
The default is the number of physical CPUs in the system.
.El
.Sh FILES
.Bl -tag -width /dev/vmm.io/* -compact
.It Pa /dev/vmmctl
control interface for creating and destroying virtual machines.
.It Pa /dev/vmm/*
device interface for individual virtual machines.
.It Pa /dev/vmm.io/*
device interface for device memory mapped into virtual machines.
.Sh EXAMPLES
Reserve three PCI devices for use by the hypervisor: bus 10 slot 0 function 0,
bus 6 slot 5 function 0, and bus 6 slot 5 function 1.
.Bd -literal -offset indent
pptdevs="10/0/0 6/5/0 6/5/1"
.Ed
.Pp
It is possible to detach
.Va ppt
from a PCI device without rebooting the host machine and then attach a host
driver, using the
.Xr devctl 8
utility.
Suppose
.Va ppt
is currently attached to
.Va pci0:0:1:0
and we want the host's
.Xr xhci 4
driver to be attached instead:
.Bd -literal -offset indent
# devctl set driver -f pci0:0:1:0 xhci
.Ed
.Pp
The same can be applied to attach
.Va ppt
back:
.Bd -literal -offset indent
# devctl set driver -f pci0:0:1:0 ppt
.Ed
.Sh SEE ALSO
.Xr bhyve 4 ,
.Xr loader.conf 5 ,
.Xr bhyve 8 ,
.Xr bhyvectl 8 ,
.Xr bhyveload 8 ,
.Xr devctl 8 ,
.Xr jail 8 ,
.Xr kldload 8
.Sh HISTORY
.Nm vmm.ko
first appeared in
.Fx 10.0 .
arm64 and riscv support first appeared in
.Fx 15.0 .
.Sh AUTHORS
.An Neel Natu Aq neel@freebsd.org
.An Peter Grehan Aq grehan@freebsd.org