1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
|
<table class="head">
<tr>
<td class="head-ltitle">TESTS(7)</td>
<td class="head-vol">Miscellaneous Information Manual</td>
<td class="head-rtitle">TESTS(7)</td>
</tr>
</table>
<div class="manual-text">
<section class="Sh">
<h1 class="Sh" id="NAME"><a class="permalink" href="#NAME">NAME</a></h1>
<p class="Pp"><code class="Nm">tests</code> —
<span class="Nd">introduction to the <span class="Ux">FreeBSD</span> Test
Suite</span></p>
</section>
<section class="Sh">
<h1 class="Sh" id="DESCRIPTION"><a class="permalink" href="#DESCRIPTION">DESCRIPTION</a></h1>
<p class="Pp">The <span class="Ux">FreeBSD</span> Test Suite provides a
collection of automated tests for two major purposes. On one hand, the test
suite aids
<a class="permalink" href="#developers"><i class="Em" id="developers">developers</i></a>
to detect bugs and regressions when they modify the source tree. On the
other hand, it allows
<a class="permalink" href="#end"><i class="Em" id="end">end users</i></a>
(and, in particular, system administrators) to verify that fresh
installations of the <span class="Ux">FreeBSD</span> operating system behave
correctly on their hardware platform and also to ensure that the system does
not suffer from regressions during regular operation and maintenance.</p>
<p class="Pp">The <span class="Ux">FreeBSD</span> Test Suite can be found in the
<span class="Pa">/usr/tests</span> hierarchy.</p>
<p class="Pp">This manual page describes how to run the test suite and how to
configure some of its optional features. For information on writing the
tests, see <a class="Xr">atf(7)</a>.</p>
<section class="Ss">
<h2 class="Ss" id="Installing_the_test_suite"><a class="permalink" href="#Installing_the_test_suite">Installing
the test suite</a></h2>
<p class="Pp">If the <span class="Pa">/usr/tests</span> directory is missing,
then you will have to enable the build of the test suite, rebuild your
system and install the results. You can do so by setting
‘WITH_TESTS=yes’ in your <span class="Pa">/etc/src.conf</span>
file (see <a class="Xr">src.conf(5)</a> for details) and rebuilding the
system as described in <a class="Xr">build(7)</a>.</p>
</section>
<section class="Ss">
<h2 class="Ss" id="When_to_run_the_tests?"><a class="permalink" href="#When_to_run_the_tests?">When
to run the tests?</a></h2>
<p class="Pp">Before diving into the details of how to run the test suite, here
are some scenarios in which you should run it:</p>
<ul class="Bl-bullet Bd-indent">
<li>After a fresh installation of <span class="Ux">FreeBSD</span> to ensure
that the system works correctly on your hardware platform.</li>
<li>After an upgrade of <span class="Ux">FreeBSD</span> to a different version
to ensure that the new code works well on your hardware platform and that
the upgrade did not introduce regressions in your configuration.</li>
<li>After modifying the source tree to detect any new bugs and/or
regressions.</li>
<li>Periodically, maybe from a <a class="Xr">cron(8)</a> job, to ensure that
any changes to the system (such as the installation of third-party
packages or manual modifications to configuration files) do not introduce
unexpected failures.</li>
</ul>
</section>
<section class="Ss">
<h2 class="Ss" id="Running_the_tests"><a class="permalink" href="#Running_the_tests">Running
the tests</a></h2>
<p class="Pp">By default, Kyua looks for tests in the current directory. To run
the whole test suite, either use the <code class="Fl">-k</code> option to
specify the top-level <span class="Pa">Kyuafile</span>:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ kyua test -k /usr/tests/Kyuafile</pre>
</div>
<p class="Pp">or just change to the test suite root before running Kyua:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua test</pre>
</div>
<p class="Pp">The above will iterate through all test programs in
<span class="Pa">/usr/tests</span> recursively, execute them, store their
results and debugging data in Kyua's database (by default in
<span class="Pa">~/.kyua/store/</span>), and print a summary of the results.
This summary includes a brief count of all total tests run and how many of
them failed.</p>
<p class="Pp">It is possible to restrict which tests to run by providing their
names, or a portion of their path, on the command line. For example, this
would execute all of the tests provided for the <a class="Xr">cp(1)</a> and
<a class="Xr">stat(1)</a> utilities:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua test bin/cp usr.bin/stat</pre>
</div>
<p class="Pp">This would execute only one of the two test programs provided for
<a class="Xr">stat(1)</a>:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua test usr.bin/stat/stat_test</pre>
</div>
<p class="Pp">This would execute just a single test case:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua test usr.bin/stat/stat_test:t_flag</pre>
</div>
<p class="Pp">Finally, this would execute that test case in debug mode:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua debug -p usr.bin/stat/stat_test:t_flag</pre>
</div>
<p class="Pp">The <code class="Fl">-p</code> option tells Kyua to pause before
cleanup so you can inspect the temporary directory to better understand why
the test failed.</p>
<p class="Pp">Note that some tests may require root privileges to execute:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ kyua debug usr.bin/stat/stat_test:h_flag
usr.bin/stat/stat_test:h_flag -> skipped: Requires root privileges
$ sudo kyua debug usr.bin/stat/stat_test:h_flag
[...]
usr.bin/stat/stat_test:h_flag -> passed</pre>
</div>
<p class="Pp">Conversely, some tests will only work correctly if run as an
unprivileged user. This will normally be noted in the test's metadata,
causing Kyua to automatically drop privileges before running the test.</p>
</section>
<section class="Ss">
<h2 class="Ss" id="Obtaining_reports_of_the_tests_execution"><a class="permalink" href="#Obtaining_reports_of_the_tests_execution">Obtaining
reports of the tests execution</a></h2>
<p class="Pp">Additional information about the test results can be retrieved by
using Kyua's various reporting commands. For example, the following would
print a plain-text report of the tests executed in the latest test run and
show which ones failed:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ kyua report --verbose</pre>
</div>
<p class="Pp">To show the results of an arbitrary test run, use the
<code class="Fl">-r</code> option to specify which results file to read:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ kyua report --verbose \
-r ~/.kyua/store/results.usr_tests.20260417-173009-335060.db</pre>
</div>
<p class="Pp">Keep in mind that if the tests were run as root, the results will
have been stored in root's <span class="Pa">kyua</span> directory, and the
easiest way to access them will be to run the <code class="Cm">report</code>
command as root as well:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ cd /usr/tests
$ sudo kyua test usr.bin/stat
$ sudo kyua report --verbose</pre>
</div>
<p class="Pp">This example would generate an HTML report ready to be published
on a web server:</p>
<div class="Bd Pp Bd-indent Li">
<pre>$ kyua report-html --output ~/public_html/tests</pre>
</div>
<p class="Pp">For further details on the command-line interface of Kyua, please
refer to its manual page <a class="Xr">kyua(1)</a>.</p>
</section>
<section class="Ss">
<h2 class="Ss" id="Configuring_the_tests"><a class="permalink" href="#Configuring_the_tests">Configuring
the tests</a></h2>
<p class="Pp">Some test cases in the <span class="Ux">FreeBSD</span> Test Suite
require manual configuration by the administrator before they can be run.
Unless certain properties are defined, the tests that require them will be
skipped.</p>
<p class="Pp">Test suites are configured by defining their configuration
variables in <span class="Pa">/etc/kyua/kyua.conf</span>. The format of this
file is detailed in <a class="Xr">kyua.conf(5)</a>.</p>
<p class="Pp">The following configuration variables are available in the
<span class="Ux">FreeBSD</span> Test Suite:</p>
<dl class="Bl-tag">
<dt id="allow_devfs_side_effects"><var class="Va">allow_devfs_side_effects</var></dt>
<dd>If defined, enables tests that may destroy and recreate semipermanent
device nodes, like disk devices. Without this variable, tests may still
create and destroy devices nodes that are normally transient, like
/dev/tap* and /dev/pts*, as long as they clean them up afterwards.
However, tests that require this variable have a relaxed cleanup
requirement; they must recreate any devices that they destroyed, but not
necessarily with the same devnames.</dd>
<dt id="allow_sysctl_side_effects"><var class="Va">allow_sysctl_side_effects</var></dt>
<dd>Enables tests that change globally significant <a class="Xr">sysctl(8)</a>
variables. The tests will undo any changes in their cleanup phases.</dd>
<dt id="allow_network_access"><var class="Va">allow_network_access</var></dt>
<dd>Enables tests that need to access the network the test host is connected
to. Such tests may require properly configured Internet access.</dd>
<dt id="disks"><var class="Va">disks</var></dt>
<dd>Must be set to a space delimited list of disk device nodes. Tests that
need destructive access to disks must use these devices. Tests are not
required to preserve any data present on these disks.</dd>
<dt id="fibs"><var class="Va">fibs</var></dt>
<dd>Must be set to a space delimited list of FIBs (routing tables). Tests that
need to modify a routing table may use any of these. Tests will cleanup
any new routes that they create.</dd>
</dl>
</section>
<section class="Ss">
<h2 class="Ss" id="What_to_do_if_something_fails?"><a class="permalink" href="#What_to_do_if_something_fails?">What
to do if something fails?</a></h2>
<p class="Pp">If there is
<a class="permalink" href="#any"><i class="Em" id="any">any failure</i></a>
during the execution of the test suite, please consider reporting it to the
<span class="Ux">FreeBSD</span> developers so that the failure can be
analyzed and fixed. To do so, either send a message to the appropriate
mailing list or file a problem report. For more details please refer to:</p>
<ul class="Bl-bullet Bd-indent Bl-compact">
<li><a class="Lk" href="https://lists.freebsd.org/">FreeBSD Mailing
Lists</a></li>
<li><a class="Lk" href="https://www.freebsd.org/support/">Problem
Reporting</a></li>
</ul>
</section>
</section>
<section class="Sh">
<h1 class="Sh" id="FILES"><a class="permalink" href="#FILES">FILES</a></h1>
<dl class="Bl-tag Bl-compact">
<dt><span class="Pa">/etc/kyua/kyua.conf</span></dt>
<dd>System-wide configuration file for <a class="Xr">kyua(1)</a>.</dd>
<dt><span class="Pa">~/.kyua/kyua.conf</span></dt>
<dd>User-specific configuration file for <a class="Xr">kyua(1)</a>; overrides
the system file.</dd>
<dt><span class="Pa">~/.kyua/logs/</span></dt>
<dd>Default location of Kyua debug logs (one <span class="Pa">.log</span> file
per test run).</dd>
<dt><span class="Pa">~/.kyua/store/</span></dt>
<dd>Default location of Kyua test results (one <span class="Pa">.db</span>
file per test run).</dd>
<dt><span class="Pa">/usr/tests/</span></dt>
<dd>Location of the <span class="Ux">FreeBSD</span> Test Suite.</dd>
<dt><span class="Pa">/usr/tests/Kyuafile</span></dt>
<dd>Top-level test suite definition file.</dd>
</dl>
</section>
<section class="Sh">
<h1 class="Sh" id="SEE_ALSO"><a class="permalink" href="#SEE_ALSO">SEE
ALSO</a></h1>
<p class="Pp"><a class="Xr">kyua(1)</a>, <a class="Xr">atf(7)</a>,
<a class="Xr">build(7)</a>, <a class="Xr">development(7)</a></p>
</section>
<section class="Sh">
<h1 class="Sh" id="HISTORY"><a class="permalink" href="#HISTORY">HISTORY</a></h1>
<p class="Pp">The <span class="Ux">FreeBSD</span> Test Suite first appeared in
<span class="Ux">FreeBSD 10.1</span> and was installed by default in
<span class="Ux">FreeBSD 11.0</span>.</p>
<p class="Pp">The <code class="Nm">tests</code> manual page first appeared in
<span class="Ux">NetBSD 6.0</span> and was later ported to
<span class="Ux">FreeBSD 10.1</span>.</p>
<p class="Pp">The test driver, <a class="Xr">kyua(1)</a>, was imported as part
of the base system in <span class="Ux">FreeBSD 13.0</span>, previously being
available only in <a class="Xr">ports(7)</a>.</p>
</section>
<section class="Sh">
<h1 class="Sh" id="AUTHORS"><a class="permalink" href="#AUTHORS">AUTHORS</a></h1>
<p class="Pp"><span class="An">Julio Merino</span>
<<a class="Mt" href="mailto:jmmv@FreeBSD.org">jmmv@FreeBSD.org</a>></p>
</section>
</div>
<table class="foot">
<tr>
<td class="foot-date">April 17, 2026</td>
<td class="foot-os">FreeBSD 15.0</td>
</tr>
</table>
|
