Stability Testing
What a Stress Test Actually Proves
A passing run is a sentence with five qualifiers in it. Most of the trouble in stability testing comes from dropping four of them.
A machine that survives a night of load has proved one narrow sentence: this workload, on this hardware, in this configuration, for that night, produced no error the tool could detect. Everything broader read into a clean run is reading, not proof. This page holds the run to its own wording, then follows the failure, because that is where the actual information lives.
The run is the unit of proof
A completed run is a statement about the run. Those patterns, on those components, at that speed, at that temperature, in that slot order, produced no detected error during that run. Change the room temperature and the sentence stops covering the new night; change one setting and it stops covering the old one. Duration stretches the sentence without changing its shape, which is why how long a burn-in should run is its own question.
"No detected" carries the other half. A tool detects what it observes, through the paths its own computation takes; the pass is a fact about the union of those paths. Whatever the instruction mix barely visited stood outside the experiment and gets no verdict from it.
What does the load actually touch?
A stress program is a workload borrowed from another purpose. The GIMPS prime search publishes its client, Prime95, free of charge for Windows, Linux, FreeBSD and Mac OSX, with version 30.19 build 20 on the project's download page, and it states its own purpose: contributors give spare processing power to the search for Mersenne primes. Practice borrows the arithmetic as a heat source. What the instruction mix presses hard, it tests; what it rarely visits, it merely warms. The page on what a memory pass covers makes the same point for its own subsystem. "The computer was stressed" is almost never true as stated. A subsystem was.
Where does a crash leave its message?
A pass proves a bounded nothing. A crash, read properly, proves a located something. On Linux a kernel fault under load typically leaves a stack dump, and the kernel documentation assigns that dump a precise value: enough information to identify the line in the kernel source where the bug happened. The same page stretches the word Oops to cover every stack trace that needs analysis, which is convenient, since a WARNING and a null pointer dereference turn out to be cousins here. The dump also carries state around the fault: the modules linked in, taint flags in parentheses, a plus for a module arriving, a minus for one leaving.
The message lands in knowable places, and that is what turns a hang into evidence. Normally klogd lifts the text out of the kernel buffers and hands it to syslogd, which writes it to a log file, typically /var/log/messages; on a systemd machine journald holds it and journalctl reads it back. If klogd dies first, the buffers still hold the text, and dmesg redirected to a file recovers it, as does /proc/kmsg, though the documentation calls it a never ending file that has to be broken into to stop. For the anatomy of a real trace, the kernel's own bug hunting page dissects an ordinary module unload. The message is still not a diagnosis; reading an instability is what converts it into one.
Three exits when the machine will not talk
The documentation also allows for the crash so total that no command can be entered and the disk is not available. It names three exits: copy or photograph what is still on the screen, capture a serial console on a second machine, or use Kdump to pull the kernel ring buffer out of old memory. The exits are not interchangeable.
| Route | What it preserves | What it demands |
|---|---|---|
| Screen, copied or photographed | Whatever had not scrolled off the console | No preparation; a higher console resolution keeps more lines readable, though the documentation notes it needs the vesafb driver and cannot recover the earliest messages |
| Serial console to a second machine | Every line as it is produced, on the machine that did not crash | The link and the second machine, in place before the run |
| Kdump | The kernel ring buffer, extracted from old memory after the crash | Advance setup; the documentation pairs the extraction with a gdb macro for reading the buffer |
Prepare the capture before the load
- Know where a failure message will land on this machine: journald and journalctl under systemd, otherwise the log file that klogd and syslogd feed.
- Pick an exit for the crash that cannot be typed into: a camera within reach, a serial link to a second machine, or Kdump configured beforehand.
- Write down the identity of the run before launching it: tool, version, planned duration, everything changed since the last pass.
- After a hang that recovers, read the buffers before restarting; dmesg redirected to a file keeps what the kernel still held.
What no completed run certifies
Nothing about tomorrow. A pass binds to the configuration and the hour that produced it, and a machine is a moving target: a kernel update swaps out the code that was under load, a driver reload shows in the next trace as a module leaving and arriving, the room warms by afternoon. Even failure evidence has grades, since the documentation notes that a kernel built with debug information and run through the decode_stacktrace script yields a better stack trace than the same crash otherwise would. Better evidence about failures is always available. Certification of their absence never is.
Where this goes wrong
The reboot reflex. The machine hangs, the power button follows, and the buffers are refilled on the way up. The crash left nothing any tool can read, and the run gets repeated to produce evidence it had already produced once.
The certificate reflex is the mirror error. One clean night becomes the word stable, the word sheds its scope, and two variables get changed before the next run on the strength of a pass that covered neither. A photographed console adds a third: the lines that mattered had scrolled off, and the picture keeps the calm part of the crash.
Keep the specimen with the machine
When a run fails, label the message at the moment of capture: the tool and its version, how long the run had lasted, what the configuration was, what changed since the last clean run. Save the message itself before anything restarts. The label is what makes the next experiment narrower than the last one. A labeled failure moves the diagnosis; an unlabeled one is a story about a bad night.
docs.kernel.org is the Linux kernel's own documentation site. Its administration guide carries the bug hunting page under tracking down and identifying problems: what a stack dump contains, Oops as shorthand for traces needing analysis, where the message is stored, and the capture routes from a crashed machine. Neighboring pages cover reporting regressions, bisecting, tainted kernels, Kdump and the ramoops oops and panic logger.