InformIT: Red Hat Linux 7 Unleashed > Recovering from Faulty Kernels

Recovering from Faulty Kernels

It happens. You execute an orderly shutdown and reboot, the monitor flashes (or your connection goes dead), and you wait for the boot, only to be greeted with a partial LILO prompt…or worse.

Disaster Recovery in Kernel Compiles

The worst that can happen with a modified kernel is boot failure. It happens to the best of us, and usually happens because of incompatible settings or forgetting to run lilo after changing a kernel. There are few things as frightening as the dread LI- boot prompt.

Before you reboot a new kernel, you should take a few precautions to ensure a backup boot method. If your hardware has a floppy drive, keep your install/rescue disks handy. Production machines should always keep a boot floppy with their current stable kernel and your /etc/lilo.conf should define at least one backup kernel image.

There is a South African proverb, "If you hear hoof-beats in the street, it is probably not a zebra." With a Linux system, your darkest hour still probably does not require reinstalling the entire distribution. Even in seemingly hopeless situations, it is far more likely that your system can be safely rescued and repaired.

When the worst happens, use your rescue disks to boot your system, manually fsck and mount the hard drive, restore order, rerun lilo, and breathe easier. You can also find a number of rescue disks and tools in the Metalab archive (ftp://metalab.unc.edu/pub/linux).

NOTE

The meaning of the preceding proverb is "When something odd happens, it is usually the most common thing, not the most exceptional" and in this context, having to reinstall Red Hat is a very exceptional situation (in 6 years, I have had to do this only once, when the Windows partition scrambled my Linux partition). In the Linux support and IRC channels, those just starting to experment with Linux most often ask (or state) "Should I reinstall" and although the answer can be yes, it is overwhelmingly more often no.

Typically, a faulty kernel will exhibit one of the following behaviors:

The machine cycles through repeated rebooting.
You see some substring of the LILO prompt, such as LIL- followed by a halt.
Linux begins to load but halts at some point during the kernel messages.
Linux loads but ends in a kernel panic message.
Linux loads, runs, lets you log in, and then dies when it is least convenient.

If you are prepared, your prognosis for a full recovery is very good. If you can get up to the LILO: prompt, the most convenient recovery is to load your backup kernel by specifying its label to the boot loader:

LILO: backup

This will boot from your previous kernel and allow you in so you can fix the problem and try your luck again. If you cannot get to the LILO: prompt, your only alternative is to use your boot disk or to use a rescue disk. The boot disk makes life much easier because the running system will be identical to your normal system. If you use a rescue disk, you must manually mount your system partitions and enable any extra modules.

Where alternate kernels and boot disks are not practical, for instance on thin clients with limited diskspace, and if you can reach the LILO: prompt, you can try to start your system in single-user mode to prevent the probing and loading of many modules, such as your network card (a frequent culprit). The default configuration for single-user (a.k.a. runlevel 1) mode is specified by the files in /etc/rc.d/rc1.d, and it is a good idea to double-check the symlinks in that directory after each system upgrade to ensure that the choices are intelligent for the purpose. Single-user mode will put you directly into a system shell; once the problem has been corrected, you can either reboot the system or exit the shell to return to multiuser mode.

Repeated Rebooting

Nine times out of ten, repeated rebooting is caused by changing the kernel file and forgetting to run lilo to register the new image with the boot loader. lilo needs the raw sector location of the kernel; copying a kernel image will move it to a new disk sector and leave the previous pointer stored by lilo dangling over an abyss.

This problem can be corrected by booting from the boot floppy and running lilo, or by using a rescue disk, mounting the boot partition under /mnt, and running lilo with the options to use a relative path:

lilo -r /mnt

Partial LILO Prompt

A partial LILO prompt is the most terrifying of all kernel boot errors. Each letter of the L-I-L-O signifies a stage in the boot process:

L- or LIL: Usually a media error or failure to include the boot partition or filesystem support (or including it as a module).
LI or LIL?: /boot/boot.b is missing, moved, or corrupt. The solution is the same for all: re-run lilo.

More information on using lilo and the diagnosis of lilo error codes can be found in /usr/doc/lilo-0.x/TechnicalGuide.ps .

Kernel Halts While Loading

Device probing is risky business and a frequent cause of kernel halts while loading. For example, if you are configuring for a gateway/firewall machine with two network interfaces, the second probe may cause the kernel to halt. Other causes of kernel halts are IRQ conflicts, memory conflicts, and mismatched devices selecting similar but not-quite-identical drivers.

You can avoid probing, memory, and IRQ conflicts for most kernel modules and devices by supplying configuration parameters in the /etc/lilo.conf append line or at the LILO: prompt. The exact parameters to use depend on your device, but you can find advice in the README files, either in linux/Documentation or in the subdirectories of the driver source code.

Resolving IRQs and Memory Ports

If you have IRQ and/or memory conflicts, and if you have a Windows partition, you can find the values used by Windows in the Control Panel, System, Devices listings and then use these settings on the lilo command line, in /etc/ modules.conf, or in your BIOS PnP settings. It is unfortunate, but a few manufacturers still believe their best business model is to restrict use of their hardware to Microsoft users. When techniques and interfaces for probing and configuring these devices are not available to Linux programmers, we are forced to reverse engineer the protocols, which can sometimes result in a lack of Linux support until after the device is obsolete. Fortunately, most manufacturers have seen the light and happily provide the information needed to incorporate their products under Linux.

Kernel Panic

A kernel panic message has a certain cryptic poetry to it, like a robotic haiku, a snapshot testament to the last moments of a Linux kernel. A kernel panic usually has the following form:

unable to handle kernel paging request at address C0000010
Oops: 0002
EIP: 0010:XXXXXXXX
eax: xxxxxxxx ebx: xxxxxxxx ecx: xxxxxxxx edx: xxxxxxxx
esi: xxxxxxxx edi: xxxxxxxx ebp: xxxxxxxx
ds: xxxx es: xxxx fs: xxxx gs: xxxx
Pid: xx, process nr: xx

xx xx xx xx xx xx xx xx xx xx

For most practical purposes, knowing where the panic occurs is more useful than interpreting the message itself. The leading text tells what triggered the event, and this is followed by the addresses held in various registers. Intrepid readers can find detailed instructions on decoding this message in the linux/Documentation/oops-tracing.txt file.

In production kernels, panic messages are rare and are usually due to a misconfiguration problem, missing modules, failure to load a module before using some essential feature, or using hardware not supported by the current kernel. With development kernels, kernel panics can become a way of life.

Kernel Oops and Bug Reporting

Linux is highly stable and resilient to application failures, but when you start experimenting with odd kernel combinations or experimental editions, hardware, and configurations, stuff happens. In the parlance of the kernel developers, an "oops" is a kernel panic message that occurs spontaneously, often mercilessly, and for no apparent reason. The message reported is similar to the kernel panic that can occur during the boot, but it may not be visible if you are running X Windows. The cause of both the boot halting and a spontaneous oops are is the same: the kernel has reached an impasse.

When an oops occurs during a user session, the kernel panic message may be displayed on one of the Linux Alt consoles (seen by pressing Ctrl+Alt and a function key) or by checking the system log file in /var/log/messages (another good reason to echo all system messages to /dev/tty12). If you can see the panic report, the activity just prior to this in the log may give some clues to the cause of the panic.

Linux is maintained and developed by volunteers, so the first advice for reporting problems and bugs is to be polite. Chances are good someone will take personal interest in this bug, and you will have a fix or a workaround in record time. You are far less likely to get a timely response if you take your frustrations out on the developers. Unlike other proprietary systems, when dealing with the Linux community, you are not dealing with underpaid droogs. You are dealing with the masters themselves, the people who take personal ownership and pride in their work. Show some respect and they will repay your thoughtfulness.

Before you report any bug, you should always check to see if this bug is known. If you have access to a Web browser, look into the linux-kernel archive (http://www.tux.org/lkml/). If you have IRC access, you can ask directly on one of the #linux, #linuxOS, or #kernelnewbie channels.

If you still think you have found a new bug in the kernel, the kernel development community will be more than interested…providing you can supply enough information to lead to a fix. If you can isolate the module where the oops occurred, you can locate the author of that module either in the linux/MAINTAINERS file or in the source code of the module itself. You can also post your report to the linux-kernel mailing list.

When reporting a suspected bug, you should specify which kernel you are using, outline your hardware setup (RAM, CPU, and so on), and describe the situation where the problem occurred. If there is a kernel panic message, copy the message exactly as displayed on your screen.

Summary | Next Section

Previous Section