201.3 Kernel runtime management and troubleshooting

Introduction

Runtime kernel management connects hardware detection, modules, /proc, sysctl, and udev. The safest workflow is to inspect current state, test a temporary change, and persist it only after verification.

What you should be able to do after this lesson:

• Inspect the running kernel and its modules.
• Load, unload, parameterize, and alias modules.
• Read and change tunables through sysctl.
• Trace device events and test udev rules.
• Diagnose common boot-time and runtime hardware failures.

Big Idea: Follow the Device from Detection to Policy

For a hardware problem, trace this sequence:

bus detection -> kernel driver -> module parameters -> device node -> udev policy -> application

Each tool observes a different stage. lspci can show a device even when no driver is bound; lsmod can show a driver even when udev created the wrong permissions; an application can fail even when every kernel stage is healthy.

Inspect Kernel and Hardware State

uname -r
dmesg --level=err,warn
journalctl -k -b
lspci -nnk
lsusb -t

dmesg and the kernel journal reveal probe failures, firmware problems, storage errors, and module messages. lspci -nnk connects a PCI device to its current kernel driver.

Module Management

lsmod
modinfo <module>
modprobe <module>
modprobe -r <module>

modprobe uses dependency and alias data under /lib/modules/$(uname -r)/. insmod inserts one module file directly and does not resolve dependencies. rmmod removes one module, while modprobe -r can account for dependencies.

Before unloading a module, verify that no mounted filesystem, network interface, or process still depends on it.

Parameters and aliases

modinfo -p <module>
modprobe <module> option=value

Persistent settings commonly live in /etc/modprobe.d/*.conf:

options example option=value
alias alternate_name example
blacklist unwanted_module

Run depmod -a after manually changing module files or indexes.

The generated modules.dep file records module dependencies. Do not edit it by hand. Historical tools such as lsdev and udevmonitor may appear in older material; current systems use interfaces such as lspci, /proc, and udevadm monitor.

/proc and sysctl

/proc exposes process and kernel runtime state. Many tunables appear below /proc/sys/ and are managed with sysctl.

sysctl kernel.hostname
sysctl net.ipv4.ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

Persistent settings belong in /etc/sysctl.conf or /etc/sysctl.d/*.conf:

net.ipv4.ip_forward = 1

Apply configured settings with:

sudo sysctl --system

A direct write to /proc/sys/... and sysctl -w are temporary until stored in configuration.

udev Device Management

udev receives kernel device events and creates names, permissions, and links in /dev. Local rules commonly live under /etc/udev/rules.d/.

Inspect a device and its parent attributes:

udevadm info --query=all --name=/dev/sdb
udevadm info --attribute-walk --name=/dev/sdb

Watch events:

udevadm monitor --kernel --udev --property

After changing a rule:

sudo udevadm control --reload-rules
sudo udevadm test /sys/class/block/sdb
sudo udevadm trigger

Use udevadm test before triggering broad changes. Match stable properties such as vendor IDs or serial numbers rather than transient kernel names.

Example local rule:

SUBSYSTEM=="tty", ATTRS{idVendor}=="1234", ATTRS{idProduct}=="5678", \
  SYMLINK+="lab-console", GROUP="dialout", MODE="0660"

The rule uses parent-device attributes discovered by udevadm info --attribute-walk. It creates a stable symlink without depending on whether the kernel called the device ttyUSB0 or ttyUSB1.

Troubleshooting Workflow

1. Confirm the running kernel and expected module directory match.
2. Check kernel logs for the first device or firmware error.
3. Verify the device is visible on its bus.
4. Inspect the bound driver and module parameters.
5. Test a temporary module or sysctl change.
6. For naming or permission issues, monitor udev and test the matching rule.
7. Persist only the proven setting.

Guided Practice: Trace One Device

Choose a noncritical USB or PCI device and record its path:

lspci -nnk
lsusb -t
udevadm info --query=path --name=/dev/<device>
udevadm info --query=property --name=/dev/<device>

In another terminal, start a monitor and reconnect the device if doing so is safe:

sudo udevadm monitor --kernel --udev --property

Identify the kernel event, the udev event, the selected driver, and at least one stable property. Do not write a rule until you can explain which stage currently produces the undesired behavior.

Troubleshooting Scenario

A serial adapter appears in lsusb, but the expected /dev/lab-console link is absent. The driver is loaded and /dev/ttyUSB0 exists. udevadm test shows that the local rule never matches because idVendor belongs to a parent object and the rule used ATTR instead of ATTRS.

Correct the match, reload the rules, test the exact sysfs path, and trigger only the relevant device. Reloading the driver would not repair a udev match error.

Exam Focus

• modprobe resolves aliases and dependencies; insmod inserts one file directly.
• Runtime sysctl changes disappear unless stored in /etc/sysctl.conf or /etc/sysctl.d/.
• modules.dep is generated by depmod.
• Use udevadm monitor, info, test, and trigger for different parts of rule diagnosis.

Recap

• modprobe is dependency-aware; insmod is not.
• sysctl changes are temporary unless saved below /etc/sysctl.conf or /etc/sysctl.d/.
• udev translates kernel events into useful device behavior.
• Logs and hardware enumeration should come before configuration changes.