There are several ways we can find out what triggered a reboot. In this article, we’re going to discuss those ways and how you can utilize available utilities and logs in a Linux system to troubleshoot such scenarios.
Inspect Reboot Time
You can check when the system reboot happened with who and last commands
Check System Messages
You can further correlate the reboot you want to diagnose with system messages. For CentOS/RHEL systems, you’ll find the logs at /var/log/messages while for Ubuntu/Debian systems, its logged at /var/log/syslog. You can simply use the tail command or your favorite text editor to filter out or find specific data. As can be inferred from the below logs, such entries suggest a shutdown/reboot initiated by an administrator or root user. These messages can vary depending upon OS type and the way reboot/shutdown is triggered but you will always find useful information by looking at system logs though it may not be explicit enough to pin-point the cause every time. One such command that you can use to filter out system logs is given below: Captured events may not always be specific. Always trace out events that give signs of warnings or errors which may lead to the system powering off/crashing.
Verify auditd Logs
For systems with auditd, it is a great place to check different events using ausearch tool. Use the below command to check the last two entries from audit logs. This will report the two most recent shutdowns or reboots. If this reports a SYSTEM_SHUTDOWN followed by a SYSTEM_BOOT, everything should be good. But, if it reports two SYSTEM_BOOT lines in a row or only a single SYSTEM_BOOT line, then most likely, the system did not shut down gracefully. A normal output should be something like below: The below output lists two consecutive SYSTEM_BOOT messages, which may indicate an ungraceful shutdown though it needs to be correlated with system logs.
Analyze systemd journal
You should have a persistent systemd-journal in order to keep a persistent journal on disk else the logs won’t persist on reboot. For this, you can either make the changes in /etc/systemd/journald.conf or create the directory yourself with the below commands: Once done, you can optionally reboot the system to capture more than one reboot entry in the journal though it is not required. Use the below command to list logged boots from the journal: Here’s its output on my server: As you can see it is listing lasts several boots. To further analyze a particular reboot, use: Here {num} will be the index given in journalctl –list-boots command in the first column. You can observe messages logged in the journal in the above output and can trace out the anomalies if any.
Conclusion
It may not always be possible to pinpoint the cause of a Linux reboot using a single command or from a single log file. As such, it’s always handy to know the commands and logs which capture system-related events and can shorten the time required to find the root cause. The above examples provide you a starting place to begin your troubleshooting. Using a combination of such tools and logs, you can be confident to know what happened and how your system rebooted. Next, find out some of the light-weights monitoring software for Linux.
