Every so often it is necessary to reboot a computer. When things are behaving strangely and your ‘pute has been up for a reasonably long amount of time, shutting everything down and powering up again is a great first step to diagnosing the issue, and often resolves the problem.

A few months ago, I noticed that my server occasionally wouldn’t come back up after a reboot. This was particularly a pain to troubleshoot becuase my server is headless (read: no monitor). After lugging the machine over to my TV and hooking it up, I powered it up and was greeted with a black screen displaying a flashing cursor.

Uh oh.

This caused quite the scare the first time this happened. Why can’t it find the OS?? After a few Google searches (“Ubuntu won’t boot”, etc.), I found a boot repair program, and was hopeful that it would resolve my issue. Boot-Repair reinstalls the grub bootloader, which would allow my mobo’s BIOS to read my SSD as a bootable disk. Sounds like it’s worth a try to me!

I shut down my server, plugged in my bootable Ubuntu 14.04 Desktop USB, changed my BIOS settings to use my recovery USB as its boot disk, and powered the machine back up.

Once booted, I used the aforementioned boot repair program to install grub on my SSD. After it was done doing its dance, I again changed my BIOS boot settings to boot into my SSD, shut down, held my breath, and booted.

I saw the trusty (pun intended1) wine–colored Ubuntu background, and was hopeful. A few seconds later, I saw my desktop background appear and nothing looked out of place. It worked!

Sigh of relief

Over the next few months, the issue would recur seemingly randomly, and it got to the point where I was reluctant to reboot due to fear of it not coming back up. If I was at my apartment, it was less of an issue, because I knew how to resolve the problem if it wasn’t coming back up, but it was a different story when I was out of town. I also couldn’t bear to just get used to the idea of reinstalling grub every time this happened. I’m an engineer; we like to solve problems, not live with them. I also couldn’t accept never rebooting my server, because as I said earlier, this is a great first step of troubleshooting issues.

So onto getting to the bottom of this. I knew that this probem was grub related, since reinstalling grub consistently resolved the issue. I first hypothezized that there was a permissions issue with /etc/grub, the file that stores the grub configuration, and some nefarious program was corrupting it. This couldn’t be the case though, since this file is only writable by root, as a quick ls -lsa revealed. I troubleshooted further to no avail.

Finally, one day I had to reboot a few times (without a hitch) and then I updated my software.

$sudo apt-get update$ sudo apt-get upgrade


I then continued troubleshooting whichever problem I was facing at the time and rebooted. My server didn’t come back up. This couldn’t have been coincidental; whatever I updated must have broken grub! I felt excited that I was close to finally closing this case.

I checked aptitude to see what was recently updated (Source: StackOverflow).

cat /var/log/dpkg.log | grep "\ install\ "


I noticed grub in the list of programs. That must be the culprit. For some reason, grub wasn’t updating properly.

A few minutes of searching Google revealed the command I needed to find discover my needle in this boot issue haystack (Source: askubuntu).

\$ sudo debconf-show grub-pc
...
* grub-pc/install_devices: /dev/disk/by-id/[My USB drive ID here]
...


This command lists, among other things, where grub is installed. As you can see from the output above, when grub is updated, it installs on my USB stick, not my SSD. This made sense to me because when I first built my server, I was booting from a USB 3.0 drive. This ended up not being practical; USB drives (or at least the one I had) don’t have the IOPS to support the constant reads and writes that occur on an OS drive. There are, of course, exceptions to this; I know at least that the FreeNAS Linux flavor is meant to be booted from a USB drive.

To change grub’s install location to my SSD, I ran

sudo dpkg-reconfigure grub-pc


This command opens a set of prompts that guide you through changing the grub install destination. The most important page is the grub install devices page. Use the arrow keys and the space key to select the drive(s) that you would like to contain grub, and then press the return key to finish the configuration. As a sanity check, I re-ran debconf-show grub-pc to verify that it listed my SSD as the install location, which it did.

Since making this change, I have updated grub and rebooted many times without issue, and my hesitation to do so has vanished. I am relieved that this case is officially closed.

If you have any questions, comments, or corrections, please post them in the comments below and I’ll try to get back to you as soon as I can. Thank you.

1. “Trusty Tahr” is the codename for Ubuntu 14.04 LTS