Alain Kelder is a Giant Dork

Fix for Linux boot failure and grub rescue prompt following a normal system update

I had a fresh Debian 7 (Wheezy) install on a new server. Some months later, on Dec 21 2013, ran "sudo apt-get update && sudo apt-get dist-upgrade", after reboot, system wouldn't boot, instead dropping me to a "grub> " prompt.

Booted to the sysresccd to check my partitions:

$ sudo parted -ls

Model: SMC SMC2108 (scsi)
Disk /dev/sda: 16.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  15.9TB  15.9TB  ext4
 3      15.9TB  16.0TB  68.6GB  linux-swap(v1)

Mounted the second partition /dev/sda2 on /mnt to check /boot/grub/grub.cfg for clues:

$ grep "set root" /boot/grub/grub.cfg 

set root='(hd1,gpt2)'
  set root='(hd1,gpt2)'
	set root='(hd1,gpt2)'
	set root='(hd1,gpt2)'

That seemed odd, because hd0 should be the first disk (i.e. /dev/sda in my case), while hd1 should be the second disk (e.g. /dev/sdb), which this system didn't have.

Rebooted, and at the grub prompt entered:

grub> ls
(hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1)

grub> search --file /vmlinuz
 hd0,gpt2

Indeed, there's no hd1, just hd0. Ok, let's try a manual boot from grub:

grub> set root=(hd0,gpt2)
grub> linux /vmlinuz root=/dev/sda2
grub> initrd /initrd.img
grub> boot

With that, grub was able to boot Linux. Sweet.

To fix this permanently, ran:

$ sudo dpkg-reconfigure grub-pc

Then answered questions, and hit "Ok":

 ┌───────────────────────────────────────┤ Configuring grub-pc ├───────────────────────────────────────┐  
 │ The following Linux command line was extracted from /etc/default/grub or the `kopt' parameter in    │  
 │ GRUB Legacy's menu.lst. Please verify that it is correct, and modify it if necessary. The command   │  
 │ line is allowed to be empty.                                                                        │  
 │                                                                                                     │  
 │ Linux command line:                                                                                 │  
 │                                                                                                     │  
 │ ___________________________________________________________________________________________________ │  
 │                                                                                                     │  
 │                                                                                                 │  
 │                                                                                                     │  
 └─────────────────────────────────────────────────────────────────────────────────────────────────────┘  

Left as is -- empty.

  ┌──────────────────────────────────────┤ Configuring grub-pc ├───────────────────────────────────────┐  
  │ The following string will be used as Linux parameters for the default menu entry but not for the   │  
  │ recovery mode.                                                                                     │  
  │                                                                                                    │  
  │ Linux default command line:                                                                        │  
  │                                                                                                    │  
  │ quiet_____________________________________________________________________________________________ │  
  │                                                                                                    │  
  │                                                                                                │  
  │                                                                                                    │  
  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  

Removed "quiet" -- I like to see the boot process.

 ┌───────────────────────────────────────┤ Configuring grub-pc ├───────────────────────────────────────┐  
 │ The grub-pc package is being upgraded. This menu allows you to select which devices you'd like      │  
 │ grub-install to be automatically run for, if any.                                                   │  
 │                                                                                                     │  
 │ Running grub-install automatically is recommended in most situations, to prevent the installed      │  
 │ GRUB core image from getting out of sync with GRUB modules or grub.cfg.                             │  
 │                                                                                                     │  
 │ If you're unsure which drive is designated as boot drive by your BIOS, it is often a good idea to   │  
 │ install GRUB to all of them.                                                                        │  
 │                                                                                                     │  
 │ Note: it is possible to install GRUB to partition boot records as well, and some appropriate        │  
 │ partitions are offered here. However, this forces GRUB to use the blocklist mechanism, which makes  │  
 │ it less reliable, and therefore is not recommended.                                                 │  
 │                                                                                                     │  
 │ GRUB install devices:                                                                               │  
 │                                                                                                     │  
 │    [*] /dev/sda (15991991 MB; SMC2108)                                                              │  
 │    [ ] - /dev/sda2 (15923339 MB; /)                                                                 │  
 │                                                                                                     │  
 │                                                                                                     │  
 │                                                                                                 │  
 │                                                                                                     │  
 └─────────────────────────────────────────────────────────────────────────────────────────────────────┘  

Selected just "/dev/sda", and hit "Ok".

Replacing config file /etc/default/grub with new version
Installation finished. No error reported.
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
done

This fixed the incorrect disk reference (hd1 is now hd0):

$ grep "set root" /boot/grub/grub.cfg 

set root='(hd0,gpt2)'
  set root='(hd0,gpt2)'
	set root='(hd0,gpt2)'
	set root='(hd0,gpt2)'

Did a test reboot, and Linux booted fine. Hooray.

I'm sure I could have manually overridden grub.cfg, but then the problem would have probably returned on each kernel update.

Postmortem follow up

Checked the dpkg.log, and during the previous update in Sept, the following grub packages were installed:

2013-09-30 19:11:14 status installed grub-common:amd64 1.99-27+deb7u1
2013-09-30 19:11:18 status installed grub2-common:amd64 1.99-27+deb7u1
2013-09-30 19:11:18 status installed grub-pc-bin:amd64 1.99-27+deb7u1
2013-09-30 19:11:19 status installed grub-pc:amd64 1.99-27+deb7u1

Then during the Dec update, which broke the ability to boot, I got:

2013-12-21 19:06:14 status installed grub2-common:amd64 1.99-27+deb7u2
2013-12-21 19:06:14 status installed grub-common:amd64 1.99-27+deb7u2
2013-12-21 19:06:14 status installed grub-pc-bin:amd64 1.99-27+deb7u2
2013-12-21 19:06:20 status installed grub-pc:amd64 1.99-27+deb7u2

Not sure the problem had anything to do with the grub package version -- more likely, I suspect, the problem might had been caused by me having a USB thumbdrive plugged in or something like that during the system update, which caused my main disk to be /dev/sdb, causing the grub update process to record hd1 instead of hd0.

Anyway, hope this helps someone else.