Merry Christmas – Desaster Recovery with Veeam
The Story
The StoryIt was approx. 6am when my little girl woke me up on Dec. 24th, the day we in Austria celebrate Christmas. Just a small side note. Unlike US and other countries in Austria the christ-child and not Santa Clause is bringing the presents for the kids. You can image how I felt when I was ripped out of a dream/sleep. After she was dressed up I decided to take a little nap on the couch but she insisted that she liked to dance and I should play some music. So I connected to my Sonos but unfortunately none of my MP3s were accessible.
My first thought was that my Ubiquiti update of the UAPs the day before didn’t worked well but I could connect to the internet so that was not the issue. I tried to connect to my Synology DS1515+ but no login showed up. Pinging it showed that it didn’t respond. So I dragged my tired body to the basement and “hurra” my DS1515+ went black. That was really great and all of this happens on Christmas Eve.
Long story short. After some research it looks like that my Synology, which is powered by an Intel Atom C2538 processor, faced the Intel C2000 bug. I also tested the PSU but the fan began to spin so I assumed the power supply is ok. So there were good news and bad news.
Good news
- I run Hyper Backup on my DS1515+ and Hyper Vault on my backup DS415+. So photos, videos, MP3 etc. are save.
- I run also a BackBlaze Cloud Backup for my photos archive
- I already moved some critical VMs (DC, vCenter, PSC, vRealize Loginsight, vRealize Operations) to my 2-node vSAN cluster
- DS1515+ is covered by 3 years warranty and I bought it July 2015
Bad news
- My backup DS415+, also powered by the C2538, can maybe fail in the near future.
- I didn’t move every critical VM to the new 2 node vSAN cluster (Remote Access Server, Veeam Server, CA, 2nd vCenter etc.)
- Services like Plex and Sonos are currently down at my home
I also opened a RMA with Synology but I think the speed this case will be handled during the holiday season will not be that fast. In the meantime I need to restore some of the critical VMs which are not already on the vSAN cluster. But unfortunately also the Veeam Backup Server was one of the VMs I need to recover.
Recover Veeam Backup Server
There are two ways to restore the Veeam Backup Server from a backup.
- Use the Veeam Extract Utility
- Install a new Veeam Backup Server from scratch
Following I will go through the pro and con of each option.
Veeam Extract Utility
Veeam Extract Utility is a small tool that can be used to extract backup data from a VBK file. You can extract this tool directly from the installation medium. To extract the tool mount the ISO file of Veeam Backup and Replication 9.5, navigate to folder Backup and open VBR-SRV.cab with the zip tool of your choice. You need to extract the following files:
- extract
- extract.exe
- veeam.backup.extractor.exe1
To use the GUI you need to rename veeam.backup.extractor.exe1 to veeam.backup.extractor.exe.
Pro/Con
Veeam Extract Utility is a handy tool where you can quickly extract VM data from VBK files. The extracted VM data can be copied to an ESXi datastore and registered to the inventory. The biggest drawback of this tool is that it can only use VBK files. To understand why this is a drawback let me quickly explain the backup methodes of Veeam Backup and Replication.
Forever Forward Incremental
When using Forever Forward Incremental, Veeam will create a Full Backup (VBK) on the first run. All subsequent backups are only incremental backups (VIB).
Forward Incremental
Forward Incremental is similar to Forever Forward Incremental. The big different is that a synthetic full or active full backup will be created periodically.
Reverse Incremental
Reverse Incremental creates a Full Backup (VBK) on the first run. During subsequent backup sessions Veeam is only backing up changed data blocks. This data blocks will be “injected” into the full backup to rebuild it to the most recent state. In addition to that Veeam is creating reverse incremental backup files (VRB) containing the replaced data blocks from the last rebuild of the full backup file.
As you can see, using just VBK files have the following drawback:
- Forever Forward Incremental – VBK can be ages old, depending on when the first backup was created.
- Forward Incremental – depending on how often synthetic full or active full backup will be created (typically once a week) your RPO can be 1 week.
- Reverse Incremental – looks like the best choice here. But what if the VM was already corrupt during the last backup cycle?
So because of this constraint Veeam Extract Utility wasn’t my choice to restore the Veeam Backup Server.
Install a new Veeam Backup Server from scratch
Because Veeam Extract Utility wasn’t the right tool, I made the decision to install a new Veeam Backup Server from scratch and do a restore from my original Veeam Backup Server.
Pro/Con
Someone could ask “Bro isn’t this a lot of work”. Short answer is “It depends”. In this particular case the answer was “No”. The reason for that was that my main infrastructure was still working, so I had my PSC and vCenter Appliance survided. If the whole infrastructure were down, so no vCenter, no Templates etc. then it would have been a little bit more work because I had to install a Windows VM from scratch and maybe recover PSC/vCenter first.
Fortunately I was lucky because I had a template available on my Backup Synology box. After approx. 30min my temporary Veeam Backup Server was up and running. I added the vCenter to the backup infrastructure and imported the backups from my backup repository. Then I recovered my original Veeam Backup Server. After that it was piece of cake to recover the remaining failed services like my remote access server or the second vCenter for my AF vSAN cluster.
Conclusion
One of my lessons learned was that even when having a backup it can be really complicated to restore individual VMs if the main backup server is affected. With Veeam it was very easy to create a temporary backup server for restoring the most important services. For my future desaster recovery plan I will also include the most underrated/underused feature namely replication to replicate my critical VMs to another Synology box.
References
Veeam Backup method graphs – Veeam User Guide for VMware vSphere
Desaster Recovery graph – https://mapolce.com/tag/disaster-recovery-plan/