This morning we suffered a hardware failure in one of our OVM lab servers (version 3.4). In this environment, OVM Manager is deployed inside the OVM cluster, and the incident crashed its virtual machine. At VM startup using xm create command from the OVM Server command line, OS startup up OK, but OVM components did not. Why? Mysql data corruption after the failure.

It dos not look good, but even first thought was start working at MySQL level, reality is that this environment is very static and there were no changes in previous 24h, so we opted for fast and integrated VOM solution, MySQL restoration tool. It was enough to follow the clear Oracle’s documentation we can find here. In this case, we only need to restore the database, so we do not execute step one.

OVMM automaticaly creates a daily full MySQL backup and stores it in the path /u01/app/oracle/mysql/backup (OVM 3.4). Its contents in our server are:

Last copy, performed yesterday, is AutoFullBackup-20180122_170402. With this information, we firstly stop all OVMM processes (Weblogic OVM Server, Oracle VM Command Line Interface and MySQL database):

Now, as oracle user, we launch RestoreDatabase.sh shell script, specifying the name of the copy we will use for data restoration.

It’s time to start services again.

Everything is OK, included the DB. Back to normal activity!

%d bloggers like this: