Troubleshooting an Express Query Manual Intervention Failure (MIF)
An Express Query Manual Intervention Failure (MIF) is a critical error that occurred during Express
Query execution. These are failures Express Query cannot recover from automatically. After a MIF
occurrence the specific file system is logically removed from the Express Query and it requires a
manual intervention to perform the recovery. Although these errors inhibit the normal functionality
of the Express Query, they are typically due to another unrelated event in the cluster or the file
system. Therefore, most of the work to recover from an Express Query MIF is to check the health
of the cluster and the file system and take corrective actions to fix the issues caused by these events.
Once the cluster and file system have an OK status, the MIF status can be cleared since the Express
Query service will be recovering and restarting automatically.
In some very rare cases, a database corruption might occur, as a result of these external events
or from some internal dysfunction. Express Query contains a recovery mechanism that tries to
rebuild the database from information Express Query is keeping specifically for that critical situation.
Express Query might be unable to recover from internal database corruption. Even though it is
unlikely, it is possible and it might occur in the following two cases:
•
A corrupted database has to be rebuilt from data that has been already backed up. If the
data needed has been backed up, there is no automated way for Express Query to recover
since that information has been deleted from the StoreAll file system after the backup. It is
however possible to replay the database logs from the backup.
•
Some data needed to rebuild the database is corrupted and therefore it cannot be used.
Even though database files, as well as information used in database recovery are well protected
against corruption, corruption occurrence might occur.
NOTE:
When a file system is in the MIF state, Express Query event recording is still occurring.
When the database is re-enabled, the recorded events are processed and the database is
synchronized with the file system again.
To recover from an Express Query Manual Intervention Failure (MIF):
1.
Check the health of the file system as described in the
“Monitoring cluster operations”
(page 83)
, and clear any pending issues related to the file system (
<FSNAME>
).
2.
Clear the Express Query MIF state by entering the following command:
ibrix_archiving -C <FSNAME>
3.
Monitor the Express Query recovery by entering the following command:
ibrix_archiving -l
While the Express Query is recovering from MIF, it displays the RECOVERY state. Wait for
the state to return to
OK
or
MIF
.
If the state returns as
OK
, no additional steps are required. The Express Query is updating the
database with all the outstanding logged file system changes since the MIF occurrence.
4.
If you have a MIF condition for one or several file systems and cluster and file system health
checks are not
OK
, redo the previous steps.
152
Troubleshooting
