When I was migrating my Exchange platform from 2010, I ensured that the backups using Veritas NetBackup were also working in Exchange 2013, both against the active or passive node of the DAG, and also the granular backups
Symptoms
At a certain moment, the backups started to fail (thanks I am monitoring the transaction log disks free space using nagios). But changing the configuration to make the backups being performed against the Active node was valid as a temporal workaround.
So it only fails when using the passive node, without knowing exactly when and what caused the problem.
This is the error:
31/05/2016 15:39:33 - Info nbjm(pid=4696) starting backup job (jobid=804387) for client EXCHANGE2.DOMAIN.COM, policy EXCH_Buzones2013, schedule Full_Senanal
31/05/2016 15:39:33 - Info nbjm(pid=4696) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=804387, request id:{D063E0BC-99B8-4733-A52B-C2D8A248F0D6})
31/05/2016 15:39:33 - requesting resource dedup_backupapp
31/05/2016 15:39:33 - requesting resource BACKUPSRV.DOMAIN.COM.NBU_CLIENT.MAXJOBS.EXCHANGE2.DOMAIN.COM
31/05/2016 15:39:33 - requesting resource BACKUPSRV.DOMAIN.COM.NBU_POLICY.MAXJOBS.EXCH_Buzones2013
31/05/2016 15:39:33 - granted resource BACKUPSRV.DOMAIN.COM.NBU_CLIENT.MAXJOBS.EXCHANGE2.DOMAIN.COM
31/05/2016 15:39:33 - granted resource BACKUPSRV.DOMAIN.COM.NBU_POLICY.MAXJOBS.EXCH_Buzones2013
31/05/2016 15:39:33 - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=dp_dedupdisk_backupapp;Path=PureDiskVolume;StorageServer=backupapp;MediaServer=backupapp
31/05/2016 15:39:33 - granted resource dedup_backupapp
31/05/2016 15:39:33 - granted resource TRANSPORT
31/05/2016 15:39:33 - estimated 1304705925 Kbytes needed
31/05/2016 15:39:33 - begin Parent Job
31/05/2016 15:39:33 - begin Exchange 14 Snapshot, Step By Condition
Status 0
31/05/2016 15:39:33 - started
31/05/2016 15:39:34 - end Exchange 14 Snapshot, Step By Condition; elapsed time: 0:00:01
31/05/2016 15:39:34 - begin Exchange 14 Snapshot, Read File List
Status 0
31/05/2016 15:39:34 - end Exchange 14 Snapshot, Read File List; elapsed time: 0:00:00
31/05/2016 15:39:34 - begin Exchange 14 Snapshot, Create Snapshot
31/05/2016 15:39:35 - started process bpbrm (4656)
31/05/2016 15:39:36 - Info bpbrm(pid=4656) EXCHANGE2.DOMAIN.COM is the host to backup data from
31/05/2016 15:39:36 - Info bpbrm(pid=4656) reading file list for client
31/05/2016 15:39:36 - Info bpbrm(pid=4656) start bpfis on client
31/05/2016 15:39:36 - Info bpbrm(pid=4656) Starting create snapshot processing
31/05/2016 15:39:36 - Info bpfis(pid=30648) Backup started
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL -
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL -
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL - snapshot processing failed, status 156
31/05/2016 15:40:14 - Critical bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: FTL - snapshot creation failed - Error attempting to gather metadata., status 130
31/05/2016 15:40:14 - Info bpbrm(pid=4656) DB_BACKUP_STATUS is 156
31/05/2016 15:40:14 - Warning bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: WRN - NEW_STREAM0 is not frozen
31/05/2016 15:40:14 - Warning bpbrm(pid=4656) from client EXCHANGE2.DOMAIN.COM: WRN - Microsoft Information Store:\STORE5 is not frozen
31/05/2016 15:40:14 - Info bpfis(pid=30648) done. status: 130
31/05/2016 15:40:14 - end Exchange 14 Snapshot, Create Snapshot; elapsed time: 0:00:40
31/05/2016 15:40:14 - Info bpfis(pid=30648) done. status: 130: system error occurred
31/05/2016 15:40:14 - end writing
Status 130
31/05/2016 15:40:14 - end Parent Job; elapsed time: 0:00:41
31/05/2016 15:40:14 - begin Exchange 14 Snapshot, Stop On Error
Status 0
31/05/2016 15:40:14 - end Exchange 14 Snapshot, Stop On Error; elapsed time: 0:00:00
31/05/2016 15:40:14 - begin Exchange 14 Snapshot, Delete Snapshot
31/05/2016 15:40:15 - started process bpbrm (4963)
31/05/2016 15:40:16 - Info bpbrm(pid=4963) Starting delete snapshot processing
31/05/2016 15:40:17 - Info bpfis(pid=35760) Backup started
31/05/2016 15:40:17 - Critical bpbrm(pid=4963) from client EXCHANGE2.DOMAIN.COM: FTL - cannot open C:\Program Files\VERITAS\NetBackup\online_util\fi_cntl\bpfis.fim.EXCHANGEDAG.DOMAIN.COM_1464701973.1.0
31/05/2016 15:40:18 - Info bpfis(pid=35760) done. status: 4207
31/05/2016 15:40:18 - end Exchange 14 Snapshot, Delete Snapshot; elapsed time: 0:00:04
31/05/2016 15:40:18 - Info bpfis(pid=35760) done. status: 4207: Could not fetch snapshot metadata or state files
31/05/2016 15:40:18 - end writing
Status 4207
31/05/2016 15:40:18 - end Parent Job; elapsed time: 0:00:45
Status 130
31/05/2016 15:40:18 - end operation
system error occurred (130)The log shows a problem with the snapshot backup (VSS).
Resolution
Snapshot options
One of the possible solutions is found in this Veritas KB: Exchange backup Snapshot processing is failed Error 130
The symptoms described here are the system error 130 and the snapshot processing failed 156. Both appear on the above logs.
So I applied it changing the "Snapshot options" values of the backup policy from 0 0 1 to 1 0 1.
This didnt solved the problem, but that does not mean that it is not a neccesary step (anyway, I left this configuration with the new value).
Exchange 2013 CU11 to a newer CU
This Veritas KB is very explicit: After upgrading Exchange 2013 to Cumulative Update 11 (CU11), Exchange backup may fail with either status code 69 or 130
It seems CU11 introduced a new feature called "mailbox anchoring" that makes this backups fail. This functionality is reverted with CU12. After upgrading to CU12 my backups started to work correctly again
If upgrading to CU12 is not an option for you, the article shows another workaround:
Workaround:
If applying 2013 CU12 or above is not an option, the following procedure may be performed on all Exchange mailbox servers:
- Launch Registry Editor (regedit).
- Browse to HKEY_LOCAL_MACHINE > SOFTWARE > Veritas > NetBackup > BEDS > Engine > Exchange.
- On the right hand pane, create a new DWORD (32-bit) value. For the name, type: PowerShellOptions, and hit Enter.
- Double click PowerShellOptions and enter 1 as the value data. Hit Enter again to save.
Setting PowerShellOptions to 1 forces monad (the NetBackup process responsible for running PowerShell queries) to connect to local Exchange Management Shell session, thereby bypassing the need to interact with mailbox anchoring.
NOTE:
If using Exchange 2013 CU11, do not set the NetBackup PowerShellOptions Registry value unless GRT backups fail to catalog mailboxes.
If GRT backups fail to catalog mailboxes with Exchange 2013 CU11 and you have archive mailboxes please upgrade to CU12.
