Windows Server Backup: Vague "I/O device error" message

At my workplace, I administer a Windows Server 2008 RC2 web server that makes daily backups to an external hard drive using Windows Server Backup. The backups have been consistently failing recently, and I've found out why.

Even though the backups were not finishing successfully, Windows Server Backup labeled the backups as “Completed with warnings” because it was able to successfully backup some of the data before failing. The backups would always fail at the same point: after backing up approximately 12 GB of data. The failures were intermittent at first, but became more consistent over time to the point where they were happening every night.

Error message

The error message Windows Server Backup was giving me was very vague:

"The operation failed due to a device error encountered with either the source or destination. Detailed error: The request could not be performed because of an I/O device error."

 The error message in the event logs was equally vague:

“The backup operation that started at has completed with errors. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.”

(Contrary to the message’s suggestion, I could not find anything useful in the event details.)

Bad sector?

Many websites suggested that this error could be caused by a bad sector on the source or destination drive. I ran chkdsk against the 1 TB external hard drive that the backups were being stored on. The command took about 5 hours to run. It was alarming to see that the command consumed 4 GB of memory while running, but according to numerous websites, this is normal. No errors were found.

Unfortunately, I couldn’t run chkdsk on the server’s hard drive because it would require that the machine be put out of service.  Chkdsk cannot be run on the system drive while the server is operational—it must reboot and then run before booting into Windows. Since the server hosts our website and another library’s website, I couldn’t take it offline without advance notice. So, that wasn’t an option.

USB port?

I tried plugging the external drive into a different USB port, thinking maybe the USB port was bad. A long shot, but you never know. No effect.

Large file?

One website suggested it could be failing because a file was too large. I knew of one particularly large file on the system: “C:\MySQL Data files\ibdata1”. This file is where MySQL stores its databases. The file is over 1 GB, so I tried excluding that file from the backup.

(In order exclude specific files from the backup, I had to disable the “Bare metal recovery” option from the backup settings.)

Reparse points

Before I could complete the configuration wizard, I got an error message:

“One of the paths specified for backup is under a reparse point. Back up of files under a reparse point is not supported. Specify a file path that contains the destination of the reparse point, and then retry the operation.”

What is a reparse point? A reparse point is kind of like a shortcut. For example, there is a reparse point for the old “C:\Documents and Settings” folder that just redirects to the “C:\Users” folder.

To see all the system’s reparse points, type the following command at the root of the drive:

dir /s /al

I found that the server had a LOT of reparse points. Most of them were in the user folders and were for the old Windows XP common folders names (for example, a reparse point named “My Pictures” existed, which redirected to the “Pictures” folder)

Due to the number of reparse points on the system, excluding all of them individually wasn’t a practical option. I decided I would just exclude the “Documents and Settings” reparse point to see if that would be enough.

This time, I could complete the configuration wizard and run the backup. The backup even completed successfully (though, it took 8 hours to complete, which is much longer than the 1 hour it used to take).

The solution

The backup was successful, but it failed to back up a one, single file—an old IIS log file. The error message was:

“Error in backup of during read: Error [0x8007045d] The request could not be performed because of an I/O device error.”

Another vague “I/O device error” message. I tried copying the file to the desktop to see if the file could be read. The copy operation failed with a similar error.

This made me think that there was something corrupted about the file, which was causing the backup to fail. Maybe a bad sector on the hard drive. I deleted the file, changed the backup settings back to “Bare metal recovery” and ran the backup again. The backup succeeded!