Does “wbengine” of Windows Server 2019 support incremental backups when targeting network shares?

November 3, 2019, 9:12 am

≫ Next: Why is "wbengine" of Windows Server 2008 R2 always creating new VHD files when targeting network shares?

This is a follow up to things I've written down in an answer to another question:

How does Windows Image Backup decide which files to backup?

Windows/NTFS manages [change journals][2] on each volume, keeping track of which file has changed, how it has changed and because those are part of all NTFS volumes by default, they are available in the VHD(X) of the backup target as well. From my understanding, `wbengine` simply compares those journals to know which files need backup. This makes it easy to support different backup targets without caring about archive bits of files or stuff like that, because those change journals are bound to volume-unique file-IDs and those IDs are exactly the same in the backup target:

C:\>fsutil file queryfileid "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"

Datei-ID: 0x000000000000000000080000000ea5f1

C:\>fsutil file queryfileid "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"

Datei-ID: 0x000000000000000000080000000ea5f1

In the above example, `C:\` is my current system volume while `D:\` is the mounted last backup on one of the two available backup targets. Even file sizes, timestamps etc. are all the same:

C:\>dir /A "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
 Datenträger in Laufwerk C: ist System
 Volumeseriennummer: 266B-2863

 Verzeichnis von C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin

03.02.2016  17:08           560.640 perlapp.exe
1 Datei(en),        560.640 Bytes
0 Verzeichnis(se), 1.337.040.216.064 Bytes frei

C:\>dir /A "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
 Datenträger in Laufwerk D: ist System
 Volumeseriennummer: 266B-2863

 Verzeichnis von D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin

03.02.2016  17:08           560.640 perlapp.exe
1 Datei(en),        560.640 Bytes
0 Verzeichnis(se), 1.281.302.233.088 Bytes frei

By using this approach, backup can decide at any time with any older VHD(X) which files need to be backed up, as long as the current journal and the one in the image have something in common, which are entry-IDs in my understanding. Without such a shared ID, e.g. because too many I/O happend productive and the backup is too old, wbengine would simply do a full backup instead of an incremental one.

Using those journals makes it as well pretty fast to know which files to backup, because one doesn't need to iterate the whole tree of files. That's what one actually sees in practice as well: Backup seems to know pretty soon which files to backup and starts processing those instead of iterating a file tree.

Behaviour in case of network shares as backup target for Windows Server 2012+.

In older versions of Windows, wbengineseems to have always recreated VHD files, making it pretty incompatible with snapshots supported by some underlying file system like BTRFS or ZFS. Additionally, each and every backup was a full one, simply because wbengine didn't have access to any former backup to compare with.

Things seem to be a little bit different with newer versions of Windows: I'm somewhat sure that one customer of mine ran into troubles with wbadmin and Windows Server 2012 and during debugging those using Process Monitor, I verified that existing VHDX files were kept instead of deleted and recreated. I've tested this right now with Windows Server 2019 again and multiple invocations ofwbadmin led to successful backups while KEEPING inodes of VHDX files the same:

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-02 200024"
total 549063256
 271         0 d---------+ 1 user group          594 Nov  2 20:58 .
 266         0 d---------+ 1 user group          108 Nov  2 20:58 ..
 273 507813736 ----------+ 1 user group 520061190144 Nov  2 22:02 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx
3814         4 ----------+ 1 user group          776 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3816       440 ----------+ 1 user group       450488 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml
3813         8 ----------+ 1 user group         6212 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml
3815         4 ----------+ 1 user group         1192 Nov  1 22:47 BackupSpecs.xml
 272  41249064 ----------+ 1 user group  42289070080 Nov  2 22:03 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 132201"
total 549318872
 271         0 d---------+ 1 user group          594 Nov  2 20:58 .
 266         0 d---------+ 1 user group          108 Nov  3 14:19 ..
 273 507813736 ----------+ 1 user group 520061190144 Nov  3 14:19 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx
3814         4 ----------+ 1 user group          776 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3816       440 ----------+ 1 user group       450488 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml
3813         8 ----------+ 1 user group         6212 Nov  1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml
3815         4 ----------+ 1 user group         1192 Nov  1 22:47 BackupSpecs.xml
 272  41504680 ----------+ 1 user group  42289070080 Nov  3 14:21 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 133308"
total 41262660
 271        0 d---------+ 1 user group        2504 Nov  3 14:44 .
 266        0 d---------+ 1 user group         108 Nov  3 14:30 ..
3851        4 ----------+ 1 user group         776 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
3853      440 ----------+ 1 user group      450488 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Components.xml
3850        8 ----------+ 1 user group        6212 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_RegistryExcludes.xml
3840        4 ----------+ 1 user group        3138 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml
3848        4 ----------+ 1 user group        2284 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer2707761b-2324-473d-88eb-eb007a359533.xml
3844        8 ----------+ 1 user group        5386 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml
3846        4 ----------+ 1 user group        1488 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml
3839        4 ----------+ 1 user group        1628 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml
3842       24 ----------+ 1 user group       21686 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera65faa63-5ea8-4ebc-9dbd-a0c4db26912a.xml
3847        4 ----------+ 1 user group        1484 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml
3845        4 ----------+ 1 user group        2940 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml
3849        4 ----------+ 1 user group        1850 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerb2014c9e-8711-4c5c-a5a9-3cf384484757.xml
3843        8 ----------+ 1 user group        6048 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml
3838        4 ----------+ 1 user group        1746 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml
3841    13068 ----------+ 1 user group    13379228 Nov  3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writere8132975-6f93-4464-a53e-1050253ae220.xml
3852        4 ----------+ 1 user group         758 Nov  3 14:44 BackupSpecs.xml
 272 41249064 ----------+ 1 user group 42289070080 Nov  3 14:44 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx

So in theory this should allow incremental backups replacing files in-place and efficient snapshots of e.g. underlying BTRFS or ZFS at the same time. Looking at storage of the snapshots of the tested Windows Server 2019, at least some of those really share some space:

    Total   Exclusive  Set shared  Filename
424.41GiB   417.69GiB     6.72GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.29-00.00.09
446.53GiB   400.16GiB    46.37GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08
483.05GiB   448.36GiB    34.70GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08
553.78GiB   209.26GiB   344.52GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09
204.68GiB   204.63GiB     0.05GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.01-00.00.07
410.24GiB   405.37GiB     4.87GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06

It's not as much as I would expect, though. The thing is that it might be that not much space is shared even though VHDX files are reused, because when I subsequently invoke backing up C:\ of that server, backup doesn't seem to get faster.

The first backup might take longer as much data has changed, but after that is finished and with the server doing nothing, a second backup only a few minutes later should be a lot faster, because of only backing up differences of the files. But that doesn't seem to be the case, instead it seems to take the same time like before. Additionally, while BTRFS can share differences in created snapshots between different invocations of wbadmin with the exact same command line, those are much smaller than expected when really only backing up changed files:

root@[...]:~# btrfs filesystem du -s --gbytes /volume1/@sharesnap/[a-zA-Z]*/GMT*
     Total   Exclusive  Set shared  Filename
446.53GiB   400.20GiB    46.33GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08
483.05GiB   448.36GiB    34.70GiB  /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08
553.78GiB   546.54GiB     7.24GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09
  39.35GiB    37.68GiB     1.67GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.36.52
  39.35GiB    31.18GiB     8.17GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.49.03
  39.35GiB    37.98GiB     1.37GiB  /volume1/@sharesnap/[...]/GMT+01-2019.11.03-16.03.06
410.24GiB   409.72GiB     0.52GiB  /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06

That is different to what I see when backing up to my USB-disks, subsequent backups are much faster if nothing has changed. What is interesting is that others seem to be not so sure about how things behave on network shares as well:

If you create a scheduled backup job to network shared folder or a mapped network drive, 
all the backups will only be performed by full backup because network location is not a volume. 
If you need to create differential backup or incremental backup to network folder,
you need to third party backup software.

https://www.ubackup.com/windows-server/windows-server-backup-differential-4348.html

While the same people write the following, which doesn't make much sense:

Tip: You can also use WBADMIN to create incremental backup to network share, like this:
wbadmin start backup –backupTarget: \\backupshare\backup1 -allCritical -include:C: -vssFull –quiet

https://www.ubackup.com/articles/wbadmin-incremental-backup-5740.html

In case of backing up to shares, if VHD is recreated always, like seems to be the case for older versions of Windows, one doesn't get incremental backups. But even though VHDX files are kept in newer versions of Windows, it seems like only backing up changed files still doesn't work the same way like it does when using USB-disks.

Using -vssFull vs. -vssCopy didn't make any difference in my tests and according to my understanding is not even intended to do so. Those arguments are only relevant to 3rd party software and don't influence which files are backed up how. Reasons to believe so are documented in my former answer:

Influence of -vssFull and -vssCopy.

https://serverfault.com/a/990394/333397

Questions

Does `wbengine` of Windows Server 2019 support incremental backups when targeting network shares? Does anyone see any improvement in backup times and storage allocations in case of snapshots?

↧

Why is "wbengine" of Windows Server 2008 R2 always creating new VHD files when targeting network shares?

November 3, 2019, 9:09 am

≫ Next: What does wbengine do when it says it compresses/compacts VHDX files over SMB?

≪ Previous: Does “wbengine” of Windows Server 2019 support incremental backups when targeting network shares?

This is a follow up to things I've written down in an answer to another question:

I'm somewhat sure that `wbengine` of Windows Server 2008 R2 always creates new VHD files if targeting a network share. Here's what some docs and other people say to that topic:

If you save a backup to a remote shared folder, that backup
will be overwritten if you use the same folder to back up
the same computer again.[...]

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc742130(v=ws.10)

Note that if you are backing up to a network share then a
full backup will be run every time (because VSS is
unavailable) overwriting the previous full backup. In this
case there is no backup policy at all, you are simply
maintaining a remote shadow/copy of the system.

https://lennox-it.uk/a-complete-guide-to-wbadmin-windows-backups

Das kommt darauf an auf welches Medium du sicherst. Wenn auf
Festplatten ect. gesichert wird, dann wird immer inkrementell
gesichert. Wird auf ein Share gesichert, dann ist es immer
ein Vollbackup, wobei das letzte Backup überschrieben wird.

https://administrator.de/forum/verständnisfragen-windows-server-backup-2012-294395.html

Looking at my own tests using Windows 2008 R2, "overwritten" seems to be a bit misleading here, because it seems that really new files get created, as not only names of the images change, but their inodes as well:

root@[...]:~# ls -lisa "/volume1/[...]/WindowsImageBackup/[...]/Backup 2019-11-01 190024"
total 247604492
435         0 drwx------+ 1 [...] users         2582 Nov  2 09:12 .
430         0 drwx------+ 1 [...] users          108 Nov  1 19:58 ..
8200 214832596 -rwx------+ 1 [...] users 220521977856 Nov  1 21:33 3e4779f7-b3e1-11e0-b58f-001999a28c91.vhd
8199  32764536 -rwx------+ 1 [...] users  42484091904 Nov  1 20:12 5d3e4b2f-b386-11e0-ac00-806e6f6e6963.vhd
8214         4 -rwx------+ 1 [...] users         1078 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml
8216        12 -rwx------+ 1 [...] users        11940 Nov  1 21:34 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Components.xml
8213         8 -rwx------+ 1 [...] users         5500 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_RegistryExcludes.xml
8203         4 -rwx------+ 1 [...] users         3138 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml
8210         4 -rwx------+ 1 [...] users         1934 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer2a40fd15-dfca-4aa8-a654-1f8c654603f6.xml
8208         4 -rwx------+ 1 [...] users         3414 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml
8207         4 -rwx------+ 1 [...] users         1488 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml
8212         4 -rwx------+ 1 [...] users         1630 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer59b1f0cf-90ef-465f-9609-6ca8b2938366.xml
8202         4 -rwx------+ 1 [...] users         1628 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml
8211         4 -rwx------+ 1 [...] users          950 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writera65faa63-5ea8-4ebc-9dbd-a0c4db26912a.xml
8209         4 -rwx------+ 1 [...] users         1484 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml
8206         4 -rwx------+ 1 [...] users         3844 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml
8205         8 -rwx------+ 1 [...] users         4288 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml
8201         4 -rwx------+ 1 [...] users         1746 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml
8204      7284 -rwx------+ 1 [...] users      7455796 Nov  1 21:33 9506dbe7-82a5-4559-96b3-0a2632ae05f9_Writere8132975-6f93-4464-a53e-1050253ae220.xml
8215         4 -rwx------+ 1 [...] users         1098 Nov  1 21:33 BackupSpecs.xml

root@[...]:~# ls -lisa "/volume1/[...]/WindowsImageBackup/[...]/Backup 2019-11-02 190054" total 247603788 435 0 drwx------+ 1 [...] users 2582 Nov 2 21:51 . 430 0 drwx------+ 1 [...] users 108 Nov 2 19:59 .. 8247 4 -rwx------+ 1 [...] users 1078 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml 8249 12 -rwx------+ 1 [...] users 11940 Nov 2 21:51 0d420c40-f14c-4622-85d0-da11

6fc45608_Components.xml 8246 8 -rwx------+ 1 [...] users 5500 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_RegistryExcludes.xml 8236 4 -rwx------+ 1 [...] users 3138 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml 8242 4 -rwx------+ 1 [...] users 1934 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer2a40fd15-dfca-4aa8-a654-1f8c654603f6.xml 8239 4 -rwx------+ 1 [...] users 3414 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml 8243 4 -rwx------+ 1 [...] users 1488 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml 8241 4 -rwx------+ 1 [...] users 1630 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer59b1f0cf-90ef-465f-9609-6ca8b2938366.xml 8235 4 -rwx------+ 1 [...] users 1628 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml 8244 4 -rwx------+ 1 [...] users 950 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writera65faa63-5ea8-4ebc-9dbd-a0c4db26912a.xml 8245 4 -rwx------+ 1 [...] users 1484 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml 8240 4 -rwx------+ 1 [...] users 3844 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml 8238 8 -rwx------+ 1 [...] users 4288 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml 8234 4 -rwx------+ 1 [...] users 1746 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml 8237 7284 -rwx------+ 1 [...] users 7455796 Nov 2 21:51 0d420c40-f14c-4622-85d0-da116fc45608_Writere8132975-6f93-4464-a53e-1050253ae220.xml 8233 214832596 -rwx------+ 1 [...] users 220521977856 Nov 2 21:51 3e4779f7-b3e1-11e0-b58f-001999a28c91.vhd 8232 32763832 -rwx------+ 1 [...] users 42481994240 Nov 2 20:11 5d3e4b2f-b386-11e0-ac00-806e6f6e6963.vhd 8248 4 -rwx------+ 1 [...] users 1098 Nov 2 21:51 BackupSpecs.xml

That is different to what I see on my USB-disks, where images keep their names and file-IDs(/inodes) and only most of the XML files get new UUIDs. On the USB-disks the parent directory of the VHD(X) changes as well, but that is simply a rename and hence doesn't influence the child files in any way. At one point during the tests I managed that `wbengine` decided to keep names of VHD files, but their inode changed always. Didn't invoke with any new command line, though, but simply subsequently:

8260 32767464 -rwx------+ 1 [...] users 42481994240 Nov  3 12:47 5d3e4b2f-b386-11e0-ac00-806e6f6e6963.vhd

8266 32764416 -rwx------+ 1 [...] users 42481994240 Nov  3 13:18 5d3e4b2f-b386-11e0-ac00-806e6f6e6963.vhd

I don't know why they implemented things this way in case of using shares, as it breaks e.g. underlying BTRFS-snapshots. But that's exactly what I see: All the snapshots my NAS creates for the folder where I store those backups almost have all associated storage exclusively instead of sharing large parts of the data. Additionally, according to log file sizes and runtime lengths of `wbengine`, all backups almost take the same amount of time, even though files in the backed up source don't change too much.

So, why did MS choose that implementation of recreating image files?

↧

What does wbengine do when it says it compresses/compacts VHDX files over SMB?

November 14, 2019, 12:26 am

≫ Next: Backup failed with VSS on EFI partitions

≪ Previous: Why is "wbengine" of Windows Server 2008 R2 always creating new VHD files when targeting network shares?

I'm backing up some Windows Server 2019 using wbadmin to some Synology-NAS using SMB. Once in a while the logs created during those backups mention that one of the created VHDX files gets compressed/compacted. The server is using German language for these logs and those use a term which means "compressing" actually, so not sure which of both is really done, but I strongly guess it means "compacting".

Looking at the logs, that process is pretty fast for C:\ most likely, which only contains Windows itself and some installed applications. Most of the data is stored on D:\ instead and that data contains some databases, plain files available using network shares, a VM and stuff like that. So there's quite some amount of I/O on that volume.

Looking at the logs of the last time that was done for D:\, I recognized two interesting things: 1. that process took around an hour to succeed and 2. the logs start at 30 % and stopped at 69 %. I know that it took around an hour because without compressing/compacting the backup finishes mostly around 22:45 o'clock.

Von Volume "Daten(D:)" wird eine Sicherung erstellt. Kopiert: (100%).
Die virtuelle Festplatte für das Volume "Daten(D:)" wird komprimiert. Abgeschlossen: (30 %).
[...]
Die virtuelle Festplatte für das Volume "Daten(D:)" wird komprimiert. Abgeschlossen: (69 %).

Ende Datensicherung 13.11.2019 23:59:36,66

So what happens during that process, especially in case of storing to SMB?

From my understanding, one needs to first know which blocks of the VHDX are not used anymore. Because there's NTFS inside that image, I guess Windows simply knows that and doesn't need to scan the file in some expensive process? After knowing which blocks are not needed anymore, those can be freed to actually reduce the storage needed for the VHDX. If it would sit on NTFS itself, I guess this would be done by unallocating blocks from some sparse file or would it really write 0s to the not needed blocks? However, the VHDX is accessed using SMB with BTRFS underneath, so most likely doesn't know about any spare file. So it might move blocks from the end of the file to allocated, but free blocks and simply trims the file afterwards?

If large amounts of data really need to be copied, that would explain why the process takes pretty long and why 30 and 69 % are logged only, that being the copied range of data inside the VHDX maybe or even only copying the data itself and other operations before and after 30 and 69 % are pretty fast again.

↧

Backup failed with VSS on EFI partitions

October 22, 2017, 7:45 pm

≫ Next: Server 2012 R2 Hyper-V WSB backup fails - Event 10178, 10172, 16010 (0x8000FFFF)

≪ Previous: What does wbengine do when it says it compresses/compacts VHDX files over SMB?

Hi,

Here I have a Windows Server 2012 R2 Standard server, using Netbackup software for the backup, which is a product of Symantec company.

The server roles as file server, problem now is the backup process always failed when processing on an EFI partition.

After consulted with the product vendor, they suggested to ask MS for help..

Here is the log:

- Type: VSS_CT_FILEGROUP [2]

- Is selectable: TRUE

- Is top level: TRUE

- Notify on backup complete: FALSE

- Paths affected by this component:

- \\?\GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot<file:///\\%3f\GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot>

- Volumes affected by this component:

- \\?\GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot<file:///\\%3f\GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot> [Does not exist]

Here is the BCD info:

C:\Windows\system32>bcdedit /enum all /v > c:\bcd.txt

Firmware Boot Manager
---------------------
identifier {a5a30fa2-3d06-4e9f-b5f4-a01df9d1fcba}
displayorder {9dea862c-5cdd-4e70-acc1-f32b344d4795}
{863b7733-7876-11e5-8f71-e4e140fa16a9}
{863b7732-7876-11e5-8f71-e4e140fa16a9}
{863b7731-7876-11e5-8f71-e4e140fa16a9}
{d344c614-8c42-11e5-80b9-806e6f6e6963}
timeout 2

Windows Boot Manager
--------------------
identifier {9dea862c-5cdd-4e70-acc1-f32b344d4795}
device partition=\Device\HarddiskVolume2
path \EFI\Microsoft\Boot\bootmgfw.efi
description Windows Boot Manager
locale en-US
inherit {7ea2e1ac-2e61-4728-aaa3-896d9d0a9f0e}
bootshutdowndisabled Yes
default {863b7735-7876-11e5-8f71-e4e140fa16a9}
resumeobject {863b7734-7876-11e5-8f71-e4e140fa16a9}
displayorder {863b7735-7876-11e5-8f71-e4e140fa16a9}
toolsdisplayorder {b2721d73-1db4-4c62-bf78-c548a880142d}
timeout 30

Firmware Application (101fffff)
-------------------------------
identifier {863b7731-7876-11e5-8f71-e4e140fa16a9}
description EFI Network

Firmware Application (101fffff)
-------------------------------
identifier {863b7732-7876-11e5-8f71-e4e140fa16a9}
description EFI SCSI Device

Firmware Application (101fffff)
-------------------------------
identifier {863b7733-7876-11e5-8f71-e4e140fa16a9}
description EFI SCSI Device

Firmware Application (101fffff)
-------------------------------
identifier {d344c614-8c42-11e5-80b9-806e6f6e6963}
description EFI SCSI Device

Windows Boot Loader
-------------------
identifier {863b7735-7876-11e5-8f71-e4e140fa16a9}
device partition=C:
path \Windows\system32\winload.efi
description Windows Server 2012 R2
locale en-US
inherit {6efb52bf-1766-41db-a6b3-0ee5eff72bd7}
recoverysequence {863b7736-7876-11e5-8f71-e4e140fa16a9}
recoveryenabled Yes
isolatedcontext Yes
allowedinmemorysettings 0x15000075
osdevice partition=C:
systemroot \Windows
resumeobject {863b7734-7876-11e5-8f71-e4e140fa16a9}
nx OptOut

Windows Boot Loader
-------------------
identifier {863b7736-7876-11e5-8f71-e4e140fa16a9}
device ramdisk=[\Device\HarddiskVolume1]\Recovery\WindowsRE\Winre.wim,{863b7737-7876-11e5-8f71-e4e140fa16a9}
path \windows\system32\winload.efi
description Windows Recovery Environment
locale en-US
inherit {6efb52bf-1766-41db-a6b3-0ee5eff72bd7}
displaymessage Recovery
displaymessageoverride Recovery
osdevice ramdisk=[\Device\HarddiskVolume1]\Recovery\WindowsRE\Winre.wim,{863b7737-7876-11e5-8f71-e4e140fa16a9}
systemroot \windows
nx OptIn
bootmenupolicy Standard
winpe Yes
custom:46000010 Yes

Resume from Hibernate
---------------------
identifier {863b7734-7876-11e5-8f71-e4e140fa16a9}
device partition=C:
path \Windows\system32\winresume.efi
description Windows Resume Application
locale en-US
inherit {1afa9c49-16ab-4a5c-901b-212802da9460}
recoverysequence {863b7736-7876-11e5-8f71-e4e140fa16a9}
recoveryenabled Yes
isolatedcontext Yes
allowedinmemorysettings 0x15000075
filedevice partition=C:
filepath \hiberfil.sys
debugoptionenabled No

Windows Memory Tester
---------------------
identifier {b2721d73-1db4-4c62-bf78-c548a880142d}
device partition=\Device\HarddiskVolume2
path \EFI\Microsoft\Boot\memtest.efi
description Windows Memory Diagnostic
locale en-US
inherit {7ea2e1ac-2e61-4728-aaa3-896d9d0a9f0e}
badmemoryaccess Yes

EMS Settings
------------
identifier {0ce4991b-e6b3-4b16-b23c-5e0d9250e5d9}
bootems Yes

Debugger Settings
-----------------
identifier {4636856e-540f-4170-a130-a84776f4c654}
debugtype Serial
debugport 1
baudrate 115200

RAM Defects
-----------
identifier {5189b25c-5558-4bf2-bca4-289b11bd29e2}

Global Settings
---------------
identifier {7ea2e1ac-2e61-4728-aaa3-896d9d0a9f0e}
inherit {4636856e-540f-4170-a130-a84776f4c654}
{0ce4991b-e6b3-4b16-b23c-5e0d9250e5d9}
{5189b25c-5558-4bf2-bca4-289b11bd29e2}

Boot Loader Settings
--------------------
identifier {6efb52bf-1766-41db-a6b3-0ee5eff72bd7}
inherit {7ea2e1ac-2e61-4728-aaa3-896d9d0a9f0e}
{7ff607e0-4395-11db-b0de-0800200c9a66}

Hypervisor Settings
-------------------
identifier {7ff607e0-4395-11db-b0de-0800200c9a66}
hypervisordebugtype Serial
hypervisordebugport 1
hypervisorbaudrate 115200

Resume Loader Settings
----------------------
identifier {1afa9c49-16ab-4a5c-901b-212802da9460}
inherit {7ea2e1ac-2e61-4728-aaa3-896d9d0a9f0e}

Device options
--------------
identifier {863b7737-7876-11e5-8f71-e4e140fa16a9}
description Windows Recovery
ramdisksdidevice partition=\Device\HarddiskVolume1
ramdisksdipath \Recovery\WindowsRE\boot.sdi

Please let me know if any other information you need to know.

Really thanks for any help.

Regards

John

↧

Server 2012 R2 Hyper-V WSB backup fails - Event 10178, 10172, 16010 (0x8000FFFF)

July 19, 2017, 1:26 pm

≫ Next: Confirmation on expectations for 3rd party backup/restore?

≪ Previous: Backup failed with VSS on EFI partitions

I have one (Server 2012 R2) guest on Hyper-V in Server 2012 R2 host. System Reserved and C backup completes data transfer and all but Server(Online) fails to complete.

"Failed to complete 0KB of 0KB"

WSB Error: The component Server2012(Online) was skipped during the snapshot and will not be available for recovery. Error: The writer experienced a transient error. If the backup process is retried, the error may not occur.

Log file:

Writer Failures
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
Instance Id: {C1C2F8BB-2ACD-45C8-9B0E-D072A7B58F79}
Writer Name: Microsoft Hyper-V VSS Writer
Writer State: 5
Failure Result: 80042336
Application Result: 80004005
Application Message: (null)
Component: EBABEA20-1601-4521-855E-A286593543D0
Logical Path: (null)
Component Result: 800423F3
Component Message:
File Spec: C:\VHDX\Server2012\Virtual Machines\Server2012\Virtual Machines\\EBABEA20-1601-4521-855E-A286593543D0.xml Recursive: 0
File Spec: C:\VHDX\\C.vhdx Recursive: 0
File Spec: C:\VHDX\\data.vhdx Recursive: 0
File Spec: C:\VHDX\C-AutoRecovery.avhdx Recursive: 0
File Spec: C:\VHDX\Vorhoff2012\Virtual Machines\Server2012\Snapshots\FAE18FA3-3CE5-4E7A-AE13-691603A0B96A.xml Recursive: 0
File Spec: C:\VHDX\data-AutoRecovery.avhdx Recursive: 0
*-----------------------------*

Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
Component: EBABEA20-1601-4521-855E-A286593543D0
Caption : Online\Server2012
Logical Path:
Error : 80780175
Error Message : Component was skipped from volume shadow copy.

Detailed Error : 800423F3
Detailed Error Message : The writer experienced a transient error. If the backup process is retried,
the error may not reoccur.

Event Viewer:

Event 10178

Source: Hyper-V-VMMS

The description for Event ID 10178 from source Microsoft-Windows-Hyper-V-VMMS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Server2012
EBABEA20-1601-4521-855E-A286593543D0
%%2147943727
0x8007052F

The locale specific resource for the desired message is not present

Event 10172

Source: Hyper-V-VMMS

VSS writers inside virtual machine 'Server2012' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: Catastrophic failure (0x8000FFFF). (Virtual machine ID EBEBBA20-1601-4521-856E-A286493543B0)

Event 16010

Source: Hyper-V-VMMS

The operation failed.

VSSADMIN LIST WRITERS shows all "State: [1] Stable Last error: No error" besides:

Writer name: 'Microsoft Hyper-V VSS Writer'

Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

Writer Instance Id: {c1c2f8bb-2acd-45c8-9b0e-d072a7b58f79}

State: [1] Stable

Last error: Unexpected error

After running this script: http://www.nullskull.com/q/10115236/issue-with-windows-server-backup.aspx and turning VMMS service off and on, and then restart, backups appear to (or at least WSB says the Server2012(Online)) have completed BUT the data transferred says 0KB of 0KB! Then, the next backups say "failed" or "completed with warnings".

The question is: HOW DO I GET WSB TO WORK?

Thanks in advance, everyone! I appreciate all replies very much.

↧

Confirmation on expectations for 3rd party backup/restore?

December 18, 2019, 11:19 am

≫ Next: Data Loss Event ID 25 "The shadow copies on Volume E: were deleted because shadow copy storage could not grow in time."

≪ Previous: Server 2012 R2 Hyper-V WSB backup fails - Event 10178, 10172, 16010 (0x8000FFFF)

Can someone confirm that it is reasonable to expect that when I use a 3rd party backup solution that I should be able to recover;

File & Folder permissions, either to original or alternate location? (To a system that is a member of the domain where the permissions are from)
File & Folder metadata, eg create date and modified dates.

I have tested with the built-in Windows backup and this is working and I do know from a few years ago my previous backup solution would as well.

Am I wrong to expect in this example, say you are retiring an old file server and want ot migrate all exist data to a new file server. You should be able to backup/restore an exact copy of that data along with all custom permissions and file time stamps to a new server?

Another example would be in a disaster recovery, where you have to restore all your data to a new file server.

Your comments and clarifications are appreciated.

Stunpals - Disclaimer: This posting is provided "AS IS" with no warranties.

↧

Data Loss Event ID 25 "The shadow copies on Volume E: were deleted because shadow copy storage could not grow in time."

February 1, 2019, 4:50 pm

≫ Next: The backup failed. The RPC server is unavailable (0x800706BA)

≪ Previous: Confirmation on expectations for 3rd party backup/restore?

It appears that this has been an issue for over 10 years now, and there is yet a resolution to be had.

Once you fill up the available shadow copy storage allocated, then push a lot of disk I/O's, your snapshots will eventually be deleted and you will incur data loss.

Here is how:

When one first creates a VSS snapshot, it shows 50GB of available shadow storage space, which is great:

C:\Users\root>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Shadow Copy Storage association
   For volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\
   Shadow Copy Storage volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\Used Shadow Copy Storage space: 0 bytes (0%)Allocated Shadow Copy Storage space: 50 GB (0%)
   Maximum Shadow Copy Storage space: 38.2 TB (100%)

But as time goes by, this will fill up and then the VSS service will only try and stay ahead by about 300MB.

So what happens if you push too many I/O's, and the VSS service can't allocate space faster than it is writing? Does it re-allocate the space requested when the disk is idle? No. Does it pause the disk if it needs to allocate more shadow storage? No.

It will simply DELETE ALL of your snapshots.

Eventually, you will fill up the 50GB of shadow copy allocated storage, like so:

C:\Users\root>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Shadow Copy Storage association
   For volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\
   Shadow Copy Storage volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\Used Shadow Copy Storage space: 50 GB (0%)Allocated Shadow Copy Storage space: 50.3 GB (0%)
   Maximum Shadow Copy Storage space: 38.2 TB (100%)

And from this point on, it will barely stay ahead, except by a measly 300MB, which isn't much when you're dealing with large volumes (the one above is about 40TB).

Here's a snapshot that is just waiting to disappear any day now...

C:\Users\root>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\
   Shadow Copy Storage volume: (E:)\\?\Volume{8a5b0a31-7564-491e-a150-2fc8889ae217}\
   Used Shadow Copy Storage space: 151 GB (0%) <-- any moment my snapshot could be destroyed!Allocated Shadow Copy Storage space: 151 GB (0%)   Maximum Shadow Copy Storage space: 38.2 TB (100%)

Note that it stops showing decimals after 100GB, but I timed it as the space increases and the allocated space is still staying around 300MB ahead of the used space.

If you happen to write too many I/O's, you will get this in your event log and ALL of your snapshots will be deleted.

This is not acceptable, and I've seen it on multiple systems from Windows Server 2008 through Windows Server 2016. I need a resolution to this issue because we have customer data at stake, and we cannot move forward unless we know, for a fact, that snapshots will not be magically deleted because the hard drive decided to be slow one day, for some arbitrary reason.

I've also read the Microsoft suggestion of putting the shadow storage on another volume. That is a band aid and NOT a good solution, as that could also incur failures. If for some reason the SAN becomes unavailable, you will lose your snapshots. If for some reason the disk degrades on the second volume, you will lose your snapshots. If for some reason the second volume runs out of space, you will lose your snapshots.

It might also be worth mentioning that the registry setting MinDiffAreaFileSize was also suggested by Microsoft, but that does nothing to alleviate the issue.

Can we please get a resolve to this issue?

↧

The backup failed. The RPC server is unavailable (0x800706BA)

May 1, 2018, 7:38 pm

≫ Next: system for archiving data from Windows

≪ Previous: Data Loss Event ID 25 "The shadow copies on Volume E: were deleted because shadow copy storage could not grow in time."

After the Windows 10 version 1803 upgrade on April 30, 2018, my System Image Backup keeps failing withe two messages. "The backup failed" "The RPC server is unavailable (0x800706BA)".

Note that the System Image Backup worked before the upgrade. The upgrade completed OK and I have not found any other issues.

I checked that =1= The RPC (Remote Procedure Call) service in Services is active and running and =2= the DCOM (Distributed COM) is enabled. There used to be a SYSTEM disk when I did the defrag, the defrag does not show the SYSTEM disk in version 1803.

Is it a bug, or is there a fix or should I try the Image Recovery (that might fail as well).

↧

system for archiving data from Windows

March 4, 2020, 1:54 am

≫ Next: Windows Server 2019 backup works to second server but can not recover.

≪ Previous: The backup failed. The RPC server is unavailable (0x800706BA)

Tell me, I set up a system for archiving data from Windows to a network folder. the documentation says that there will be a complete archiving of files, Nov I see that on one server there is a complete archiving of files, and on the other server an additional archiving occurs.

↧

Windows Server 2019 backup works to second server but can not recover.

March 2, 2020, 3:59 pm

≫ Next: Windows Server 2019 Windows Admin Center can not find all the drives for the Storage Replica partnership.

≪ Previous: system for archiving data from Windows

Attempted to do backup 2nd server data section and then recovery.

In Windows Server Backup selected: Backup Once, Different Options, Custom, Add items, select Bare metal recovery, Ok, Next, Remote shared folder, Server., \\server1\folder (left default Inherit), Next

The backup was listed as completely successfully.

I tried to restore by booting from Windows Server 2019 disk. Selected Next to Windows Server 2019 window. repair your computer, Troubleshoot, System Image Recovery, Windows Server. But the only backup is one done to bare metal recovery USB drive earlier and no sign of the backup just done.

Tried Select a system image. Again only earlier backup is seen.

Tried Advanced..., Search for a system image on the network, Yes to Are you sure you want to connect to the network, Entered the \\server1\folder

Then I am getting enter credentials.

Does not look like a place to enter credentials. The script has passed by User Name: and Password.

Closing the windows gets remote procedure call failed. 0x800706BE

↧

Windows Server 2019 Windows Admin Center can not find all the drives for the Storage Replica partnership.

March 25, 2020, 7:37 pm

≫ Next: WBAdmin Incremental Backup

≪ Previous: Windows Server 2019 backup works to second server but can not recover.

Trying to use Windows Admin Center Storage Replica to back one Windows Server 2019 Standard server data volume to another Windows Standard server data volume. Windows Admin Center Storage Replica can see the data volumes but can not see the drives I want to use for the log volume. Windows Admin Center Storage Replica can only see a volume with a Windows Server 2019 OS backup or data and not any other volume.

↧

WBAdmin Incremental Backup

March 26, 2020, 7:26 am

≫ Next: Can not schedule backup to run under group managed service account

≪ Previous: Windows Server 2019 Windows Admin Center can not find all the drives for the Storage Replica partnership.

Hi Folks,

Each day I perform a backup:

WBAdmin start backup -VssFull -quiet -AllCritical -systemstate -backupTarget:\\NAS0...\d0\<date-time>

Generally, I get a full backup taking ten hours, but on occasion I get an incremental backup taking ten minutes. So, I know incremental backups are possible. I would like toforce baremetal or incremental on my schedule. I'm certain that I will always get a baremetal backup with the above instruction, if it is the first backup in "\\NAS0...\d0\<date-time>", but I would like to force any other backup to the same location to be incremental. I have tried various combinations:

-VssFull or -VssCopy
with and without -AllCritical
with and without -Include

How do I request an incremental backup?

Thanks for the help,

Chris.

↧

Can not schedule backup to run under group managed service account

April 12, 2013, 12:01 am

≫ Next: BSOD error: faulty driver on teh kernel stack

≪ Previous: WBAdmin Incremental Backup

I scheduled to run system state backup under group managed service account but am getting error "The backup operation attempted at ... has failed to start, error code '2155413599'

gmsa is member of local Administrators and Backup Operators. When I changed that scheduled task to run under local admin account, system backup succeeded.

↧

BSOD error: faulty driver on teh kernel stack

June 23, 2020, 8:57 am

≪ Previous: Can not schedule backup to run under group managed service account

Windows Server 2008 R2

Recently got a BSOD that stated there was a faulty driver in the kernel stack and after looking through drivers, found one driver that has an issue (Microsoft Tunneling Teredo Adapter).

Tried uninstalling and reinstalling the adapter but no success and still says: The device cannot start (Code 10).

Kernel-Power alert (Event ID 41) in event manager at 5:38am on 24th May 2020.

The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.

BugCheck alert (Event 1001) at 5/24/2020 5:39:38 AM

The computer has rebooted from a bugcheck. The bugcheck was: 0x0000010c (0x0000000000000015, 0x0000000000000000, 0x0000000000000000, 0xfffff880092fba88). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 052420-88421-01.

↧

Latest Images