Single Blog

Situation:

We had a number of production VMs running a Citrix desktop workload on Citrix Xenapp 6.5 via Citrix PVS 7.1 (SP3) with cache to RAM with overflow to hard drive, set at 2048Mb. VMware Hypervisor 5.5 and Appv 5 SP2 with HRP4 running in full infrastructure mode (no Microsoft SCCM) and 30GB of packages running in Shared content store mode from a network file share \\server\appvshare$

Servers have 12GB Ram, with 4 vCPU’s and Windows 2008 R2 SP1

C: Drive is 40GB (with 35GB utilised)

D: Drive is 20 GB (typically 12GB utilised at any one time) with a 6.1GB Page file redirected here, an the vdiskdif.vhdx at 4mb

We have a daily reboot for the entire Citrix desktop estate estate at 3am.

Appv GPO Settings are to use Shared content mode, use the local path of c:\programdata\appv (not app-v), autoload = ALL, user and computer refresh on logon = true

Problem/Symptoms:

After an IE11 upgrade on the Citrix vDisk, every 2-3 days some of the citrix servers would reboot (as per policy) and quickly fill up the D drive, finally crashing the server and rendering it unusable. Hoorah for PVS cache to disk.

As the machine had crashed there was little way to troubleshoot it after the fact, so we needed to capture it during.

Steps to resolve:

Increase C: Drive to 60GB using the process listed in the first section of this article – NO FIX

Increase D: Drive to 40GB to resolve. The server stopped crashing as the server didnt run out of D drive space, we then noticed that the vdiskdif.vhds was sitting at 13.6GB and growing (vs unaffected servers sitting at 4Mb) When checking the system processes, nothing was obviously running or processing. This was a nice workaround but not a final solution or fix.

We rebooted and remotely monitored the system with PSTools (sysinternals) executable “pslist -s \\servername”. During the reboot the appvclient.exe was siting 25% CPU, whilst the GPO to do a full client sync was ‘called’ and run (powershell: get-appvpublishingserver | sync-appvpublishingserver)

Our crashing was finally narrowed to the AppV client that was sporadically somehow crashing and filling the D drive. When the AppV client was stopped or the server removed from any ‘global’ publishing – the server never crashed. As soon as the production APPV packages were republished to the device, and after every 2 or 3 reboots the appvclient.exe would run, start caching files, and start filling up the D drive as it was ‘copying / caching’ the packages.

This felt very much like the AppV client was ignoring the shared content mode, and trying to cache everything locally.

Final Resolution:

The ‘gut feeling’ on this problem was that the AppV client wasnt getting the correct settings from GPO in time. The local AppV Client settings (registry) were reviewed and it seemed there were some improvements to be made.

OLD SETTING

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Streaming]

“AutoLoad”=dword:00000001

“SharedContentStoreMode”=dword:00000000

Turns out the Appv Client settings were set to disable Shared content mode, and to autoload all apps which we are assuming its doing sometimes before it gets its required settings from GPO.

NEW SETTING

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Streaming]

“AutoLoad”=dword:00000000

“SharedContentStoreMode”=dword:00000001

After creating a new vDisk version with the ‘New Setting’ registry keys listed above tattooed in the vDisk image  the servers haven’t crashed since.

Also we were using a GPO powershell script that tells the client to go and ‘autoload’ all the apps. So the autoload registry option was just confusing things, hence we disabled it.