Thursday, August 11, 2016

XenApp - Print Spooler Failed to Install Driver - Part 2

Last week, we looked at some sporadic issues with Printing in XenApp. We managed to identify what the issue, and have found some potential causes. Now we are going to dive into this a bit more, and attempt to find a solution

In order to do this, we are going to need to gather more data. Since I  discovered this issue after it was escalated to me, I soon found out this has been a long standing problem. After some quick digging through event logs, I was able to determine that this issue is occurring frequently, but I have been unable to determine a pattern.

I patched together this quick script to scrape all of the servers in the XenApp farm daily, and place that information in an xml file in my reporting directory.

$filter = @{}
$filter.add("LogName", "Microsoft-Windows-PrintService/Admin")
$date = (Get-Date).AddDays(-1)
$filter.Add("StartTime", $date)

$computers = Get-XAServer -OnlineOnly -ZoneName "Default Zone" | select -ExpandProperty ServerName
$list = @{}
foreach($computer in $computers){
    Write-Host -Foreground Yellow "Gathering Details from $computer"
    $obj= Get-WinEvent -ComputerName $computer -FilterHashtable $filter
    $list.Add($computer, $obj)


#Export The log

$ReportDirectory = ($env:userprofile) +"\Reports"
$day = (Get-Date).Day
$month = (Get-Date).Month
$hour = (Get-Date).Hour
$minute = (Get-Date).Minute
$filename = "print`-$month`-$day`-$hour`-$minute.xml"
$fullExportLocation = "$ReportDirectory\Print215\$filename"
Write-Host -ForegroundColor Cyan "Exporting object to $fullExportLocation"
$list | Export-Clixml -Path $fullExportLocation

I've created a scheduled task to run the above script on my PC. This assumes that you have the proper XenApp powershell cmdlets installed and working. To Verify your XenApp cmdlets are correctly setup, run Get-XAServer in powershell. If you see a list of XenApp servers, you are good to go.
It's best to set the task to run before the images refresh themselves. Our images are on a nightly refresh, so I set my script to run at 9pm.

You can examine the events at a later point in time by using the Import-CLIXml Cmdlet.

Friday, August 5, 2016

XenApp - print spooler failed to import the printer driver

We have a XenApp deployment here. Occasionally, a print driver fails to load on one of the application servers. Since we have dozens of these that spin up from the golden image every night, the problem will resurface eventually. 

In our environment, we are using 2 print servers, with point and print enabled. We push out printer connections via group policy. Since this is a high traffic Citrix environment, we have multiple users that are connected. The server they are connected to is randomly determined. 

If we reboot an affected server, it will usually work fine. You can connect to the shared printer and move along. This is not a good solution when rebooting the server will disrupt several users, however. I wanted to find the real root cause of the issue. The issue is sporadic, and seemingly random.

Problem Part 1 - No default Printer
When Windows has no default printer, certain things don't work. 
I'll give you an example you can see for yourself.
Lets delete the registry key for the current printer in powershell:
$Device = (Get-ItemProperty 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\Windows').Device

Set-ItemProperty 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\Windows' -Name Device -value $null

Check Devices and printers, verify that there is no default printer selected at all.
If you did this correctly, you can open up Internet Explorer, Navigate to any website, and click Print Preview.
What you should see is, well... nothing.. 
And that's the problem. Essentially, this is an unhandled exception within windows. 
Now, for some reason, this won't affect versions of IE lower than 10. I think IE9 and below use a different print management system.
This also typically will not affect other applications like Google Chrome, which seem to work fine without a default printer. 

Problem Part 2: Driver Installation Failed, Driver Store Missing inf
The next part of the problem is the killer. Once a driver fails to install, any printer utilizing that driver will not load. 
Key Observations
Event 600: Client Side Rendering
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:04:53 AM
Event ID:      600
Task Category: Client-side rendering
Level:         Error
Keywords:      Client Side Rendering (CSR),Classic Spooler Event
User:          SYSTEM
The print spooler failed to import the printer driver that was downloaded from \\Print2\print$\x64\PCC\ into the driver store for driver RICOH MP 2554 PCL 6. Error code= 80070020. This can occur if there is a problem with the driver or the digital signature of the driver.
Event Xml:

      RICOH MP 2554 PCL 6

Event 601: Client Side Rendering
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:04:53 AM
Event ID:      601
Task Category: Client-side rendering
Level:         Error
Keywords:      Client Side Rendering (CSR),Classic Spooler Event
User:          Contoso\User1
The print spooler failed to download and import the printer driver from \\Print2 into the driver store for driver RICOH MP 4054 PCL 6. Error code= 80070003.
Event Xml:


      RICOH MP 4054 PCL 6
Event 215: Installing a Printer Driver (Hundreds of these)
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:13:41 AM
Event ID:      215
Task Category: Installing a printer driver
Level:         Error
Keywords:      Printer Setup,Printer
User:          SYSTEM
Installing printer driver RICOH MP 4054 PCL 6 failed, error code 0x3, HRESULT 0x80070057. See the event user data for context information.
Event Xml:
      IsInfInDriverStore failed
      RICOH MP 4054 PCL 6
      Windows x64
      Not package aware

Testing: Is there anything at the INF Path?
On CTXIMG6-0001 , open C:\Windows\System32\DriverStore\FileRepository\oemsetup.inf_amd64_neutral_48ea2774648628eb\

Okay, so we know the folder isn't there, so it never made it to the repo.

What about the Cab file? Can this be accessed from the CTXIMG and dropped in manually?

The Cab File opens fine, so the computer should have had access

Adding the Directory Manually
I added the directory to C:\Windows\System32\DriverStore\FileRepository\oemsetup.inf_amd64_neutral_48ea2774648628eb, the location Point and Print was looking for the driver. Also, Note that this is different from the cab file name on the print server. This indicates to me that Point and Print keeps track of the folders, and names them according to it's own convention.
It took some fiddling with psexec and Nt Authority\System in order to drop the file in the correct location, as the DriverStore is locked down.

Once the file was added, I popped open a Run Dialog, and keyed in the path to the printer, and it worked!

So what Happened?
I'm not entirely sure yet, but it could be one of the following.
Theory 1:
Point and Print goes to download the Cab file from the print server, but the download fails.
Point and Print does not update its records to show that this failed, and thus expects that the inf file is there, as seen in Event 215.

Theory 2: 
2 or more users try to download the printer driver at the same time, causing the Point and Print system to mess up, and enter an unhealthy state. They both think the file is downloaded, but it isn't.

What can be done to fix this:
I am hoping that it is possible to preload drivers onto the Golden Image. I'm thinking if I created a driver store on the Image, and set the Device Path in the registry to point to this as well, the Plug and Play/Point and Print (side note, does anybody know if Plug and Play and Point and Print are the same thing?). It would be fairly easy to navigate out to the print servers and copy all the printers from their share. My concern is understanding the format that PNP is looking for. Is it expecting Cab Files? Is it looking for the files to have the exact same names as it would have in the driver repository after a PNP download? Or, is it more lenient?

I'll be testing this over the next few weeks. Stay tuned for an update on what I have learned.