Monday, September 19, 2016

Makeshift Webserver with NCat

Sometimes I'll be working on a project, and I'll find myself in need to share a single file over http. For instance, I may need to test that a program or script is correctly grabbing files over http with Invoke-WebRequest, or curl. I may need to quickly have a file upload available, and the only reliable way is via an http server (polycom certificates for example)

I don't want to setup IIS, WAMP, or anything permanent, I just need a quick webserver on a single port temporarily.

For this, I will use ncat.

To continue on with the use cases below, head over to https://nmap.org/ncat/ and download nmap. For this example, place the ncat.exe in the Directory you are going to be running the script from, as well as any files you are wishing to serve up.

Image File
$mime =  "image/jpeg"
.\ncat -lk -p 8080 --sh-exec "echo HTTP/1.1 200 OK\r\nContent-Type: $mime\r\n& echo(&type jpeg-home.jpg"
Assumptions
jpeg-home.jpg is in current working directory.
ncat.exe is in the current working directory
that this is a jpeg file. If it is a gif or png, you will need to adjust the mime type

HTML File
I've found that this can be done without the mime type. Perhaps this is implied if not specified.
.\ncat -lk -p 8080 --sh-exec "echo HTTP/1.1 200 OK\r\n& echo(&type index.html"

SSL Certificate
$mime = "application/pkix-cert" .\ncat -lk -p 8080 --sh-exec "echo HTTP/1.1 200 OK\r\nContent-Type: $mime\r\n& echo(&type lyncpool.crt"


Other File Types

You will need to change $mime to match your file type. The list I have used to find the appropriate mime type is available here: https://www.sitepoint.com/web-foundations/mime-types-complete-list/

Friday, September 2, 2016

Group Policy and Unintended Consequences

As a sysadmin, one pet peeve of mine would be the inconsistency of group policy. Often, I find the description of a policy doesn't match the observed behavior when used.

Take for instance the following Windows Store policy.

From the description, I felt safe assuming that this would only disable the offer from appearing and pestering the end user. 

The unintended side effect was that the update from Windows 10 version 1511 to version 1607 was registering as unneeded in SCCM.

Over the years I've seen dozens of such inconsistencies. As I discover them, I will change the policies to reflect the new understanding, and add a comment as to why, perhaps to warn my fellow colleagues who are looking at something after me, and often just to warn myself. It may have been years since I have set the policy, and there are plenty of rather obscure places to venture within Group Policy. 

It sure would be nice if Microsoft would further standardize the wording of Group Policy. The double negatives can be mind boggling. 
I'm sure I can't be the only one who has thought at one time or another, "Wait, So do I disable the enable this policy, or if I enable the policy and set the option to disabled, or... umm... what?"

Thursday, August 25, 2016

Taming the Chrome - Implementing Google Chrome in XenDesktop 5.6

If you are like me, you have probably noticed that Google Chrome doesn't want to play nicely inside of your VDI. In the recent years, Chrome has come a long way towards being more enterprise friendly, but it isn't quite there yet.

We are going to start this journey with the low hanging fruit. First off, you are going to want to grab a copy of the following.
  1. Chrome for Work MSI - https://www.google.com/work/chrome/chrome-browser/
  2. Chrome ADMX Templates -  https://www.google.com/work/chrome/chrome-browser/
  3. Google Updater ADMX templates  -https://support.google.com/chrome/a/answer/6350036#Obtaining_the_Administrative_Tem

A Word on Direct Write
I've noticed many complaints on the Citrix subreddit, as well as various comments about Direct Write causing issues under certain conditions. The text will appear blurry, or pixelated. I've seen this happen when using chrome over RDP or ICA.  To mitigate this problem, the --disable-direct-write seems to help.

    Version 52 and above of the Chrome for Work MSI have removed the --disable-direct-write flag.
https://productforums.google.com/forum/#!topic/chrome/Q6QGBRvULbQ Unfortunately, google does not publish old versions of the binary. I have a working copy of an older version, if somebody needs it, email me at devon.dieffenbach@gmail.com


Installing the ADMX Templates on the Golden Image
For the ADMX settings for Google Updater and Google Chrome, I have chosen to make these particular changes to the Local Group policy of the Golden Image. Extract your ADMX templates to %systemroot%\PolicyDefinitions.
Chrome.admx goes into %systemroot%\PolicyDefinitions
chrome.adml (from en-us subfolder) goes into %systemroot%\PolicyDefinitions\en-US
Do the same for GoogleUpdate.admx and GoogleUpdate.adml

 When Dealing with VDI, I have noticed that ADM templates have some issues with loading, whereas ADMX do not.
 

Configuring Local Group policy
launch gpedit.msc from the Golden image.
Chrome settings are located under Administrative Templates -> Google -> Google Chrome
Update settings are located under Administrative Templates -> Google -> Google Update

Prevent Google Update
Because --disable-direct-write is missing in later versions of Chrome, I have found it necessary to disable Google Update. If you choose not to do this, you run the risk of Chrome updating to the latest version and making the text look ugly for users of that image.
The following settings are necessary to disable automatic update:
Administrative Templates -> Google -> Google Update -> Applications
  1. Allow Installation Default: Enabled
  2. Update Policy Override Default: Enabled | Policy (Manual Updates Only)


Install / Configure Chrome Launcher
In order to ensure a consistent user experience, there are various chrome flags that need to be set. For our environment, I have narrowed this down to the following: --disable-print-preview --disable-popup-blocking --disable-smooth-scrolling --allow-no-sandbox-job --disable-gpu --disable-direct-write

Unfortunately, I have not found a reliable way to set these preferences for all users. I started out by removing shortcuts for all users and replacing them with a shortcut to a bat file with the chrome Icon.

    Chrome seems to have some sort of background daemon that dictates the installation of its shortcuts. Chrome shortcuts do not behave in the standard way. In my case, I have tried deleting them, but they will often show back up. This is just another one of those bizarre Chrome specific quirks that seem specifically designed to annoy sysadmins.


In order to address the issue of not being able to control startup switches, and because chrome's shortcut system was difficult to predict, I figured the best way to ensure my flags were passed, was to replace the chrome.exe with a wrapper, and have that wrapper call a renamed original chrome.exe.

I wrote a quick and dirty solution in C# and have posted the solution (and binaries) to github.
Binary Download
Visual Studio Solution

Once you have a copy of ChromeLaunch.exe, you will need to navigate to the program folder where Chrome is installed on your golden image. Rename chrome.exe to gchrome.exe. Place chromelaunch.exe into the directory and name it gchrome.exe

Current caveats:
  • I've not yet implemented argument passing. If something launches the chrome browser and passes other arguments, they will be discarded. You can fix this yourself in the solution, or wait for me to eventually get around to doing it.
  • I'm not sure if having chrome.exe renamed to gchrome.exe will break other applications. I have not seen it so far, but hey… it's not like chrome.exe was working anyways? Right?

    Out of personal preference, I made the binary with the hidden golden icon. I extracted the icon from the original chrome.exe using Nirsoft's IconsExtract  http://www.nirsoft.net/utils/iconsext.html. It looked cool, and I wanted to differentiate it between the rest of the Applications that are auto populated in the menu from Citrix Receiver. 

Thursday, August 11, 2016

XenApp - Print Spooler Failed to Install Driver - Part 2


Last week, we looked at some sporadic issues with Printing in XenApp. We managed to identify what the issue, and have found some potential causes. Now we are going to dive into this a bit more, and attempt to find a solution

In order to do this, we are going to need to gather more data. Since I  discovered this issue after it was escalated to me, I soon found out this has been a long standing problem. After some quick digging through event logs, I was able to determine that this issue is occurring frequently, but I have been unable to determine a pattern.

I patched together this quick script to scrape all of the servers in the XenApp farm daily, and place that information in an xml file in my reporting directory.

$filter = @{}
$filter.set_item("ID",(215,319))
$filter.add("LogName", "Microsoft-Windows-PrintService/Admin")
$date = (Get-Date).AddDays(-1)
$filter.Add("StartTime", $date)

$computers = Get-XAServer -OnlineOnly -ZoneName "Default Zone" | select -ExpandProperty ServerName
$list = @{}
foreach($computer in $computers){
    Write-Host -Foreground Yellow "Gathering Details from $computer"
    $obj= Get-WinEvent -ComputerName $computer -FilterHashtable $filter
    $list.Add($computer, $obj)

}

#Export The log

$ReportDirectory = ($env:userprofile) +"\Reports"
$day = (Get-Date).Day
$month = (Get-Date).Month
$hour = (Get-Date).Hour
$minute = (Get-Date).Minute
$filename = "print`-$month`-$day`-$hour`-$minute.xml"
$fullExportLocation = "$ReportDirectory\Print215\$filename"
Write-Host -ForegroundColor Cyan "Exporting object to $fullExportLocation"
$list | Export-Clixml -Path $fullExportLocation



I've created a scheduled task to run the above script on my PC. This assumes that you have the proper XenApp powershell cmdlets installed and working. To Verify your XenApp cmdlets are correctly setup, run Get-XAServer in powershell. If you see a list of XenApp servers, you are good to go.
It's best to set the task to run before the images refresh themselves. Our images are on a nightly refresh, so I set my script to run at 9pm.

You can examine the events at a later point in time by using the Import-CLIXml Cmdlet.

Friday, August 5, 2016

XenApp - print spooler failed to import the printer driver



Problem 
We have a XenApp deployment here. Occasionally, a print driver fails to load on one of the application servers. Since we have dozens of these that spin up from the golden image every night, the problem will resurface eventually. 

In our environment, we are using 2 print servers, with point and print enabled. We push out printer connections via group policy. Since this is a high traffic Citrix environment, we have multiple users that are connected. The server they are connected to is randomly determined. 

If we reboot an affected server, it will usually work fine. You can connect to the shared printer and move along. This is not a good solution when rebooting the server will disrupt several users, however. I wanted to find the real root cause of the issue. The issue is sporadic, and seemingly random.

Problem Part 1 - No default Printer
When Windows has no default printer, certain things don't work. 
I'll give you an example you can see for yourself.
Lets delete the registry key for the current printer in powershell:
$Device = (Get-ItemProperty 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\Windows').Device

Set-ItemProperty 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\Windows' -Name Device -value $null

Check Devices and printers, verify that there is no default printer selected at all.
If you did this correctly, you can open up Internet Explorer, Navigate to any website, and click Print Preview.
What you should see is, well... nothing.. 
And that's the problem. Essentially, this is an unhandled exception within windows. 
Now, for some reason, this won't affect versions of IE lower than 10. I think IE9 and below use a different print management system.
This also typically will not affect other applications like Google Chrome, which seem to work fine without a default printer. 

Problem Part 2: Driver Installation Failed, Driver Store Missing inf
The next part of the problem is the killer. Once a driver fails to install, any printer utilizing that driver will not load. 
Key Observations
Event 600: Client Side Rendering
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:04:53 AM
Event ID:      600
Task Category: Client-side rendering
Level:         Error
Keywords:      Client Side Rendering (CSR),Classic Spooler Event
User:          SYSTEM
Computer:      ctximg6-0001.contoso.com
Description:
The print spooler failed to import the printer driver that was downloaded from \\Print2\print$\x64\PCC\oemsetup.inf_amd64_48ea2774648628eb.cab into the driver store for driver RICOH MP 2554 PCL 6. Error code= 80070020. This can occur if there is a problem with the driver or the digital signature of the driver.
Event Xml:
    Microsoft-Windows-PrintService/Admin
    ctximg6-0001.contoso.com

      \\Print2\print$\x64\PCC\oemsetup.inf_amd64_48ea2774648628eb.cab
      RICOH MP 2554 PCL 6
      80070020

Event 601: Client Side Rendering
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:04:53 AM
Event ID:      601
Task Category: Client-side rendering
Level:         Error
Keywords:      Client Side Rendering (CSR),Classic Spooler Event
User:          Contoso\User1
Computer:      ctximg6-0001.contoso.com
Description:
The print spooler failed to download and import the printer driver from \\Print2 into the driver store for driver RICOH MP 4054 PCL 6. Error code= 80070003.
Event Xml:

    Microsoft-Windows-PrintService/Admin
    ctximg6-0001.contoso.com

      \\Print2
      RICOH MP 4054 PCL 6
      80070003
  
Event 215: Installing a Printer Driver (Hundreds of these)
Log Name:      Microsoft-Windows-PrintService/Admin
Source:        Microsoft-Windows-PrintService
Date:          8/5/2016 8:13:41 AM
Event ID:      215
Task Category: Installing a printer driver
Level:         Error
Keywords:      Printer Setup,Printer
User:          SYSTEM
Computer:      ctximg6-0001.contoso.com
Description:
Installing printer driver RICOH MP 4054 PCL 6 failed, error code 0x3, HRESULT 0x80070057. See the event user data for context information.
Event Xml:
    Microsoft-Windows-PrintService/Admin
    ctximg6-0001.contoso.com
   
      IsInfInDriverStore failed
      -
      C:\Windows\System32\DriverStore\FileRepository\oemsetup.inf_amd64_neutral_48ea2774648628eb\oemsetup.inf
      RICOH MP 4054 PCL 6
      -
      Windows x64
      Not package aware
      -
      0x3
      0x80070057

Testing: Is there anything at the INF Path?
On CTXIMG6-0001 , open C:\Windows\System32\DriverStore\FileRepository\oemsetup.inf_amd64_neutral_48ea2774648628eb\

Okay, so we know the folder isn't there, so it never made it to the repo.

What about the Cab file? Can this be accessed from the CTXIMG and dropped in manually?

The Cab File opens fine, so the computer should have had access

Adding the Directory Manually
I added the directory to C:\Windows\System32\DriverStore\FileRepository\oemsetup.inf_amd64_neutral_48ea2774648628eb, the location Point and Print was looking for the driver. Also, Note that this is different from the cab file name on the print server. This indicates to me that Point and Print keeps track of the folders, and names them according to it's own convention.
It took some fiddling with psexec and Nt Authority\System in order to drop the file in the correct location, as the DriverStore is locked down.

Once the file was added, I popped open a Run Dialog, and keyed in the path to the printer, and it worked!

So what Happened?
I'm not entirely sure yet, but it could be one of the following.
Theory 1:
Point and Print goes to download the Cab file from the print server, but the download fails.
Point and Print does not update its records to show that this failed, and thus expects that the inf file is there, as seen in Event 215.

Theory 2: 
2 or more users try to download the printer driver at the same time, causing the Point and Print system to mess up, and enter an unhealthy state. They both think the file is downloaded, but it isn't.

What can be done to fix this:
I am hoping that it is possible to preload drivers onto the Golden Image. I'm thinking if I created a driver store on the Image, and set the Device Path in the registry to point to this as well, the Plug and Play/Point and Print (side note, does anybody know if Plug and Play and Point and Print are the same thing?). It would be fairly easy to navigate out to the print servers and copy all the printers from their share. My concern is understanding the format that PNP is looking for. Is it expecting Cab Files? Is it looking for the files to have the exact same names as it would have in the driver repository after a PNP download? Or, is it more lenient?

I'll be testing this over the next few weeks. Stay tuned for an update on what I have learned.