Troubleshooting IIS Shared Configuration

Bad Data. (Exception from HRESULT: 0x80090005)

If your using a Shared Configuration in IIS7 or IIS8 this error message is one of the biggest challenges with the solution. As I mentioned on the Top Ten IIS Configuration Tips article, IIS Shared Configuration is a great solution for creating and maintaining web farms. But the are a handful of errors that can be very challenging to resolve. And if your searching for this or other similar errors you might be in a full panic right now. The good news is I’ve always been able to recover from these errors, it just might take a little back tracking.

What does it mean?

Most of these errors boil down to one core problem, IIS doesn’t have the right encryption key to decrypt or encrypt data in the applicatonHost.config. Usually, the Application Pool Identity. So you may see this when trying to set a custom identity on an app pool or set a connect as account for a virtual directory on a file share.

IIS MachineKeys

Before digging into errors, its important to understand the MachineKeys that IIS is using. RSA MachineKeys are the Private Keys that IIS uses to encrypt and decrypt sensitive data (passwords). In Windows 2008 and Windows 2012 MachineKeys are stored in C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys. The IIS related keys are listed below. Each Key file ends with “_MachineGUID”. To confirm the MachineGUID, look in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography

• IIS WAS Key: 76944fb33636aeddb9590521c2e8815a
• IIS Configuration Key: 6de9cb26d2b98c01ec4e9e8b34824aa2
• IIS6 Compatibility Key: c2319c42033a5ca7f44e731bfd3fa2b5

Exporting, Importing, and Comparing

The Key to resolving most of the Shared Configuration problems is exporting keys from your master server, importing into your node servers, and comparing them later.

Use the following two commands to export the IIS WAS and Configuration Keys from your Master server:

& $env:windir\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -px 'iisConfigurationKey' '\\server\config\iisConfigurationKey.xml' -pri
& $env:windir\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -px 'iisWasKey' '\\server\config\iisWasKey.xml' -pri

On your Node servers, use the following two commands to import the IIS WAS and Configuration Keys. Note the -exp param lets you export the keys from this server at a later date:

& $env:windir\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -pi 'iisConfigurationKey' '\\server\config\iisConfigurationKey.xml' -exp
& $env:windir\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -pi 'iisWasKey' '\\server\config\iisWasKey.xml' -exp

To easily perform comparisons of a server’s keys versus an exported key, use the following script. This will tell us if this server’s keys match the master exported keys.

param (
   [Parameter(Mandatory=$true)]
   [String]$remotekeyPath,
   [Parameter(Mandatory=$true)]
   [String]$KeyName
)

$myConfigkey = Join-Path $env:temp '$KeyName.xml'

Write-Host ''
& $env:windir\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -px $KeyName $myConfigkey -pri
Write-Host ''

$key1 = Get-Content $remotekeyPath
$key2 = Get-Content $myConfigkey

if (Compare-Object $key1 $key2)
{
	Write-Host '$KeyName files DO NOT match!!'
}else
{
	Write-Host '$KeyName files match.'
}

Alternative Method

If the above method is not working here’s an alternative method to fix a Shared Configuration. Note, this is a bit more invasive and I would use it as a last resort because there’s a risk you could damage other applications on the server.

1. Stop IIS
2. Backup contents of C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys
3. Delete all files under C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys except the oldest file.
4. Ensure IIS_IUSRS group has Read/Browse access on C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys, confirm permissions has propogated to key files correctly.
5. Restart IIS
6. Reenable Shared Configuration

Troubleshooting IIS Shared Configuration

Bad Data. (Exception from HRESULT: 0x80090005) This is error message shows up when trying to save a password via the UI to a Shared Config and that server’s MachineKeys do not match.

Steps to Resolve:

1. Disable Shared Configuration, do not continue to use the current configuration files.
2. Use the Export commands above to export the Machine Keys from the server that exported applicationHost.config.
3. Use the Import commands above to import the exported Machine Keys. Be sure to include the -exp parameter.
4. Re-enable Shared Configuration

A few alternative solutions can also be tried if the first solution doesn’t work. You can switch your Application Pool Identity to ApplicationPoolIdentity or Network Service. These changes will save regardless of MachineKey state. You may also want to try restoring from IIS Backups. As a final last resort, reinstall IIS from the server. This will generate new MachineKeys.

 

Key not valid for use in specified state. (Exception from HRESULT: 0x8009000B): When you try to save passwords in IIS Manager you could receive this error if you have problems with your MachineKeys. The most common cause of this error is that you copied/imported IISWASKey and IISConfigurationKeys from a machine that had previously had the keys imported without the -exp flag. Basically, the keys you imported were not valid to be exported. To resolve, restore your previous keys on this machine from a backup, Export/Import Keys from another server in the farm, or reinstall IIS to create new keys.

 

Keyset does not exist (Exception from HRESULT: 0x80090016): When you try to save passwords in IIS Manager you could receive this error if your missing the IISWASKey and IISConfigurationKeys. Check the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys directory and see if they exist. To resolve, restore your previous keys on this machine from a backup, Export/Import Keys from another server in the farm, or reinstall IIS to create new keys.

 

Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008) This error is similar to the errors above. It usually happens when you’re trying to save encrypted values into applicationHost.config using the UI. It generally means that the UI is unable to encrypt the value correctly, and has nothing to due with storage. I’ve found it happens due to the following conditions:

1. IIS has Shared Configuration enabled.
2. The Shared Configuration location is on a remote UNC. (I think its possible this could happen using a local path, I just haven’t seen it)
3. The MachineKeys on this server do not match the server that originally exported the applicationHost.config

This usually happens when you’re adding a new server to a web farm. To resolve this error, Export Machine Keys from the server that originally exported the applicationHost.config. Import the Keys into this server. Disable Shared Configuration, and Enable Shared Configuration. Try to save a password to test.

 

Failed to decrypt attribute ‘password’. Event ID 5172, 5036, 5005, 7001, 7023. This error means that IIS could not decrypt the password value in Redirection.config to be able to read applicationHost.config. This usually happens if you enabled Shared Configuration and then later changed your Machine Keys. The new MachineKeys fail to decrypt value. To resolve, re-enable Shared Configuration to regenerate the password with the current Machine Keys.

 

Can not log on locally to \\server\share as user domain\user with redirection.config password. You’ll see this error in the System Event log, and when opening IIS Manager you’ll see a message saying There was an error when trying to connect. Both of these errors mean that IIS was able to decrypt the UNC Password Value, but failed to log into the share to load applicationHost.config. As a result the applicationHost.config is unavailable. This usually happens when you’re configuring Shared Configuration with a script since the UI will validate the password.

To resolve, check the password you’re using in your script. One way to troubleshoot is to open redirection.config and enter the password in plain text. Once you have a valid password in place IIS will start correctly.

 

Failed to decrypt attribute ‘password’ because the keyset does not exist.

Log Name: Application
Source: IIS-W3SVC-WP
Event ID: 2307

You’ll see the this error in your Application Log

The worker process for application pool 'pool-name' encountered an error 'Failed to decrypt attribute 'password' because the keyset does not exist' trying to read configuration data from file

This usually means there’s a problem using the MachineKeys to decrypt encrypted data in the config. This can be caused by file permissions on the MachineKeys directory. To resolve, ensure IIS_IUSRS group has Read/Browse access on C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys, confirm permissions has propagated to key files correctly.