I was recently asked to troubleshoot a crashed Windows Server 2003 Domain Controller that was also acting as single tier enterprise CA. Unfortunately, the crashed server’s computer account was already deleted from the domain and a metadata cleanup had been performed.
The crashed server was one out of three Domain Controllers within a single domain forest and the only one running Windows Server 2003. The others were running Windows Server 2008 R2.
Now, what was the problem? After a data center crash the server was not able to start. Before displaying the logon screen there was an error stating that the Directory Service could not start and that the only way to go further was to reboot into DSRM.
After the datacenter was brought back online and the network connections to the failed DC were disconnected the failed DC was deleted from the forest and a metadata cleanup was executed. Basically, this would have been a great idea for a crashed Domain Controller but the fact that the server also was a CA had been ignored. This is when I was engaged.
Integrity check with ntdsutil
As one of the first steps I checked the integrity of the Active Directory database file ntds.dit and, as expected, it was corrupted.
Going further did not help as the corrupted database file could not be repaired any more.
CA and DC on one server
Running a Certification Authority on a Domain Controller is possible but it is not the best idea since there are dependencies generated doing so. As long as everything works as expected you will not have any trouble but as soon as something goes wrong this configuration can become a real problem. One of those definitely is the fact that you are not able to start Certificate Services while Directory Services are down.
Since you can only backup a CA’s database while Certificate Services are running this is, let’s say, somewhat less than perfect. Another problem is that you are not able to demote a Domain Controller running dcpromo.exe while Certificate Services are installed on the same server. So I could neither backup the old CA’s database or certificate nor uninstall Directory Services so I could continue using the crashed server as a CA.
Fortunately, there was exactly one full backup of the VM made before the crash I could use for a restore. Unfortunately, a metadata cleanup was executed as I mentioned before so, nevertheless, I had to migrate the CA to a new server.
You can use the following steps as guidance through the migration process.
1. Backup the CA
After the restore had finished and Directory Services and Certificate Services could be started on the isolated(!) VM I backed up the CA’s database, private key and certificate via Certification Authority MMC and exported the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration with all of its subkeys. To simplify things I attached a new temporary virtual hard disk to the restored VM on which I could save the backed up files, detached it and attached it to the new VM later on.
2. Deploy a new VM and join it to the domain
I deployed a new virtual machine running Windows Server 2008 R2, gave it the same name and IP address the crashed server used before and then joined it to the domain. Since the old computer account was deleted as part of the metadata cleanup there was no problem doing so. Please note that all information about the enterprise CA had not been removed from AD configuration.
3. Import the CA certificate
Before installing AD CS role services I imported the CA certificate I had exported in step 2. Make sure to import the certificate to the new server’s personal certificate store.
4. Install Active Directory Certificate Services
After the certificate had been imported I installed the Active Directory Certificate Services (AD CS) server role. I decided to only install the Certification Authority role service and to leave the other services temporarily uninstalled. According to the old configuration I selected Enterprise CA as setup type and Root CA as CA type. On the Set Up Private Key page I selected Use existing private key and Select a certificate and use its associated private key. In the Certificates list, I selected the CA certificate I had imported before.
5. Restore the CA database and the source CA’s registry settings
I restored the CA database using the command certutil.exe -f -restoredb <CA Database Backup Directory> and imported the registry settings I backed up in step 1.
Now the CA was basically migrated to the new server using the same name and private key as it did before. But there were some issues after the migration:
1. Unable to install CA Web Enrollment role service
First of all I tried to install the Certification Authority Web Enrollment role service and got the error Active Directory Certificate Services setup failed with the following error: The parameter is incorrect. 0x80070057 (WIN32: 87).
This error occures if you have forgotten to change the SetupStatus DWORD in the imported registry backup from [hex]6003 to [hex]6001. You can find this DWORD under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration. Alternatively you can execute the command certutil -setreg config\setupstatus 0x6001 on the new server. After changing the registry the installation of the role service worked without issues.
2. CDP Location #1 Expired and/or DeltaCRL Location #1 Expired
After installing the Web Enrollment role service I got errors stating that CDP Location #1 and DeltaCRL Location #1 had expired.
This issue was solved by running certutil -CRL.
3. DeltaCRL Location #2 Unable To Download
Two errors fixed, one left:
Trying to connect to the Delta CRL’s http location led to the following IIS error:
This issue was caused by IIS not allowing URIs that do not match upon double escaping. Delta CRLs are among those because of the plus sign in the filename. So I decided to enable double escaping for the virtual directory but not for the entire web site. In my case /CertEnroll was the virtual directory hosting the Delta CRL file.
I started a command shell, changed the working directory to c:\windows\system32\inetsrv and executed the command
appcmd set config “Default Web Site/CertEnroll” -section:system.webServer/security/requestFiltering -allowDoubleEscaping:true
Please make sure to executed this command in one line.
After fixing these three issues the CA continued to work as expected and all issued certificates stayed valid. Since there was no Windows Server 2003 Domain Controller left in the domain the forest and domain functional levels could be raised to Windows Server 2008 R2. Nevertheless, you realize that there can be several issues running CAs on DCs on the one hand and migrating them on the other hand. Nowadays, when server provisioning no longer is a big deal because of virtualization technologies being in place you should always try to deploy a dedicated VM if you are going to build a PKI; even in small environments and even if you want to use a single tier PKI.
For further information about migrating a CA please have a look into Microsoft’s TechNet Library.
Azure and beyond 