Configuring the BizTalk Group, System.EnterpriseServices.TransactionProxyException, and MSDTC
While standing up a multi-server environment for a client, the BizTalk Group configuration step was failing with a System.EnterpriseServices.TransactionProxyException. A quick search online indicated this was a MSDTC related issue and is documented in the Knowledge Base. I double checked that the LocalDTC was configured correctly on both servers (it was), there were no firewall issues (firewall turned off on both machines), there was no clustering to contend with, and DTCPing was successful. Finally I tried DTCTester and got an error – however, none of the usual suspects for this (mentioned in the KnowledgeBase article) were the issue. Note: when running DTCTester on a 64 bit OS, you have to add a DSN using the SQL Client pointing to the machine you want to connect to using the 32 bit ODBC Data Source Administrator, in C:\Windows\SysWOW64\odbcad32.exe.
Reading through the MSDN article on troubleshooting MSDTC, there were two sections that were helpful (but still slightly confusing). In the Registry, the GUIDs under HKCR\CID were the same (the servers were generated from the same image). This was true even through DTCPing succeeded without warning, despite the suggestion of the troubleshooting article.
from an Adminstrator Command Prompt resolved that. I then had to reset the services to start automatically and start them, and double check that configuration was still in place correctly. DTCTester still didn’t work from a remote machine (but did work locally), so I went ahead and added the RPC key to HKLM\SOFTWARE\Policies\Microsoft\Windows NT, with the suggested values of 1 for EnableAuthEpResolution and 0 for RestrictRemoteClients. Restarting services was not enough to things going, but a reboot of the machine worked and we were back in business.
The bottom line is, if DTC is not working even though it seems like it should be from the network/configuration end, it could be that your infrastructure team imaged a server with the same GUIDs used for DTC!