Sysadmin Field Notes

May 16, 2005

What a nightmare. I just spent basically three days troubleshooting a problem under windows. Basically, we needed to talk to a service from an ASP page. Since the MSXML 4.0 control didn't work as advertised (the asych mode where you can control timeout doesn't work at all; it works with version 3.0, but that older one has a bunch of caveats that make it dicey as well), we built it in C#/.NET, and used COM interop to call it. It worked fine on our notebooks. It worked fine on the CMS content server where login as a real user is required. It even worked fine when you put "Server.CreateObject("interop.progid")" into an an aspx test file (running of course, as the ASPNET worker process).

However, in the one case where we needed it to work (under an ASP classic file), nothing. Total failure. Blew up right on the Server.CreateObject() line with:

Active Server Pages error 'ASP 0241'
Create Object Exception
The CreateObject of '(null)' caused exception C0000005.

Yes, we had already given the IUSR/IWAM users access to the windows temp directory, since .NET web service stubs use the XML Serializer to create a temporary stub class. Filemon reported the .cs file being generated, and a bunch of reads to the dll file...

Put the IUSR/IWAM users into the admin group, no help. Ran a bazillion filemon traces, no obvious access denied messages. Nothing would make this thing budge. So I tried just about every trick in my book, and made some progress in narrowing it down in fits and starts. Finally, when comparing the filemon runs from where it worked locally to the non-working ones, realized that csc.exe was never running...turned on the "leave behind temporary files" (described in the above article) and the dll was coming out zero length, even though if you ran csc.exe on the cs file with the provided .cmdline file, it built a dll just fine....

Turns out that the IUSR/IWAM (probably just IWAM, but I was too tired to pay much attention) users have to be able to access csc.exe and cvtres.exe to compile that file. I had thought of that, but I had just added an inheriting permissions rule for that to the framework directory. Well, guess what; the handy IIS Lockdown tool adds an explicit DENY to the ACL for the web anonymous users in everything under %windir%. We had a long permissions list so the deny was waaaaaay at the bottom since the group starts with a "W". And of course, denys always win. Which is why putting the IUSR's in the admin group didn't help either.

So there you go; if you have COM objects wrapping .NET code that uses the XML serializer, and you want it to work for a classic ASP page, you have to make sure that the IUSR/IWAM users can run the compiler. Obvious, but not obvious when you see a bunch of reads to csc.exe by DLLHOST.exe, and assume that maybe because of some other config switch the compiler might be running in process.

If only there was "truss" or "strace" for windows. Would have taken 30 seconds, because I would have immediately seen the failed "createProcess" or "fork", or whatever it is that Windows calls it. Lack of basic administration tools continues to hamper Windows, even at this time. Without filemon, which isn't even a Microsoft tool, this would have been impossible. Aside from perhaps installing a full debugger and hooking into the right calls that way. But without knowing where to look by narrowing it down via Filemon, this would have been an ugly process.

Anyway, just another typical windows incident where on the surface things look attractive: "Use this API, call a web service, run this little utility, call it from COM," but the reality as you move to a production environment is another thing entirely. When the abstraction breaks and doesn't "Just work" because of one of the million different things that might go wrong on each system, you're left scratching your head, without the tools to solve it easily.

All they have to do is add an officially supported truss/strace equivalent, and most of this goes away. In this insntance, I'm sure that the .NET code was generating a nice exception somewhere, but it was getting lost across the COM boundary.

This was, incidientally, the 2nd hardest time I've ever had troubleshooting something. If anyone's interested, I can write up the hardest later this week.