Dec 19 2008 · Win32 Platform
So, a little story. I was working on some code that periodically spawns off a child process to do its thing, then read the results in the output stream from the parent process. Now, this periodic spawning was being done asynchronously in the parent process via. a thread. All was well and good, but there was also another thread in the parent process which created a file with a temporary name, downloaded and wrote a bunch of bytes into it, then renamed the file to a proper name. Unfortunately, I began to notice that the rename operation was failing as the parent process couldn’t get access rights to the file that needed to be renamed due to a sharing violation. I checked, double checked, and triple checked that I was closing the file, and everything looked perfect. After a bit of headbanging, I figured I should probably check to verify that another process is not holding the file handle. I then ran a wonderful utility called Handle to see exactly what processes had the file handle and, to my surprise, it was a child process that was spawned. Now, this was weird as hell as the child process did nothing with this file or directory or any file i/o whatsoever. After a bit more headbanging, I made the unpleasant discovery of child process inheritance. This forum thread, discussing a similar issue with a .NET’s TcpListener, actually pointed me to the issue.
Now most of this was in C#. One tricky aspect of this issue is that the file i/o mentioned was done in a bit of native code using fopen/fclose. I never encountered the issue with similar code using a C# FileStream. My assumption is that the file handles from the C# functions were explicitly not inheritable, while those created by fopen were. The underlying Win32 CreateFile API function does provide for this feature, but it’s not exposed via. fopen.
If this parameter is NULL, the handle returned by CreateFile cannot be inherited by any child processes the application may create and the file or device associated with the returned handle gets a default security descriptor.
CreateFile ignores the lpSecurityDescriptor member when opening an existing file or device, but continues to use the bInheritHandle member.
The bInheritHandle member of the structure specifies whether the returned handle can be inherited.
Now it was time for the really tricky part, fixing this. I didn’t want to change the native code (which in retrospect may have been an appropriate course of action and much easier to do), so instead I tried to see if I could prevent the process from inheriting the handle. The Win32 CreateProcess function has a bInheritHandles argument that can be set to false to prevent child processes from inheriting handles. Unfortunately, I was using the C# Process class and it provides no means to set such a flag. I eventually P/Invoked the CreateProcess function (this blog entry helped a great deal), but faced disappointment as I discovered that I can’t redirect standard output from the child process without bInheritHandles being set to true. I eventually altered the child process’ code to write its output to a file (which was actually better behavior for the app) and finally closed the door on this issue.