I started out by changing the actual code in Lucene to simply avoid the Math.Log(0.0) statement. However, I started to worry about forking the code. I don't want that kind of maintenance headache as new versions of Lucene continue to come out. I was able to work things out so that I could descend a couple of classes, override some methods, and plug these into my API layer, and everything worked fine.

The one problem left was the arithmetic overflow that I got later on in the Lucene code. I was talking with Chris Bensen, and he recalled running into something similar before and sent this link to MSDN on _controlfp documentation. I implemented the following code in .NET in the constructor of my API layer, and the overflow error went away, but of course, I end up getting NaN results due to the new FPU masking behavior. I'll have to test thoroughly to make sure that this doesn't impact calculations further down the line.

    [DllImport("msvcrt.dll")] private static extern int _controlfp(int IN_New, int IN_Mask); // this imports the call 
    private const int _MCW_EM = 0x0008001f; 
    private const int _EM_INVALID = 0x00000010; 
    
    public LuceneAPI()
    {
      // Add this code to squelch arithmetic overflow/underflow exceptions spawned from the FPU.
      // See comments in the FixedDefaultSimilarity class for more information.
      _controlfp(_MCW_EM, _EM_INVALID);
    }

I still would like to find out the technical details for why I need to do any of this, but at least I have things running.