I just experimented a bit with your brd file, and I think it's a bug, but a very concealed one. And an old one too, Eagle 7.7 behaves the same. The issue arises when width and isolate are both very small. If you make one of them sufficiently large, you get a (nearly) round cutout.
For example with your brd file, the issue is still there when width=14mil and vanishes when width=15mil (and the resulting isolate value from DRC/netclass = 5mil).


So I suspect the problem is a rounding error in an intermediate calculation result in the program. Increasing the precision (float to double) or changing the sequence of math operations (multiplication before division) might already cure it.