I think I have solved my problem. During the undo the event handlers fire, I check to see if the object is open for undo. If it is I just exit the event handler. Then later in the event handler method I check if the object should still have the event handlers, in my case whether the object has xdata that I want, if it does not, then it should not have the event handler and the handler is removed. I was trying to remove it while it was open for undo, but I think it had to be removed later when it wasn't open for undo