I have the same issue along with similar behavior by text. I work around this by creating a model space viewport for editing on its own tab that will display the entire project at the desired scale and locking it. This keeps all text and points at the correct size, maintains the correct displayed ltscale for dashed lines and allows me to manipulate layers without affecting the other viewports.