If you are working through your viewport from model space, I would zoom your viewport to the desired scale, and then lock the viewport. This will allow you to zoom in and out as much as like, without changing the scale of your viewport, thereby leaving your annotation the same size.
Andy Reese
C3D 2016, SP3
Dell Precision M4800, Windows 10 64-bit