Simple idea - have a toggle in the bottom right hand corner of the window near the toggles that determine you can/cannot select links or things by face etc. except this toggle toggles your ability to select model components on and off.
Many times, annotation and modeling work happen at the same time. Somethimes, though, you're just in annotation mode and you're creating a view that requires a large amount of annotation components - detail items, tags, dimensions, etc. When I'm making an equipment plan, for instance, I don't want to be able to select rooms, or walls, or floors, or really any model component and frustratingly accidentally affect the model. The same thing applies to when I am adding detail to a wall section, or any number of views that require detail.
Working in a view and having to cycle through the Tab key 10+ times to select the correct annotation object when you really just want to select annotation objects over the model is very frustrating. An Annotation mode that greys out model components and doesn't allow you to select them would be very helpful.