Still doesn't make sense to me.

Let's take the simplest example.  

I draw a 1" circle in model space.

I place a letter, say 'A' in the circle that is 1/2"

All I really want to do is make sure the circle and the stay at the same proportion.

I guess a way to do it would be to redraw a version of the letter A instead of using text so you don't have to deal with annotation and its complexities/mysteries.