Firstly set the annotation scale to 1:1 from the status bar at the bottom right corner and then set "ANNOAUTOSCALE" to "-4" in model space and paper space.
Also set the "SELECTIONANNODISPLAY" to "0" which would eliminate visibility of all the alternate scale set to the block.
See the attached drawing with text at 1:1 scale.
--------------------------------------------------------------------------------------------------------------------------
ವಿನೋದ್ ಕೆ ಎಲ್( System Design Engineer)

Likes is much appreciated if the information I have shared is helpful to you and/or others.
Please mark "Accept as Solution" if my reply resolves the issue or answers your question, to help others in the community.
--------------------------------------------------------------------------------------------------------------------------