The text height is controlled in the label style.
The height in the style governs what height your text will appear when plotted.
If you text height in the style is 2.5mm then the text will appear at 2.5mm when plotted from a layout containing a viewport no matter what scale the viewport is.
When your in model space the text size is scaled to 2.5mm (using the above example) for a plot if it was done at the annotation scale set for model space. Change the annotation scale and the text scaling changes in model space.
So it sounds like you might have a text style set up to show in model space at say 2.5mm at a scale of 1:100 but if you then go to a paperspace layout and have a viewport set to 1:1000, the text is 10 times the size.
If you set your text height in the style to the size you want to plot, and you are using a 1:1000 viewport, set the model space annotation scale to 1:1000 to see how it will appear.
I'm sure I've made it sound more complicated than it is, maybe someone else can simplify it.
If a post provides a fix for your issue, click on "Accept as Solution" to help other users find solutions to problems they might have that are similar to yours.
Andrew Puller
Maitland, NSW, Australia
Windows 10 Enterprise 64bit
Intel core i7 11800 @ 2.30 GHz with 32GB Ram
Civil 3d 2021