There are only 2 ways you can fix the text height. The second one I have not tried so it may or may not work, because you are using an xref rather then vault or a data shortcut.
#1: Make a new drawing out of the old xref with the initial text size of the label to the scale you desire. Xref that drawing into your full drawing.
#2: Change the scale of the drawing in the settings.
As I said the second one may not work because an xref is being used and it may use the scale of the source drawing over the current drawing.
R10 - 2020 ACAD
2008 - 2020 Civil 3D
2014 - 2020 Plant 3D
2014 - 2020 Revit
V8i - V8i SS4 Microstation
Infraworks
2018-2020 Inventor
2020 Navisworks