I was just recently baffled by this same problem. My contour, point, and alignment labels were too small in some of the layout views regardless of how I set the viewport scale or the annotation scale. What finally fixed it was completing the page setup for each layout tab.
That speaks to the importance of thoroughly setting up your layouts from the beginning.
This problem was most-likely the problem lsimms1 mentioned, with the plot scale in paperspace being set wrong (e.g., page size in inches, for a metric drawing).