For best answers to this questions, post a drawing. You should also explain whether your are using ordinary dimensions, AEC dimensions or both.
If the dimension style is annotative, then it's important to leave the ANNOAUTOSCALE off(0 or negative). This keeps you from generating scales that you don't intend. A long-time bug in autocad for me is that when I open up a drawing that I have previously set to a specific annotation scale, it reverts to 1:1. This would likely generate more references in the annotation dictionary for every annotative object.
If the current annotation scale is set to something other than intended, ANNOALLVISIBLE is on(1), and the viewport is in another scale, such as when you zoom an unlocked viewport accidentally but don't set a specific annotation scale, AutoCAD reverts to a scale of its choice.
If both ANNOAUTOSCALE and ANNOALLVISIBLE remain off throughout the drawing life, then the only unintended negative consequence is disappearing annotation objects when the current annotation scale is set to an identical scale factor but not to an identical scale name. For example, 1:1 and 1'-0"=1'-0' aren't the same and 1:16 and 3/4"=1'-0" aren't the same scale. Remember that this requires a project level decision, especially for sheet files, because leaving ANNOAUTOSCALE on in a sheet file will result in extra scales being added to every attached xref when the scale is changed. So never edit anything other than paper space objects in any ACA sheet.
Note that the default state for the Aec Model (Imperial Stb).dwt is for both ANNOAUTOSCALE AND ANNOALLVISIBLE to be on, the default dimstyle is Annotative, and the default scale is 1:1.
-----------------
If annotative scaling isn't needed, then use non-annotative dimensioning. Add a new dimstyle for each scale (but only one or a few per drawing) by changing the dimscale for that dimstyle and change the style name to make it obvous. I use annotative scaling but am careful to toggle annoallvisible before plotting, to ensure I'm not missing annotation objects.
------------------
Architect, Registered NC, VA, SC, & GA.