Are you Linking the Building Model into a Site Plan with the Toposurface, or creating your Toposurface in the same Project as your Building Model? If the latter, typically, the Toposurface is moved to its proper position relative to the Model. The Model is not moved at all. Also, Levels, Contour Labels, Spot Elevations are relative to their "Elevation Base" - typically the Survey Point or the Project Base Point. Those are what you move -- not the Building Model.
Note: the Point Elevations inside Toposurface Edit Mode have no significance outside of Edit Surface mode. Ignore them.