Are you absolutely sure it's the Xref's that are causing the lag? if not, try to unload all of them. If you don't have any more lag after that try reloading them one at a time to see if it's all xref's or just one of them. Or maybe it's just too many xref's and your computer can't handle them all.
s.geddes is not alone, I am having the same issues, at first I though it might be contextual ribbons but without the ability to turn them off I cant eliminate that as a potential cause. The same files open fine in 2014 so it shouldnt get worse in a new release.
Access a broad range of knowledge to help get the most out of your products and services.