Are you absolutely sure it's the Xref's that are causing the lag? if not, try to unload all of them. If you don't have any more lag after that try reloading them one at a time to see if it's all xref's or just one of them. Or maybe it's just too many xref's and your computer can't handle them all.
AutoCad LT 2014