I believe I found the issue....large number of layer filters. After reading up on other posts, layer filters can slow down a drawing. When attaching xref's or inserting other drawings with a large number of layer filters it can take a while for the dwg to attach or insert.
