I've been running into an issue with the Layer Properties Manager recently in AutoCAD 2011. It frequently becomes very slow by refreshing the layers multiple times when a selection is made within the Manager. The refresh is so slow, I can count each layer as it is diplayed, and sometimes it refreshes as many as 5 times for each selection. This is painfully slow, even on drawings with relatively few layers.
This does not occur with any other feature in AutoCAD.
You might also want to check for the layer filter bug that infiltrated hundreds of thousands of drawings around the CAD world - every time you XREF'ed or inserted all the previous layer filters would get stacked up. The bug has been fixed, but some of those drawings still contain that junk. Do a search for "layer filter delete" and you should get numerous solutions.