In addition to what I mentioned above, if you have photoshop (or whatever), and can open your PDF/s and check the "image" parameters, you can likely reduce the resolution. I've scanned in documents at 600dpi, that choke my computer (Win7 64bit, i7 chipset quad core, 16g ram, SSN drive, nvidia K4000 graphics card - it's a beast), but reducing the PDF down to 150 dpi, or even 96 dpi, you can still get quite good quality, and your computer will be much happier dealing with multiple PDF's in a drawing. It not only makes the drawing smaller (physically), but it reduces the file size.
It does take some trial and error to determine the sweet spot for dpi for your PDF's, but doing so will make your life much happier. That said, my biz partner does a lot of hand-drawings, that I scan, and incorporate into our Autocad Architecture 2013 files. I normally have a drawing just for images, such as PDF's, and reference this image dwg, into my main construction document dwg. Still, when I have 20 PDF's, ranging from 1mb - 10mb, I do get that zoom-stagger, where my computer seems to say "WTF! are you trying to make me do??".
I put all PDF's on one layer name (in the image drawing), so when in a model space tab that is for floor plans, I'll freeze the PDF/s xref layer, and the zooming works like butter. It only bogs down when I'm in Model space, and all my layers are ON.
I've yet to figure out a work around, for how Autocad deals with PDFs....Jpegs, PNG's, tiffs, bmp's dont look that great due to the compression issues, and if you want them to look good, the file sizes explode - only pdf's seem to avoid image degradation. I've concluded that with PDF's, you have to take the time to find the right balance between PDF file size, and PDF file resolution (dpi) - and how it works with your computer/plotter.
Good luck people!