I had the same error and ended up narrowing it down to my output resolution settings. The Maya file I received from a client had the resolution set to 1200x1729. And after much troubleshooting with smoothing levels, displacement shader settings, and importing the meshes into a clean scene, I tried rendering with a new camera with default settings and it finally started rendering without crashing Maya. So, I used another vertically-oriented preset (in this case "Letter") to get close to matching the client's dimensions and then adjusted the ppi from 300 to 72 and changed the height dim to 1730 with "Maintain width/height ratio" enabled. With that, my new resolution ended up being 1338x1730. and it rendered without issue.
Just reading this thread, it appears this error might happen for a number of reasons. In my case it was a resolution setting. So, I figured I'd share and add it to the pile of potential causes.
Just an aside:
Someday, it would be nice to see a bottom-up rebuild of Maya. I know this is a heavy lift. But I've been a Maya user before it was Maya (aka Alias Power Animator), and the stability issues seem to have increased over time. I won't use the "B" word, but it's catching up quickly. Just my 2 cents.
