Hi,
Based on my experience, there are 2 ways to setup VDM on 3DS Max 2012.
First is the fastest way is to use send to 3DS Max feature on Mudbox. After you generated a VDM file in your high and low poly mesh you can click the "Send to 3DS Max" feature on Mudbox to automatically setup straight away on 3DS Max 2012. If you have 2 tiled UVs it will use the Multi/Sub or if you have 1 tile UV it will just use the standard material. Inside the displacement slot the vector displacement shader was linked and inside this shader the generated VDM is located.
Second method is the manual way, you can save your low poly mesh (obj) and the vector displacement map file. Import manually your low poly mesh in 3DS Max and look for vector displacement shader and plug in to displacement slot inside your chosen material for your generated VDM file.
Inside your vector displacement shader, you have to plug in your VDM file and choose tangent if you generated the VDM file as tangent.
For rendering, you need to setup your renderer as mental ray for it to work. Uncheck the Global Settings and Smoothing under Renderer Tab. After this setup you can now render.