I got some good news 🙂
Nitrous DirectX 11 was unuseable for me as well (3ds Max 2017 and 2018), instant freeze of the whole application after a few seconds navigating in a 10 mio. poly file.
I took some time on the weekend and did a Graphics driver clean-up using DDU (http://www.guru3d.com/files-details/display-driver-uninstaller-download.html) and installed nVidia driver version 361.91 (Driver only, no GeForce Experience...)
It's the one that 3ds Max 2017 is officially approved with, see: https://knowledge.autodesk.com/sites/default/files/file_downloads/3dsmax2017_GFX_ResultsMarch2017.pd...
Since then I'm able to use DirectX 11 Viewport without any issues!
Here's one more thing you need to do after installing the new driver: Disable driver updates in Windows Update!
If you don't, Windows Update will just throw a newer driver onto your system, even while working... (got me a blackscreen during testing in Max...)
See: https://www.howtogeek.com/302595/how-to-stop-windows-10-from-automatically-updating-hardware-drivers...
My System:
Windows 10 Pro 64bit (Version 1607 Build 14393.1198)
Intel Core i5 2500k @ 3.30 GHz
16 GB RAM
EVGA GeForce GTX 970 SC (nVidia)
Hopefully this is going to work for others as well 🙂