Well - in case anyone else runs across this... I found the answer here: http://forums.autodesk.com/t5/Visual-Basic-Customi
In short, SetWindowToPlot expects the values in DCS (Display Coordinate System). DCS is not always equal to WCS - so you have to translate the points from WCS to DCS. I am still a bit puzzled by the fact that 95% of the time (if not more) DCS == WCS.
Start with some of our most frequented solutions or visit the Installation and Licensing Forum to get help installing your software.