When using Render to Texture inside of 3DS Max to bake projection style maps of data from one model onto another, you need to use the Projection Modifier on the target (low poly) model. Once this modifier is applied you then would use the Reference Goemetry > Push > Amount to adjust the cage. The Cage is essentially a visual representation of the distance that the rays will individually travel in order to find target hits, so you will want your cage to be larger than the source (high poly) model.
The goal here is to ensure that the cage is not hidden by any of the source (high poly) model, while not being so larger that it gets a "hit" from another piece of geometry on the model. This is what I suspect is going on with your model is that the rays are finding hits from other geometry and using that as the data. To fix this, you can enter into the vertex component level of the Projection modifier and adjust individual vertices of the cage to better fit.
