Unwrap UVW - Baking texture

[RadioRahim]

I have spent over 5 hours trying to consistently get a proper baked texture (shadow map) exported from 3ds max. I was at one point successful.
In order for you to see what it is supposed to look like please look at my scene below:
































I am trying to bake the textures onto the plane at the bottom (bricks). I was successful at one point exporting the shadow texture seen here:


 I was then able to map this to a transparent layer and work the baked textures into my scene, but.. I can not remember what settings I used and I haev tried for hours to fiddle with the settings with no end in sight.. 

Before rendering a texture we must use a UVW unwrap modifier in order to use the Flatten Mapping feature that is required before baking a texture. When I do this my texture is rendered in the wrong size (as the UVW mapping is destroyed with Flatten Mapping), however this should not matter to much as I am only creating a shadow texture that will fir over this texture. I can work with a duplicate model and save the original, placing the shadow texture over the original undamaged model. 

The problem is presently that the output of the render is using the damaged UVW map. Below are tow photos, first, the undamaged UV map at the proper scale:



and second, the UV map after it has been damaged by the Flatten Mapping function:



As you can see the scale & position has changed  and the mapping has been set our in shapes. While this might not be a problem if I were actually able to load this UV mapping back into the texture, this does not actually work (and the issue here is about render output).. I have never been able to properly save and then load a UV map in 3ds Max anyways. In any case, the renderer is using this changed UV and I am unsure of how I was able to produce the working example shown above. 

It should be noted that I have Render to Files only selected, Save Source(duplicate Source to be Baked) selected, Views and render are bother set to Baked. No projection mapping enabled. 


If anyone can tell me the proper settings to export only the shadows for a model I would be very appreciative, as spending over 5 hours trying to make this work is beyond frustrating. 

[RadioRahim]

I found a video I mad e of the successful render and it seems that the problem here is in the flattened UVWs. I need to somehow flatten my UVWs like the video but I do not know how I did this! It looks like I rotated and scaled the mapping, but when i try to flatten it again it reverts back to those squiggly lines. Please help!

EDIT -  After scaling the texture to fit the image size and rotating it, I tried some settings in the UVW window... convert polygons to vertex. This seems to have solved the squiggly line issue before rendering

[Anonymous]

If you already have proper UV`s on your model why do you insist using automatic unwrap/Flatten? Just use the existing UV`s like in the video.

[RadioRahim]

this is only a workaround. I want to learn this properly. How am I supposed to flatten UV's while preserving the proper scale? The only way this bake shadows is going to work is that the tetxure all fit inside that square in the UVW window. So its basically a workaround. 

IS there no way to reserve the scaling I want while still being able to export the shadows?

[Alfred.DeFlaminis]

Hello @RadioRahim,

There may be some misconceptions about what must be done here so if you have any questions I am happy to help.  It seems that you are scaling your UV's out of the 1:1 square in order to tile textures rather than doing the tiling on the material level.  You should be able to use the same UV island as your typical mapping, or you could flatten them and apply them to channel 2.  Loading and saving UV's works well and maybe I should spend some time with you on that, I'm happy to help.  

It's quite possible you have specific instructions and I just don't understand them, so any clarification would be greatly appreciated.  There's no part of the process typically that says you *have* to flatten UV's though.  As a matter of fact many engines use the 1:1 square uvs for the light maps on Autobake.  

All you need to do after flattening them if that's the route you want is to pack them afterwards.  Tools->Pack Uvs.  

Best Regards,

[RadioRahim]

"It seems that you are scaling your UV's out of the 1:1 square in order to tile textures rather than doing the tiling on the material level.  You should be able to use the same UV island as your typical mapping, or you could flatten them and apply them to channel 2"

Yes I usually do scale my UVs that way (didn't know about the other way) I am open to talk about that as I want to learn the best way for games as this is what I am creating models for. I am not sure  how baking works where we have to flatten the UVS first. In all the tutorials I watched everyone did this first so I thought we had to do this. Let me know if I am wrong. 

 All i really want to do here is use a light source in 3dsMax to cast shadows, and then bake the shadows into textures that I can later map onto objects with transparencies mapped to black. This way I can have transparent shadows in my game overlaying the other objects in the game. I was successful with 2 of the experiments. I simply used UVW Unrwap, flattened the textures, rendered them out as shadows only in a dds file with transparency, then remapped  these to duplicate objects in 3ds Max to create shadows. This worked twice but other times I got grievous errors:

"A ray encountered an invalid color (NaN), using black instead" and my bakes come out wrong, or entirely black.
This forced me to abandon an entire day's work and had to revert to an older version of my project without shadows as one of the objects simply would not render out shadows because of these errors. 

What might help me here is if you wrote out step by step the best method for exporting shadows form objects that are mapped with UVW that expand beyond the 1:1 frame, (if it means flattening I will be open to that). Also if you could address the render error I talked about that would be helpful. 

Thank you for your time. 

[Alfred.DeFlaminis]

Hello @RadioRahim,

I apologize for my late reply, we've got a training week here at Autodesk and my days are filled with meetings.  I would be happy to show you my method of baking the shadows.  I'll create a video for you over the weekend, it would be my pleasure. 

In the meanwhile, do you want me to create that video on your scene file specifically?  I can do that if you'd like.  I'm going to create a folder for you since I can't check my messages as often as I'd like this week.  You don't have to use it if you'd rather not but it might make the video more relevant.  Thanks!

I have set up a private folder for you if you'd like to upload your files there. You should get an email about it shortly, and if you don't see it please check your SPAM folder. These files are private and will not be shared with anyone but yourself and the support staff. Please let me know if you have any questions. Please use at least an 8 character password or the Box account may not work properly. Once you have uploaded your files, please post a comment here or send me a PM so I know to check the Box account. Thanks!

Best Regards,

[Fadhil_Farook]

It's not necessary that you have to flatten UV"s every time if you've already unwrapped it. That error message could be because of overlapping UV's. If you've unwrapped it the proper way, then you can just use that instead like the other member mentioned.

[Alfred.DeFlaminis · Accepted solution]

Hello @RadioRahim,

@Fadhil_Farook is correct that the normal UV's could be used for this, provided they are all in the 1:1 square.  If they are not, then the baker wouldn't be able to generate them properly.  This is probably why you are flattening and packing them in the first place, though I can't be 100% positive what the requirements are on this.  

I went looking for resources about Render to Texture and I thought I'd share some here.  If you have more questions I can handle those in a video if you'd like.  I just need confirmation this is still an issue for you, thanks!  

Baking shadows in 3Ds max

Render to Texture Basics

Baking Maps with Render To Texture by Paul Neale

3DS Max Lightmapping

Generating Lightmaps

Although Youtube isn’t part of Autodesk, this post may have information that could help you. Please use best practices to safeguard your systems if you decide to adopt any of the suggestions on this site.

Best Regards,

[RadioRahim]

Hey I'm back again.. looks like I had things running smoothly for a while, but now I am faced with some issue I dont know how to solve. I have made a video illustrating the issue:

Any help is greatly appreciated.