Multiple Textures Blended By Vertex Colours

[FluffyVox]

Hi,

A colleague recently showed me a Blender set-up that enables up to 9 textures all blended by painting vertex colours in the viewport. Everything was visible in the viewport with no lag. There were 3 main material types (eg, rock, grass, dirt) that are controlled by painting red, green and blue vertex colours. Then within each of those, 3 textures can be used to create the end result - again, by painting red, green and blue to control which texture is visible.

I began wondering how to recreate it in Max and I haven't got a clue where to begin in all honesty! I'm familiar with all of the usual multi-sub object materials, composite maps and blend maps, but I don't see how I could achieve anything approaching the elegance of what he had. I'm a bit out of the loop as I've only just got back into Max after a long hiatus. Is this type of thing possible with the OSL maps or do you think it'd need a specific shader written for it?

This is a common thing to do for games so any insight into the latest methods of doing this in Max would be much appreciated. Thanks!

[dmitriy.shpilevoy]

There are few ways of doing it in max. For example with composite

problem is you will only see correct result when rendering. For it to display in viewport you would need to make a custom shader and use it with DX material.

This one allows 4 textures mix:
https://dimwalker.gumroad.com/l/dwvcdxmat

[FluffyVox]

Ah, thank you! I tried that exact set-up with the composite map but didn't think it was working properly due to using the viewport. I haven't got the tech knowledge to write a custom shader and having to render every time is too slow and cumbersome when you're looking for realtime feedback in the viewport as you paint.

Appreciate the link to the material. It looks useful for a simplified solution but I'm struggling to understand why Blender's viewport can display whatever you throw at it and yet for Max, you need to start writing custom shaders :(.

[Diffus3d]

Scriptspot is a good place to look for things that others have built for specific needs I've found, though their search doesn't even work from what I can see.  This shader for example seems to do what you need, though I am not sure if it's supported with your intended renderer.  You might also be able to get viewport rendering if you use a directx shader.  (Maybe with a the standard shader plugged into it using the composite maps, but I don't know for sure.)  

Traditionally vertex blending was always done in game engines as a way to preserve texture memory.  It used to be that using high end 3D apps for vertex color shading was sort of limiting because it has less control than uvs.  (That's not to say it was never used, just problematic with things like turbosmooth.)  With uvs you have more pixel level control but have to deal with layers as noted by @dmitriy.shpilevoy.  Some folks use live link with substance painter to layer textures. 

In games you need entire levels to look good from every angle, but most high end 3D shots are only what the camera sees.  It didn't make a ton of sense to do it in 3dsmax because if it was going to a game engine that engine almost certainly had it's own quirks and setup procedures and wouldn't support any shader exported with a convoluted network of composite maps.  I suppose they never got around to making a shader specifically for it for this reason. 

Best Regards,

[FluffyVox]

Not quite the solution I was hoping for but thank you for taking the time to reply. I was simply wondering if I could just replicate what I saw in Blender but it clearly isn't possible in Max without doing a load of extra work (like writing a custom DX shader). Blender unfortunately takes the win on this one but that's fine - I'm not a fanboy of either. 

[MartinBeh]

First thought that crossed my mind: Try OSL

If we can access multiple vertex color channels in OSL (have not tried yet), this should be doable.

[MartinBeh]

Here is a simple OSL shader to alpha-blend up to six color maps:

// Alpha-composits up to six colors over another
// M. Breidt - 05.09.2024 - based on composite.osl

shader CompositeN
[[ string help = "Alpha-composits six colors over another",
   string category = "Compositing",
   string label= "Composite" ]]
(
	color TopRGB    = 0.5 
		[[ string label = "Top layer RGB" ]],
	float TopA      = 1.0 
		[[ string label = "Top layer Alpha" ]],
	color Layer2RGB    = 0.0 
		[[ string label = "Layer 2 RGB" ]],
	float Layer2A      = 0.0 
		[[ string label = "Layer 2 Alpha" ]],
	color Layer3RGB    = 0.0 
		[[ string label = "Layer 3 RGB" ]],
	float Layer3A      = 0.0 
		[[ string label = "Layer 3 Alpha" ]],
	color Layer4RGB    = 0.0 
		[[ string label = "Layer 4 RGB" ]],
	float Layer4A      = 0.0 
		[[ string label = "Layer 4 Alpha" ]],
	color Layer5RGB    = 0.0 
		[[ string label = "Layer 5 RGB" ]],
	float Layer5A      = 0.0 
		[[ string label = "Layer 5 Alpha" ]],
	color BottomRGB = 1.0 
		[[ string label = "Bottom layer RGB" ]],
		
	output color Out = 0.0
)
{
	Out = BottomRGB;
	Out = (Layer5RGB * Layer5A) + Out * (1.0 - Layer5A);
	Out = (Layer4RGB * Layer4A) + Out * (1.0 - Layer4A);
	Out = (Layer3RGB * Layer3A) + Out * (1.0 - Layer3A);
	Out = (Layer2RGB * Layer2A) + Out * (1.0 - Layer2A);
	Out = (TopRGB * TopA) + Out * (1.0 - TopA);			
}

If you now feed the first component of UVW Channel #2,3,4..  into the respective "Alpha" slots and your color maps into the respective "RGB" slots, it should blend them and show you the result both in the viewport and in rendering.

Finally, you need to add one VertexPaint modifier per UVW channel onto the mesh object, set it to the respective UVW map channel number, and then paint in greyscale.

But I admin that switching between the channels is a bit cumbersome... =8-) 

[MartinBeh · Accepted solution]

I got curious - attached is a .max file for 3ds Max 2023 that contains a working prototype for this with one base layer and three additional layers (all layers are OSL procedurals). Select the plane and check out the VertexPaint modifiers that control the three layers.

Hope this helps!

[Diffus3d]

@MartinBeh is a hero, good stuff.  

[FluffyVox]

Hi Martin

Thank you so much for looking into this and providing the example OSL shader and accompanying scene. It's a huge step in the right direction and I look forward to experimenting with it. 👍

It's quite a useful set-up for a lot of game related tasks in particular so I'm sure others will find it equally valuable.

All the best!

[mark.clay]

Would this be suitable using Vray? 

[MartinBeh]

I don't have Vray to test but every renderer that supports OSL should be able to use it. 

And looking at this it seems Vray can do OSL.

Just give it a try and let us know... 🙂