topic Re: Tangent to Vertex Colour in 3ds Max Programming Forum

Tangent to Vertex Colour

klvnk — Tue, 25 Apr 2023 07:47:07 GMT

a continuation of a thread on cgtalk which is neigh on impossible to post to these days

this works for me btw

geo = $ convertToMesh geo meshop.defaultmapFaces geo 0; geo.showVertexColors = true norMod = Edit_Normals() addModifier geo norMod tcols = #() for i = 1 to getNumFaces geo do ( local face = getFace geo i local tface = getTVFace geo i local v1 = getVert geo face[1] local v2 = getVert geo face[2] local v3 = getVert geo face[3] local w1 = getTVert geo tface[1] local w2 = getTVert geo tface[2] local w3 = getTVert geo tface[3] local e1 = v2-v1 local e2 = v3-v1 local delta_u1 = w2.x - w1.x local delta_u2 = w3.x - w1.x local delta_v1 = w2.y - w1.y local delta_v2 = w3.y - w1.y local tangent = (delta_v1 * e2 - delta_v2 * e1) / (delta_v1 * delta_u2 - delta_v2 * delta_u1) for j = 1 to 3 do ( local normalID = norMod.getNormalID i j local normal = normalize(norMod.GetNormal normalID ) tangent = normalize(tangent - (dot tangent normal)*normal) local bitangent = cross normal tangent local TBN = Matrix3 tangent bitangent normal [0,0,0] TBN = inverse TBN smNor = [0,0,0] norsID_bit = #{} norMod.ConvertVertexSelection #{face[j]} norsID_bit for k in norsID_bit do smNor += norMod.getNormal k smNor = normalize smNor tnor = normalize(smNor * TBN) tnor = tnor *0.5 +0.5 tcol = [tnor.x *255.0, tnor.y *255.0, tnor.z *255.0] as color append tcols #(tcol, i, j) ) ) convertToMesh geo visited =#{}; for v = 1 to tcols.count do ( vcverts = getVCFace geo tcols[v][2]; vci = vcverts[tcols[v][3]]; if not visited[vci] then ( visited[vci] = true; setVertColor geo vci tcols[v][1]; ) ) update geo

Re: Tangent to Vertex Colour

547057474 — Tue, 25 Apr 2023 10:21:03 GMT

this is my result from you code,also weird,aren't you ?

Re: Tangent to Vertex Colour

klvnk — Tue, 25 Apr 2023 15:13:56 GMT

ok Yoda

perhaps you want the "exploded" version you could try this...

meshop.setmapsupport geo true; setNumCPVVerts geo (3 * getNumFaces geo) --defaultVCFaces geo; --this is wrong :/ vi = 1 for f = 1 to getNumFaces geo do ( setVCFace geo f [vi, vi + 1, vi + 2]; vi += 3; ) for v = 1 to tcols.count do ( vcverts = getVCFace geo tcols[v][2]; vci = vcverts[tcols[v][3]]; setVertColor geo vci tcols[v][1]; ) update geo

Re: Tangent to Vertex Colour

547057474 — Tue, 25 Apr 2023 11:13:48 GMT

for my case,this box has 108 tri faces,per face has 3 verteces, <tcols> collected 324 colors,and the num of cppvertes is also 324,so i dont think it will draw twice per vertex. isn't it?

Re: Tangent to Vertex Colour

547057474 — Tue, 25 Apr 2023 11:39:59 GMT

It's a bit strange

Re: Tangent to Vertex Colour

547057474 — Tue, 25 Apr 2023 11:42:20 GMT

i made in Houdini ,this is i wanted,but not work in max

Re: Tangent to Vertex Colour

klvnk — Tue, 25 Apr 2023 12:04:56 GMT

what result do you get from the original script @ the top of the thread with a default box ?

Re: Tangent to Vertex Colour

klvnk — Tue, 25 Apr 2023 15:14:52 GMT

to "match" the uv's (hard edges at the uv seams) use

note I've changed the exploded version above as it had an error

meshop.setmapsupport geo true; setNumCPVVerts geo (getNumTVerts geo); for f = 1 to getNumFaces geo do setVCFace geo f (getTVFace geo f) visited =#{}; for v = 1 to tcols.count do ( vcverts = getVCFace geo tcols[v][2]; vci = vcverts[tcols[v][3]]; if not visited[vci] then ( visited[vci] = true; setVertColor geo vci tcols[v][1]; ) ) update geo

Re: Tangent to Vertex Colour

547057474 — Wed, 26 Apr 2023 02:33:17 GMT

you mean this one ?

geo = $ convertToMesh geo addModifier geo (vertexpaint()) geo.showVertexColors = true addModifier geo (Edit_Normals()) norMod = geo.Edit_Normals tcols = #() for i = 1 to getNumFaces geo do( local face = getFace geo i local tface = getTVFace geo i local v1 = getVert geo face[1] local v2 = getVert geo face[2] local v3 = getVert geo face[3] local w1 = getTVert geo tface[1] local w2 = getTVert geo tface[2] local w3 = getTVert geo tface[3] local e1 = v2-v1 local e2 = v3-v1 local delta_u1 = w2.x - w1.x local delta_u2 = w3.x - w1.x local delta_v1 = w2.y - w1.y local delta_v2 = w3.y - w1.y local tangent = (delta_v1 * e2 - delta_v2 * e1) / (delta_v1 * delta_u2 - delta_v2 * delta_u1) for j = 1 to 3 do ( local normalID = norMod.getNormalID i j local normal = normalize(norMod.GetNormal normalID ) tangent = normalize(tangent - (dot tangent normal)*normal) local bitangent = cross normal tangent local TBN = Matrix3 tangent bitangent normal [0,0,0] TBN = inverse TBN smNor = [0,0,0] norsID_bit = #{} norMod.ConvertVertexSelection #{face[j]} norsID_bit for k in norsID_bit do smNor += norMod.getNormal k smNor = normalize smNor tnor = normalize(smNor * TBN) tnor = tnor *0.5 +0.5 tcol = [tnor.x *255.0, tnor.y *255.0, tnor.z *255.0] as color append tcols tcol ) ) convertToMesh geo for i = 1 to meshop.getNumCPVVerts geo do setVertColor geo i tcols[i] update geo

This result is the closest to houdini,most of them are correct, but the errors are concentrated in the corners

Re: Tangent to Vertex Colour

klvnk — Wed, 26 Apr 2023 08:51:09 GMT

slight variation of the above to illustrate whats going on

geo = $ convertToMesh geo addModifier geo (vertexpaint()) geo.showVertexColors = true convertToMesh geo tcols = #() tcols.count = getNumCPVVerts geo; addModifier geo (Edit_Normals()) norMod = geo.Edit_Normals vci = 1; for i = 1 to getNumFaces geo do( local face = getFace geo i local tface = getTVFace geo i local v1 = getVert geo face[1] local v2 = getVert geo face[2] local v3 = getVert geo face[3] local w1 = getTVert geo tface[1] local w2 = getTVert geo tface[2] local w3 = getTVert geo tface[3] local e1 = v2-v1 local e2 = v3-v1 local delta_u1 = w2.x - w1.x local delta_u2 = w3.x - w1.x local delta_v1 = w2.y - w1.y local delta_v2 = w3.y - w1.y local tangent = (delta_v1 * e2 - delta_v2 * e1) / (delta_v1 * delta_u2 - delta_v2 * delta_u1) for j = 1 to 3 do ( local normalID = norMod.getNormalID i j local normal = normalize(norMod.GetNormal normalID ) tangent = normalize(tangent - (dot tangent normal)*normal) local bitangent = cross normal tangent local TBN = Matrix3 tangent bitangent normal [0,0,0] TBN = inverse TBN smNor = [0,0,0] norsID_bit = #{} norMod.ConvertVertexSelection #{face[j]} norsID_bit for k in norsID_bit do smNor += norMod.getNormal k smNor = normalize smNor tnor = normalize(smNor * TBN) tnor = tnor *0.5 +0.5 tcol = [tnor.x *255.0, tnor.y *255.0, tnor.z *255.0] as color tcols[vci] = tcol; vci += 1; ) ) convertToMesh geo for i = 1 to meshop.getNumCPVVerts geo do setVertColor geo i tcols[i] update geo

adding the VertexPaint modifier "explodes" the vertex color channel ie a unique color vert for every face corner. The above duplicates the exploded order... so you may need to re-evaluate the Houdini result.

as will this snippet...

geo = $; addModifier geo (vertexpaint()); convertToMesh geo; for f = 1 to getNumFaces geo do format "% %\n" (getVCFace geo f) (getFace geo f);

or this

geo = $ addModifier geo (vertexpaint()); geo.showVertexColors = true; convertToMesh geo; setVertColor geo 5 red; setvertselection geo 5; subobjectLevel = 1; update geo;

Re: Tangent to Vertex Colour

547057474 — Wed, 26 Apr 2023 08:54:55 GMT

when iterate faces,the <getVCFace> will collected #{1..numCPPverts},it is similar to houdini's <linear vertex index>,I don't know any other way to get the vertex number of each face in max,per face must has unique vertex num to set color.

houdini's result export to unity is also correct,so ,

forgot about this ,may be max just can not do this .

Thank you very much for your help. It has given me a lot of inspiration. Perhaps one day in the future, I can solve this problem and I will come back and post the code

Re: Tangent to Vertex Colour

klvnk — Wed, 26 Apr 2023 09:28:45 GMT

perhaps a handedness issue or a ccw/cw thing going on ?

Re: Tangent to Vertex Colour

denisT.MaxDoctor — Wed, 26 Apr 2023 19:01:21 GMT

Where can I get the test geometry (box)?

could you post the file (v 2020) or code to make the box?

Re: Tangent to Vertex Colour

547057474 — Thu, 27 Apr 2023 02:48:30 GMT

this model made in blender ,because i am not good at max modeling

Re: Tangent to Vertex Colour

denisT.MaxDoctor — Thu, 27 Apr 2023 05:25:42 GMT

could you guys tell the whole story from the beginning?

what do you calculate?

Re: Tangent to Vertex Colour

547057474 — Thu, 27 Apr 2023 06:19:22 GMT

basicly i wanted to convert world space smooth vertex normal to tangent space ,then save the tangent normal to the vertex color.

after others help,i achieved in houdini,it is correct because i export to unity,the shader is worked

but i need this in max,so i write a maxscript

and i got this , almost there ,but some corner is error,i dont know why cause this,and i have no idea to fix it

Re: Tangent to Vertex Colour

denisT.MaxDoctor — Thu, 27 Apr 2023 17:13:00 GMT

As I understand it, you are trying to "reverse engineer" the normal map where you bake the difference of the real vertex normal and render vertex normal as vertex color value. After that, you want to use this data in the real-time shader as an offset to the real normal to visually smooth "hard edges".

Klvnk's code is right and works as expected:

but the pipeline itself is wrong. We shouldn't do programmable normal averaging but have use apply correct (desired) smoothing groups to "edge" faces first. At least we have to specify "hard edges" which need to be smoothed.

Moreover, I do not see the point of such a decision at all. If you have a custom shader why not bake desired (final) normals instead of normal offsets?
Well... I see some reason. It makes sense to use offsets if the mesh is deformable.

Re: Tangent to Vertex Colour

domo.spaji — Fri, 28 Apr 2023 00:31:59 GMT

What about more basic thingy...

To transfer normals from high-res mesh to low-res through vertex color "low-res" mesh should have (at least)= vertices as high-res.

So, whats the point🤔

Re: Tangent to Vertex Colour

denisT.MaxDoctor — Fri, 28 Apr 2023 00:45:50 GMT

@domo.spaji wrote:
What about more basic thingy...
To transfer normals from high-res mesh to low-res through vertex color "low-res" mesh should have (at least)= vertices as high-res.
So, whats the point🤔

Of course, baking (render) normals from high resolution geometry into a normal map is a common way to do this sort of thing. But it looks like there is a requirement caused by custom shader design.

Re: Tangent to Vertex Colour

denisT.MaxDoctor — Fri, 28 Apr 2023 21:52:36 GMT

I think it should be something like this:

undo on ( node = $ mesh = convertToMesh node numf = getNumFaces node node.showVertexColors = true vcolors = #() for i = 1 to numf do ( face = getFace mesh i tface = getTVFace mesh i v1 = getVert mesh face[1] v2 = getVert mesh face[2] v3 = getVert mesh face[3] w1 = getTVert mesh tface[1] w2 = getTVert mesh tface[2] w3 = getTVert mesh tface[3] e1 = v2 - v1 e2 = v3 - v1 delta_u1 = w2.x - w1.x delta_u2 = w3.x - w1.x delta_v1 = w2.y - w1.y delta_v2 = w3.y - w1.y tangent = (delta_v1 * e2 - delta_v2 * e1) / (delta_v1 * delta_u2 - delta_v2 * delta_u1) rnormals = meshop.getFaceRNormals mesh i normal = getfacenormal mesh i for k = 1 to 3 do ( tangent = normalize (tangent - (dot tangent normal) * normal) binormal = cross normal tangent tangent_tm = Matrix3 tangent binormal normal [0,0,0] tnormal = rnormals[k] * (inverse tangent_tm) tnormal = (tnormal + 1)/2 vcolor = (tnormal * 255) as color append vcolors vcolor ) ) meshop.setMapSupport mesh 0 on setNumCPVVerts mesh (mesh.numfaces * 3) on for i = 1 to numf do setVCFace mesh i ([1,2,3] + 3*(i-1)) for k = 1 to getNumCPVVerts mesh do setVertColor mesh k vcolors[k] update mesh )

As I said above, it makes sense to bake the difference between geometry and render normals.
All this doesn't make sense if the shader can use (as mostly all modern shaders) specified normals.