topic Re: Xgen - import photoshop mask? in Maya Forum

Xgen - import photoshop mask?

Anonymous — Tue, 03 May 2016 19:23:26 GMT

Hi there,

Is it possible to paint a map outside of Maya, to mask things like clumping within xGen? I'm having a bunch of issues with the 3d paint tool and figure it'd be simpler to just knock one up in photoshop and import it. I have a horrible feeling that it's just 3d paint tool or nothing though?

Thanks!

Ricky

Re: Xgen - import photoshop mask?

pshwayka — Tue, 03 May 2016 22:45:37 GMT

It's a little tricky because xGen uses ptex maps, but it can be done. You may want to try a test scene with just a density mask:

Create a sphere, and apply a new xGen description. For your test, just use the default options in the "create xGen description" window.
For the mask field right under density, click the down arrow and "create map" like you normally would. Leave the start color at white, and then just click the save button. Get out of painting mode.
Save your scene. (this step is important.)
Open the Hypershade and go to the textures tab. There should be a file1, and in the attribute editor for that node you will see that Maya has created an iff file. Note the name and location of that file.
Open the iff file with fCheck, and save it as a .psd file.
Open the psd file in Photoshop, and edit as desired.
Save the file in an image format. (I like png.)
Back in Maya, replace the file1 iff image with your new png file.
Go back into xGen, and next to the mask field, click on the paintbrush icon as if you are going to paint the mask again. Note that your edited mask has replaced the original. IMPORTANT: Click the save icon next to the mask field, even though you have not made any new edits. I believe this step turns your new image file into a ptex map for xGen.
Your new density map should now be in effect when you click the preview button.

Another way you could probably do this (instead of converting the iff file that xGen creates) would be to make a temporary material assignment and a psd network using "color" as the selected attribute, then make a mask image out of that in photoshop. That method would allow you to have the UVs as a layer in Photoshop so you could orient yourself...but don't include that layer when you save your mask image. This method would still require steps 1-3 and 6-10 above.

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 04:39:27 GMT

Incredible reply pshwayka! I will be giving this a go first thing tomorrow 🙂

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 15:20:40 GMT

Sadly I got a host of errors and warnings. Am I doomed?

// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(896:0.264782,0.319337) //

// Error: XGen: Runtime error while setting up paint tool (C:\Program Files\Autodesk\Maya2016\plug-ins\xgen\scripts\xgenm\ui\widgets\xgExpressionUI.py:939) //

// Warning: file: C:/Program Files/Autodesk/Maya2016/scripts/others/art3dPaintSetPaintAttr.mel line 45: Some surfaces have no file texture assigned to the current attribute. //

// Error: XGen: Trouble closing ptx file.

The only difference encounted following your steps was that fCheck didn't allow me to save as a psd. So I saved as a targa instead. I don't suspect that's what's causing all these messages though?

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 15:33:46 GMT

Got it working!

It didn't like the png format for me. I tried tga and iff and both seem to be working!

Thank you so much! Going to try on my cat now...

Re: Xgen - import photoshop mask?

pshwayka — Wed, 04 May 2016 15:35:36 GMT

Cool! Keep me posted.

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 15:40:58 GMT

I am getting hit with these errors each time I update the preview though. Any idea what that's about? It's just a default sphere :

// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(854:0.405213,0.473110) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(854:0.405213,0.473110) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(842:0.280164,0.467221) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(845:0.379402,0.242790) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(851:0.264782,0.319337) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(944:0.379402,0.242790) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(959:0.280164,0.467221) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(1013:0.442642,0.341042) //
// Error: line 1: XGen: Failed to evaluate subd. description8,pSphere1(1019:0.280164,0.467221) //

Thanks,

Ricky

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 16:25:33 GMT

And now the usual process is working (after countless failed attempts previously)!

I've no clue why, but perhaps the order of steps that you outlined previously is what enabled it to work using the 3d Paint tool.

Thanks again pshwyka! It's clumping time.

Re: Xgen - import photoshop mask?

pshwayka — Wed, 04 May 2016 16:41:37 GMT

Sorry....I got sidetracked for a while. I just tried the process again and didn't get those errors, so I'm not sure why you are. Hmmm.

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 19:18:59 GMT

Mysteriously the errors aren't there on my cat mesh, however I have a different issue now...

I'm trying to clump some hairs on the body with the face masked out. However I'm getting a small patch above the eyes that's getting shorter. I've tried flooding the mask black and setting clump to 0 to debug, but the influence remains. It's the same on both sides. Actually all the hairs on the cat move ever so slightly when I preview (still set to 0) but not enough to be a problem. All "Effects" are also set to 0.

The only way I've found to get rid of this is to uncheck clumping entirely, which is obviously not want I want - although manually painting the clumps might actually be quicker in the end!

Do you have any ideas pshwayka?

Re: Xgen - import photoshop mask?

Anonymous — Wed, 04 May 2016 19:36:38 GMT

Ok, so simply adding a clump mod, setting the amount to 0 and hitting preview causes numerous changes throughout the character, including some unwanted short spots. The worst is after I add the mask, but currently looking like I'll have to manually groom the effect 😕

Re: Xgen - import photoshop mask?

Michael_Todd — Fri, 13 May 2016 22:03:46 GMT

Are you using region mapping?

If you are then this can cause some issues if you are using maps painted outside of Maya. When generating clump maps, if you have a region amp, then the clump map uses the region map as a control. What this does is to make sure that the clump cell borders match with the region borders to prevent clumping region misalaignment with the region map. If this were to happen, clumps that should be on one side of a region could be pulled to the wrong area as the cell borders do not align.

Is there any chance you could psot the scene for me to take a look at? It's hard to assess what the true issue is without seeing the setup..

Cheers

Re: Xgen - import photoshop mask?

pshwayka — Fri, 13 May 2016 22:15:38 GMT

Hi Michael,

Speaking of region maps...I went to edit one region on a model the other day, and every time I applied a brush stroke, all of the regions' colors changed...but only during the brush stroke. This was on a 2013 Mac Pro. I tried the same operation on the same model under Windows, and it worked fine.

Have you seen this happen before?

BTW...every time I see your avatar, I think of Rutger Hauer in Blade Runner.

Re: Xgen - import photoshop mask?

Michael_Todd — Fri, 13 May 2016 23:08:36 GMT

Good, that's who its supposed to be!

Sounds like a video card/driver issue....

Re: Xgen - import photoshop mask?

pshwayka — Fri, 13 May 2016 23:38:30 GMT

Actually, my memory of the issue was incorrect...the region map stays the wrong color after it is painted...and I can recreate it at will. This is on 2016 Ext2 (El Capitan.) It seems to be an issue with the new OpenGL core profile-strict mode, because it doesn't happen under regular OpenGL, and it also does not happen in Ext1 SP6 under the old OpenGL core profile.

Might be something to look at if Maya 2017 is going to be supported on El Capitan...

Re: Xgen - import photoshop mask?

Michael_Todd — Sun, 15 May 2016 08:20:21 GMT

That must be an Artisan/drivers issue specific to OSX. I'm not sure if it's a known issue though, in terms of being looked into.

It looks like the color value being applied is the inverse of the selected color.

Re: Xgen - import photoshop mask?

pshwayka — Sun, 15 May 2016 13:43:52 GMT

Iterestingly, it doesn't seem to occur during texture painting using the 3d Paint tool. (Sorry...didn't mean to hijack this thread.)

Re: Xgen - import photoshop mask?

Michael_Todd — Sun, 15 May 2016 23:26:38 GMT

if you hold cmnd down while painting, do you get the correct color?

Re: Xgen - import photoshop mask?

pshwayka — Sun, 15 May 2016 23:38:19 GMT

Command+LMB (or tablet pen drag) is actually like alt+LMB (it rotates the view) but ctrl+LMB causes the painted area to be a different incorrect color...so painting with green as the selected color ends up as yellow instead of purple as I showed previously.