Proboolean or Pro Cutter Help?

[Anonymous]

I'll try to explain this as plainly as i can. I am using 3ds Max 2020 version on windows 10-64 bit

I've made a cube and applied some turbo smoothing and spherify filters to it to make a high polygon close to perfect sphere. Then I added some noise layers in sequence to create terrain like a planet would have, the result being a fully tessellated random generated planet.
Next step:

I then made another box but kept it relatively low-poly, Turbo smoothed it and spherified like before, only i made it just ever so slightly smaller in dimensions than the first sphere. I did not tessellate or noise generate this new sphere. I position both object to occupy the same space so that now the two spheres overlap, the result is that the tessellated noise from the first sphere poke out above the second sphere creating the illusion of landmasses and oceans. 

Now here's my problem:

Because my plan is to combine the two spheres so that i can have a single sphere with a water layer and landmass layer. For optimization purposes i want to get rid of all the left over polygons that lie below the smooth sphere from the Tessellated one. So here's where i attempt to use the Proboolean tool.

First i select both spheres and convert them to editable Polys (I have also tried to just add "Edit Poly" in the modifier menu) Then i select the Tessellated sphere and create a compound proboolean object. I select the "Subtraction" parameter and then the select objects button, and select the smooth sphere. The goal is to essentially use the smooth sphere as a delete tool to delete everything under it. The problem is, after a significant amount of load time i get an error box that says "Invalid Boolean" and then "Check operands for self intersection." and the boolean fails.

Next i tried the exact same operation but instead used the "Union" parameter to try and join the two spheres which should in theory also delete any under lying polygons and connect the surfaces. Again "Invalid Boolean" and "Check operands for self intersection.". although sometimes the operation locks up my entire computer and takes several minutes to regain any responsiveness, only after i've Alt+f4 out of 3ds Max.

I know it has something to do with the polygon count of the tessellated terrain because when i've done these exact same operations on lower polygon iterations i've been able to do it. However once you boolean an object it locks you out of being able to edit the modifiers so then i am unable to up the iterations on the turbo smoother again to increase the poly count after the boolean. 

I've vaguely read online about having to weld vertices or clean up the polygons somehow before you can boolean but also read that in some cases that is not the case. Also with a high poly terrain going through and doing these things by hand would take an eternity.

Another thing i've tried is use the procutter tool instead but the results were the same.

Any suggestions on how to fix the issue? OR any suggestions to be able to delete all interior polygons another way?

Thanks for getting through my word wall.

[10DSpace]

Hi @Anonymous 

Interesting workflow.  The problem is that the hipoly version of the mesh you created no doubt has overlapping vertices/polys and proboolean cannot handle that.  The solution (any combination of relaxing areas with overlapping verts, welding verts, using pro-optimizer, repeating relax, etc) will unfortunately likely destroy at least some of your hard work by removing surface details.  I followed your workflow to quickly reproduce the invalid boolean error message you got and then set about to fix the overlapping verts using the above combination of techniques I mentioned).  The proboolean then worked by subtracting the low poly sphere from the high poly (see image below) 

I guess the question is, "Is the cure worse than the disease?".   I do not know of a practical way around the need to make sure that the topology of your mesh is good prior to using Proboolean.  So I think that taking care to relax the areas of obvious overlap using paint deformation in vert subobject mode on your hipoly model, while tedious, may actually be the most direct way to get what you want.   At least if you want to stay within Max entirely.  Of course if you have a sculpting program like Zbrush you could export the hi poly version, tweak it there, use dynamesh and zremesher to clean the topology and then send it back to Max.  There are likely similar ways to do this with Mudbox and maybe 3DCoat as well, I am just not familiar with them.   Anyway I hope this helps in some way.

[Anonymous]

Thabks for taking the time to help! I can see that you got it to work in your screenshot however I suppose I'm just not that familiar with some of these tools yet. Relax? And the process of welding verticies and what that does. It's hard to predict what the noise filters will do as they undoubtly overlap. The construction I ended up with was something like

Noise5

Noise4

Noise3

Turbosmooth

Noise2

Noise1

Noise0

Spherify

Some fractal some not. Also I've been trying to import high poly models like this into unreal engine and sometimes that doesn't work too. Interesting problem to work around..

[10DSpace]

Yeah with a stack of modifiers like that you most likely have a lot of vertices on top of each other and that creates unnecessary problems throughout your pipeline (in addition to Pro-boolean issues) like when you try to UVW unwrap or export to another program like Ureal.    Basically, most 3D Programs want good topology which means a single surface with only single vertices on the topological surface.  No vertices underneath other vertices or crumpled up sections with multiple vertices in a chaotic bunch (they wouldn't be visible in renders anyway).  The vertices can be very close together so that you can achieve height variations like you are trying to do; just not on top of each other.  

Relax is a very useful tool within Edit Poly.  (see below)

Basically, in your case, it could help correct your problems by "relaxing" or smoothing out the vertices so that they do not overlap but are on that ideal topological surface.  The trick is to not smooth it out so much that you loose all of the height variation you want. So you adjust the brush size and the strength setting and then paint on the surface of your mesh with the brush.  If you switch to the "push/pull option to the left of "Relax" in the Paint Deformation section you can reintroduce height variations in any section of the mesh you want, again by tweaking the brush size and strength settings.  Give this a shot.  Here is a link to the relevant Max docs:

https://knowledge.autodesk.com/support/3ds-max/learn-explore/caas/CloudHelp/cloudhelp/2016/ENU/3DSMa... 

Hope this helps.