Hi @Anonymous
The invalid boolean message is often the result of using a mesh as an operand in the boolean operation which has overlapping vertices or polygons. You did not show the mesh that is generating the error, but since you describe it as "complex" it very likely has some overlapping areas that Proboolean cannot handle. This has been discussed here:
https://forums.autodesk.com/t5/3ds-max-forum/proboolean-issue-invalid-operation/td-p/7046197
The solution is to clean up the mesh by removing any overlapping regions. Think of the ideal boolean operand as a solid object with a single continuous surface without any vertices or polygons occupying the same space since the overlapping geometry confuses Proboolean as to which surface to use in the boolean operation. Hope this helps.
