I did that in C++, the routine I used is Delaunator https://github.com/mapbox/delaunator.
Not a port, but I made a wrapper for Autolisp, https://github.com/CEXT-Dan/ads_geo
Here’s a detailed guide https://mapbox.github.io/delaunator/
It’s incredibly fast, basically you feed in a pointset, and it returns two lists indexing out triangles and half edges. The half edges are the matching segments between two triangles.
What I did is iterate the half edges, find all the half edge pairs that have a -1 I.e. (-1 22) , then you know triangle with edge 22 is outside. Check the angles in triangle that contains edge 22, if it’s below a certain threshold, delete it.
Now you have a structure where you can iterate the outside edges, connect them to create the boundary polyline
The same wrapper contains a fast KD-Tree and a std::unordered_map<AcGePoint3d,VALUE> I created a custom hash for Points. Armed with these tools, another approach might be to investigate this paper. Concave hull: A k-nearest neighbors approach for the computation of the region occupied by a set of points. https://www.scitepress.org/papers/2007/20808/20808.pdf
Python for AutoCAD, Python wrappers for ARX https://github.com/CEXT-Dan/PyRx