Not a trivial problem.
As Manish suggests, you can easily determine whether tags overlap.
https://stackoverflow.com/questions/1119451/how-to-tell-if-a-line-intersects-a-polygon-in-c
To determine an alternative arrangement, you will have to move them around until they do not overlap.
Once you have that sorted out, you will possibly not be happy with the result, and the next problem will be how to distribute them nicely.
The last step sound like machine learning to me...
Here is a powerful computational geometry algorithms library:
https://www.cgal.org/
Here is a simpler polygon clipper:
http://angusj.com/delphi/clipper.php
You might want to research what algorithms people use to place labels on maps... they face the same challenge.
Here is a solution that places labels avoiding overlaps using either a greedy or simulated annealing strategy:
https://github.com/d3fc/d3fc-label-layout
Looking forward to seeing how you end up solving this!
Thank you!
Cheers,
Jeremy