When putting a hole annotation, you select the hole then LMB at the note location and viola, the annotation is in place
When putting a leader text annotation, BOM balloon, or weld symbol, you LMB the edge to attach it to then LMB to locate it then RMB to accept it.
The Hole annotation requires the least effort and should be the standard.
You could add functionality to add jogs and additional leaders as necessary.