Select the edges you want to remove and use bacKspace key the Shortcut) or select "Remove" in the editable poly rollout. Remove does what you want while delete as you have seen is more destructive to surrounding area. Hope this helps.
FYI, There is also a circumstance in which you want to remove edges connecting 2 other edges and in that case, the backspace key shortcut will remove the edges but not their vertices. If you want to remove both the edges and their vertices, then use Ctrl + Backspace.
