Selection filters are important. The selection filter palette should be available in one click, not one click > flyout, especially cause its easy to "mouse-out" of the flyover zone when really moving fast . Also, most of the time you only need to select one entity and so have to select "select all" to first turn things off, then select what you want.
My suggestion is to access the whole filter list in one click (or tear off menu), hold down a modifier key and click on one item and that will automatically deselect everything else except the one item. Essentially the modifier says, "select this one only". Also, clicking on the same single item with the modifier key again will select all again...which isn't necessary with the "Select All" box of course...but just an option.