Within the callback function set by AcDbGripData::setWorldDraw you can draw the glyph for a custom grip.
If you do not set colors the default grip colors set at the AutoCAD's options dialog are used except GRIPCONTOUR color.
For example lets say I draw a small filled rectangle for the grip. I first use
How can you use GRIPCONTOUR color for the boundary line of the grip?
Not possible the polyline will also be drawn with GRIPCOLOR not GRIPCONTOUR.
You need access to the value of GRIPCONTOUR set it with pWd->subEntityTraits().setColor(GRIPCONTOUR VALUE); and then draw the polyline or polygon as you suggested a second time.
But how do you get access to the GRIPCONTOUR value from within worlddraw callback function?
Yes, you must set color and other properties before using polyline method.
I do not use GRIPCONTOUR color, I draw with my own value. You can prepare the value of GRIPCONTOUR system variable beforehand and store it in some global variable if acedGetVar is impossible inside grip callback function.