I relaize this is a long time after the initial post. I went through a similar trial with rotating and translating at the same time with TransformBy and maybe can save somebody else all the trial and error time I spent. These are all my observations, some may be wrong, and some may be things other folks already know. Here goes.
Rotations (the upper left 9 elements of the matrix) are done first and are done about 0,0,0. Translations (the far right, upper 3 elements of the matrix) are made after this.
(Also, just FYI - The square root of the sum of the squares of the top row, left 3 elements is the X scale. The square root of the sum of the squares of the 2nd row, left 3 elements is the Y scale. The square root of the sum of the squares of the 3rd row, left 3 elements is the Z scale.)
The trick for me was I had to figure out where my reference point was after the rotation to be able to know how much to translate. Below is the experimental routine that worked for me.
Sub Dwg_Format_Matrix()
'
' Routine to Rotate & Move Objects by TransformBy Method
'
Dim AcD As AcadDocument, AC_Obj As AcadEntity, BkAng, RefWant(0 To 2) As Double
Dim AngWant, DegPi, InsPt(0 To 2) As Double
Set AcD = ThisDrawing
DegPi = 180 / 3.14159265358979 'Conversion factor between degrees and PI
InsPt(0) = 2 'Reference Point, X
InsPt(1) = -1 'Reference Point, Y
InsPt(2) = 0 'Reference Point, Z
BkAng = 145 / DegPi 'The initial rotation of the object in the XY plane
AngWant = 0 'The desired rotation of the object in the XY plane
'RefWant is the desired location of the reference point
RefWant(0) = 0: RefWant(1) = 0: RefWant(2) = 0
'Get Transform Matrix components
Dim XF(0 To 3, 0 To 3) As Double, V_Len, V_Ang
'V_Len is the length of the vector from 0,0,0 to the reference point.
V_Len = Sqr(InsPt(0) ^ 2 + InsPt(1) ^ 2)
'V_Ang is the initial angle in the XY plane of the vector from 0,0,0 to the reference point.
V_Ang = Atn(InsPt(1) / InsPt(0))
If InsPt(0) < 0 Then V_Ang = V_Ang + 180 / DegPi
'BkAng is now the angle to rotate objects
BkAng = AngWant - BkAng
'V_Ang + BkAng is the final angle in the XY plane of the vector from 0,0,0 to the reference point.
V_Ang = V_Ang + BkAng
XF(0, 0) = Cos(BkAng): XF(0, 1) = -1 * Sin(BkAng): XF(0, 2) = 0: XF(0, 3) = RefWant(0) - V_Len * Cos(V_Ang)
XF(1, 0) = Sin(BkAng): XF(1, 1) = Cos(BkAng): XF(1, 2) = 0: XF(1, 3) = RefWant(1) - V_Len * Sin(V_Ang)
XF(2, 0) = 0: XF(2, 1) = 0: XF(2, 2) = 1: XF(2, 3) = 0
XF(3, 0) = 0: XF(3, 1) = 0: XF(3, 2) = 0: XF(3, 3) = 1
' Transform the objects
For Each AC_Obj In AcD.ModelSpace
AC_Obj.TransformBy (XF)
Next
End Sub
