I'm sure there are several ways to do this but here's something that I was able to get to work that I'm guessing will fit into what you already have fairly easily. VBA has some quirks about it that make this a bit harder than it seems like it should be.
The whole idea of a modeless dialog is that you display it and it returns control back to the function that displayed it so it can continue to run while the modeless dialog does its thing. You want a modeless dialog that doesn't have modeless dialog behavior. A typical approach is to use events. You connect to an event, start something, and then wait for an event to know that it has finished. That's the approach I chose to take, but a VBA limitation is that you can't use events in a standard code module so I had to create a class module to listen for the event. That actually ends up working good in this case because I can have the function in the class just sit and wait for the form to be dismissed and then return so from your module where you need to display the form and get values back, it's a single function call that returns the form values.
Here are the various bits of code. First, here's the macro code in the code module that you execute to start the whole thing. This creates a new instance of the clsModelessFormTest class and then calls its GetCheckedState sub which returns a single argument indicating if the check box on the dialog was checked or not. It could return multiple arguments to pass whatever information you need.
Public Sub TestModelessForm()
Dim testClass As New clsModelessFormTest
Dim state As Boolean
Call testClass.GetCheckedState(state)
MsgBox "Checked state is: " & state
End Sub
Here's the code of the clsModelessFormTest class. I declares a variable as frmModelessTest, which is my form and is described below. It declares it using the WithEvents keyword so that is can receive the FormFinished event from the form. It also has a couple of other variables that are global within the class. The GetCheckedState sub is the sub that was called above. It creates a new form and displayes it modelessly. Control is returned back to this sub but it enters a while loop and waits for the m_finished variable to be True. It spins in the while loop calling the DoEvents function. This is a VBA function that turns control back over to windows to handle any pending operations. When the user clicks OK button in the form, it gets the FormFinished event and sets the variable with the value returned and sets the m_finished variable to True indicating that it's done. This breaks out of the while loop and returns from the sub, returning control back to the calling function.
Private WithEvents m_formTest As frmModelessTest
Private m_finished As Boolean
Private m_checkedState As Boolean
Private Sub Class_Initialize()
m_finished = False
End Sub
Public Sub GetCheckedState(ByRef checkedState As Boolean)
Set m_formTest = New frmModelessTest
m_formTest.Show vbModeless
Do While Not m_finished
DoEvents
Loop
checkedState = m_checkedState
End Sub
Private Sub m_formTest_FormFinished(isChecked As Boolean)
m_checkedState = isChecked
m_finished = True
End Sub
Finally, here's the format I created called frmModelessTest. It contains a single check box named chkTest and a button named cmdOk. Here's the form code. Notice at the top that it's declaring an event called FormFinished and has a single argument called isChecked. You can define any number of arguments to pass back whatever information from the form that you need. When the user clicks the OK button it just gathers up the information from the form and uses the RaiseEvent statement to cause the FormFinished event to be fired and then it unloads the form.
Public Event FormFinished(isChecked As Boolean)
Private Sub cmdOk_Click()
Dim isChecked As Boolean
If Me.chkTest.value = True Then
isChecked = True
Else
isChecked = False
End If
RaiseEvent FormFinished(isChecked)
Unload Me
End Sub
I realized this seems like a lot, and as I said before it's definitely not as simple as it seems like it should be, but if you think through what it's doing it makes some sense. Good luck.