I've not done this with Oracle, but have made it work with SQL Server and I have to manually populate the Data Store name before I am able to connect.
I hope it goes better for you than it has for me because our connection is terribly slow and unstable.
Your response is a bit unclear to me. You do realize that I reply was referring to Map 3D so you could bring in and work with data there, right?
I as I said, I don't use Oracle so I can't promise this will work, but here are a couple of things to try based on the screen shots you attached.
1. Instance Name: When connecting to SQL Server based SDE data the server name needs to be included in the string and I'm making an assumption that Oracle might need it as well, which In your case it would make the string = "sderacle11g:rsgis2"
2. On the second page of your attachement you show the empty dropdown box for Data Store. Try typing the name of your Oracle database in there and see if it lets you connect afterward.
Just a "stupid" question ...
Have you installed the "required" ESRI DLLs into the bin/fdo (MAP > 2011) OR fdo/bin (MAP < 2012) folder ?
Are you using 32 bits ou 64 bits MAP ?