Maya Forum
Welcome to Autodesk’s Maya Forums. Share your knowledge, ask questions, and explore popular Maya topics.
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Any mouse drag selection causes viewport to black out (Xvnc server)

Message 1 of 3
237 Views, 2 Replies

Any mouse drag selection causes viewport to black out (Xvnc server)

Running on CentOs 8.4.2105 and the 2020 release of maya and in any viewport when I click the left mouse button to drag make a selection (where the dotted line rectangle appears) the viewport instantly goes black so I have to operate "blind". After I release the left mouse button often the viewport contents appear again and often the intended selection is highlighted. If I do it too slowly or select nothing there's a chance it will remain black... until I click on another view window. Such clicking causes the black viewport to re-render I suppose.


This is with the Viewport 2.0 renderer. I tried adding the MAYA_ENABLE_LEGACY_VIEWPORT=1 to the $HOME/maya/2020/Maya.env file and that caused the legacy renderer options to appear, but those behave exactly the same.


I've tried this on a Virtualbox OS as well as actual hardware (AMD A10-5800K cpu, GeForce GTX 970 Nvidia graphics card) and the behaviour is consistent (broken). A 2020 windows based installation on an old Dell laptop behaves as expected, the dotted line rectangle appears over whatever was present in the rendering window.


ETA - I ran a test where the real machine mentioned above (NOT the virtualbox) is connected to a normal display and I'm using a regular mouse + keyboard and the blackout problem doesn't exist. I guess I forgot to mention I was using the dedicated hardware as a headless server and I'd use vnc to connect to it, and so what's actually running on the hardware is an Xvnc instance. *That* seems to be the cause of the problem then. Might be an Xvnc configuration setting...


Thanks for any advice, suggestions, solutions...


(On behalf of my son Andy who is trying to use Maya for his 3d modeling class)


Message 2 of 3
in reply to: andyxdr2

Ok more info related to this in case anyone cares.


The Xvnc server is the tigervnc release that's standard on CentOs 8, I think it's 1.11 or so. Xvnc can be manually  started by their convenience perl script vncserver. This creates a virtual (headless) X11 server that (I think) has hardware acceleration and responds to vnc clients. Since it is not a standard X server and hasn't enjoyed massive testing compared to the standard ones, there's no reasonable expectation it'll be a perfect implementation of X11. So running maya on linux with this as the display mechanism mostly works but there is the blackout problem when you try to drag select anything making it unusable in my opinion.


I tried to rebuild tigervnc from source but the latest github release is only slightly newer than the installed one so it's unlikely this particular issue has been resolved. I don't want to become expert in the Xvnc code to fix it myself, and since the build process for tigervnc did not build Xvnc anyway (that's a while other complicated step it seems) I lost interest in pursuing that.


Background: I'm trying to enable my son to run an educational version of maya which is required for a 3D modeling class he's taking. He has an older windows 7 installation but maya won't install on that without doing an update  and I don't want to risk breaking all the existing stuff he's installed on that OS. He has a ubuntu installation on the same hardware but maya requires CentOs or RedHat to install.


So my solution was to make use of a spare computer. Long story short I installed CentOs 7 on it and the best method has been for him to use his linux desktop's X11 server supporting GLX to connect to the dedicated CentOs 7 machine running maya. This does not exhibit the blackout problem when trying to drag select a region, and performance is tolerable.


Another scheme is to then copy the CentOs7's root linux filesystem over to another machine running linux then use chroot to run maya within it. maya then thinks it's running on CentOs and has all its required support infrastructure. It's necessary to manually launch the license service before running maya. This approach is ideal because maya is then running locally and performance is optimal, as opposed to blasting massive graphical/screen update data over gigabit networking.


The only problem THERE is this doesn't work on his computer. maya.bin fails with an illegal instruction on older AMD cpu's. A8 and older Athlon II cpu's fail this way. His computer is the Athlon II. The A10 series cpu works fine, as does my Intel I7. Autodesk ought to fix that issue as their system requirements don't warn about the program not running on specific CPU's and it's trivial to reproduce.



Message 3 of 3
in reply to: andyxdr2

More info on the illegal instruction, in this case on the Athlon II cpu.

/proc/cpuinfo shows this:

model name : AMD Athlon(tm) II X2 240 Processor

stepping : 2

microcode : 0x10000c7
cpu MHz : 2800.000
cache size : 1024 KB


flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save

bugs : tlb_mmatch fxsave_leak sysret_ss_attrs


If I run gdb on the maya.bin executable I discover the illegal instruction is in


Nearby instructions:

0x7fffec284bed: add %ah,-0x70(%rsi)
0x7fffec284bf0: mov 0x636369(%rip),%rdx # 0x7fffec8baf60
0x7fffec284bf7: push %rbx
0x7fffec284bf8: lea 0x643b89(%rip),%rsi # 0x7fffec8c8788
0x7fffec284bff: mov 0x6351ca(%rip),%rax # 0x7fffec8b9dd0
0x7fffec284c06: lea 0x64b5d3(%rip),%rdi # 0x7fffec8d01e0
0x7fffec284c0d: lea 0x64b5cc(%rip),%rbx # 0x7fffec8d01e0
0x7fffec284c14: movq (%rdx),%xmm0
0x7fffec284c18: mov 0x635431(%rip),%rdx # 0x7fffec8ba050
=> 0x7fffec284c1f: pinsrq $0x1,(%rax),%xmm0
0x7fffec284c26: mov 0x6354ab(%rip),%rax # 0x7fffec8ba0d8
0x7fffec284c2d: movq (%rdx),%xmm1
0x7fffec284c31: pinsrq $0x1,(%rax),%xmm1
0x7fffec284c38: movaps %xmm0,0x64b6f1(%rip) # 0x7fffec8d0330
0x7fffec284c3f: movaps %xmm1,0x64b6da(%rip) # 0x7fffec8d0320

The line with the arrow (pinsrq) is the instruction that fails. $rax contains 0x7fffec8c8178 and is a valid memory location. xmm0 contains v2_int64 = {0x7fffec56eab6, 0x0}.

Executing the instruction on these older AMD cpu's generates an illegal instruction.



Can't find what you're looking for? Ask the community or share your knowledge.

Post to forums  

Autodesk Design & Make Report