Input Handling: several issues

[slice3d]

Hi guys!

I have several questions regarding input handling between Unity & Scaleform.

1. Modifier keys (Alt, Control, Shift, etc) not passed from Unity to SF with mouse events. Why? How should I get keyboard modifiers in this case:

stage.addEventListener(MouseEvent.MOUSE_UP, handleMouseUpEvent, false, 0, true);
protected static function handleMouseUpEvent(event:MouseEvent):void
{
trace(event.altKey); // False when alt key is pressed!
}

2. How should I figure out in Unity that a specific mouse/keyboard input events have been handled in Scaleform?
Seems like the infrastructure for this is prepared:
bool SFManager.HandleMouseMoveEvent <-> bool SFMovie.HandleMouseEvent <-> bool SFMovie_Imports.SF_HandleMouseEvent <-> Scaleform Runtime
bool SFManager.HandleMouseEvent <-> bool SFMovie.HandleMouseEvent <-> bool SFMovie_Imports.SF_HandleMouseEvent <-> Sceleform Runtime
bool SFManager.HandleKeyDownEvent <-> bool SFMovie.HandleKeyEvent <-> bool SFMovie_Imports.SF_HandleKeyEvent <-> Scaleform Runtime
But the Scaleform Runtime always returns false to SFMovie_Imports.SF_HandleMouseEvent and SFMovie_Imports.SF_HandleKeyEvent methods no matter what I do with InputEvent / MouseEvent in ActionScript (setting event.handled to true doesn't make any sense).

Thanks,

Vyacheslav.

[slice3d]

Bump.

Still haven't figured out which value is returned from ActionScript as bool from the Scaleform Runtime.

[Raja85]

Hey even I'm facing the same issue. I have an input texfield in flash, but in Unity the arrow keys are not detected to move the cursor position. If you have found any solution please share it with us.

Thank you.

[slice3d]

Nope, I haven't figured out in which case SF_HandleMouseEvent or SF_HandleKeyEvent will return true from the Scaleform Runtime.
Still hope we'll be able to get an official answer from the Scaleform team.

But at least I've found some information inside of 'sf_4.2_integration_tutorial.pdf' document:
4.3.3 Hit Testing
4.3.4 Keyboard Focus -> 6.1.2 ExternalInterface
4.3.5 Touch Events

So they suggest us to use simple hit-testing (SFManager.DoHitTest(float x, float y)) to catch mouse events and to make an

ExternalInterface.call("Inform_Unity_That_Keyboard_Input_Should_NOT_Be_Processed") as soon as any TextField in SF has received focus and 

ExternalInterface.call("Inform_Unity_That_Keyboard_Input_Should_Be_Processed") when TextField looses it's focus.

Though, in my opinion, it is not the best way to cover all cases (for example - I wish to have multiple DisplayObjects on screen which should not catch mouse events based on some conditions). But it's the only option we currently have considering the limitations of simultaneous (executed in one frame) ExternalInterface calls.

[slice3d]

Okay, now in December SDK "handled" always returns "true" instead of "false" from the SF runtime for any input events... What's the point in this? Am I missing something?

Anybody out there from the Scaleform team?

Can you provide an example on how you propose us to handle Input between Scaleform and Unity in something close to an RPG game with a lots of floating windows, chat text fields, and a character which can simply move with WSAD keys and shoot with mouse?

Or at least suggest an idea on how it should work - I am able to implement this by myself if this idea will be practically realizable.

IMHO, SFMovie.DoHitTest() is almost useless in this case, and as I see it, implementing custom workarounds will require using a lots of ExternalInterface / Invoke calls, what is not desirable due to this issue.

To be more specific, I'll give an example case where using proposed DoHitTest method fails:

1. We have a UI window MovieClip on which mouse input should be intercepted.

2. Below the UI window we have several MovieClips floating over the whole screen - healthbars, for example. Mouse input should not be intercepted by these MovieClips.

3. Using DoHitTest() will return true if mouse will be over any MovieClip what will lead to 'healthbars intercepting mouse input'.

What should I do in this case?

Thanks for your attention!

[slice3d]

Well, I've solved all my issues with input handling.

Assets\Scaleform\Doc\sf_4.2_AS3_extensions.pdf document was really helpful.

The final recipe of complete input handling system in my game project is:

1 Custom C# code for redirecting input data between UI and Game.
    1.1 In C#, use SFManager.DoHitTest(float x, float y) to check if a specific screen point contains InteractiveObject which should intercept mouse events.
    For some reason, in the latest Scaleform SDK this function became private, but I think you can make it public back without any problems.
    But note that you should offset incoming y value by 24px if in Unity Editor before passing mouse coordinates to this method:

#if UNITY_EDITOR
    bool mouseIsOverUIElement = SFMgr.DoHitTest(mousePos.x, mousePos.y + 24);
#else
    bool mouseIsOverUIElement = SFMgr.DoHitTest(mousePos.x, mousePos.y);
#endif

    1.2 In ActionScript, use InteractiveObjectEx.setHitTestDisable(o:InteractiveObject, true) on InteractiveObject's which shouldn't take part in hit testing.

    This solved the case described in my previous post.
2 Custom wrappers for a standard CLIK TextArea and TextInput components.
    These wrappers intended to notify Unity as soon as any textField receives focus, so the Scaleform should intercept any Keyboard input.
    Just some draft code to get the idea:

override protected function changeFocus():void 
{
    if (this.hasFocus)
    {
        // Notify Unity that keyboard input should not be processed in game
        ExternalInterface.call("someFunc");
    }
    else
    {
        // Notify Unity that keyboard input should be processed by game
        ExternalInterface.call("someOtherFunc");
    }
}

3 To enable support of function keys handling in TextInput or TextArea, in the SFKey.CreateKeyDictionary() method add these lines:

SFKeyDictionary.Add(UnityEngine.KeyCode.LeftArrow, Code.Left);
SFKeyDictionary.Add(UnityEngine.KeyCode.RightArrow, Code.Right);
SFKeyDictionary.Add(UnityEngine.KeyCode.UpArrow, Code.Up);
SFKeyDictionary.Add(UnityEngine.KeyCode.DownArrow, Code.Down);
//... Any other keys you wish to be passed to the Scaleform

Hope this information will be useful for someone like me. 🙂

[slice3d]

Only one problem have remained, and I think this should be fixed by the Scaleform team:

---

@slice3d wrote:

1. Modifier keys (Alt, Control, Shift, etc) not passed from Unity to SF with mouse events. Why? How should I get keyboard modifiers in this case:

stage.addEventListener(MouseEvent.MOUSE_UP, handleMouseUpEvent, false, 0, true);
protected static function handleMouseUpEvent(event:MouseEvent):void
{
trace(event.altKey); // False when alt key is pressed!
}

---

[Cleroth]

It's nice to see how much they care about their customers.

[adam.petrone]

Hello,

We are in the process of adding Modifier key support from Unity.

[diogo.vasconcelos]

Ive only seen this topic now.
Ive solved the key moddifier problem (need to get the Shift Key), using the Input.GetKeyDown() / Input.GetKeyUp() from the UnityEngine.

The problem is in Unity (http://forum.unity3d.com/threads/30343-current-event-not-detecting-shift-key) because in the OnGUI event handling, it does not catch a moddifier key (I think, I dont remember very well as I have bump into this problem 1 year ago).

The solution is to use the Input.GetKeyDown/Up in the Update method of you SFCamera script.

Here what I did.

new public void Update()

{

     CheckShiftKey();
}

protected void CheckShiftKey()
{
if (Input.GetKeyDown(KeyCode.LeftShift))
{
SFMgr.HandleKeyDownEvent(SFKey.Code.Shift, SFKeyModifiers.Modifiers.Key_ShiftPressed, 0);
}

if (Input.GetKeyDown(KeyCode.RightShift))
{
SFMgr.HandleKeyDownEvent(SFKey.Code.Shift, SFKeyModifiers.Modifiers.Key_ShiftPressed, 0);
}

if (Input.GetKeyUp(KeyCode.LeftShift))
{
SFMgr.HandleKeyUpEvent(SFKey.Code.Shift, SFKeyModifiers.Modifiers.Key_ShiftPressed, 0);
}

if (Input.GetKeyUp(KeyCode.RightShift))
{
SFMgr.HandleKeyUpEvent(SFKey.Code.Shift, SFKeyModifiers.Modifiers.Key_ShiftPressed, 0);
}
}

Hope it helps

Regards,

Diogo

[slice3d]

adam.petrone,
Great news, Adam!
And please don't forget about the scroll wheel as well - it shouldn't take too much to implement.

diogo.vasconcelos,
Yes, I've totally rewritten Input handling system for the Scaleform interaction in my project, and now it doesn't rely on Unity GUI input events at all.
Here's how it works:
1. Get user input data
2. Process this data in UI (Scaleform)
3. Process input data not catched by the UI in game (move camera, move player, fire, etc.)
4. Update UI in LateUpdate() (make all Camera.WorldToScreenPoint(someTransformLinkedWithUIElement.position) here to determine UI elements screen positions and then - perform SFCamera.Update())
5. Render frame and Display UI

By doing it this way I don't get this annoying 1-frame UI update delay between mouse/camera movement and UI reaction.
This is a must-have if you plan to use info billboards (titles, healthbars) on top of your characters in your game for example.

Now to modifier keys handling...
Currently I'm using similar approach to yours, but the problem with this workaround is that it will not work in some cases.
For example, if I'll use it for the Alt key handling - you will never get the KeyUp event when you press the Alt+TAB shortcut and then get back to your game - Alt key will remain pressed for the Scaleform.

Of course even this case can be resolved by introducing the altKeyIsDown flag and constantly checking for the GetKey(Keycode.Alt), but it is also some sort of a workaround, and not feels like the 'right' solution.

[diogo.vasconcelos]

Great work slice3d.

As I said, I did that one year ago, dont remeber very well everything, but in my case I needed to get the Shift key and that was the way I used, good to see that you found a solution for yours (could be handy for me in the future).

I also did the LateUpdate trick, because like you, I had some UI being positioned on a 3d object (like a healthbar) and that 1 frame delay was terrible.

[am964]

Great work Slice3D and thank you for being such a valued contributer!

[slice3d]

Not a problem, Ankur!
Thanks to you for making Scaleform available for Unity developers! 🙂

[beneton]

Hello slide3d, this solution does not work for me and I can't figure why.

I have many buttons on my screen and for some of them SFMgr.DoHitTest (Input.mousePosition.x, Input.mousePosition.y, Scaleform.HitTestType.HitTest_ShapesNoInvisible) returns TRUE, and for some it returns FALSE.

There is nothing different in them.

Actually I have a background image, that the upper part returns TRUE, and the bottom part returns FALSE.

Have came across that issue?

I also tried using all different types of HitTestType, and the issue persists.

Thanks,

[slice3d]

Hello beneton, 

Probably, there is something wrong with your setup.

Do you have an example project which you can send me to checkout?

[RichMC]

You folks seem to be on top of this one. If there is anything I can do to assist in your efforts, PM me.

[beneton]

Hello slice3d, as I'm in a tidy schedule I had to improvise a solution for this game, it's not near what I wanted but it works for now.

With that in mind, I don't have an isolated test case. But I can try making one next month.

Is there anything that I can make in an SWF file that will make SFMgr.DoHitTest not work in a part of an sprite?

That question is more for RichMC I think.