Error when local server is created for Reinforced Learning

[FelipeCapalbo]

[ FlexSim 23.2.2 ]

I tried to create a reinforcement learning model to optimize a layout by changing the object's position on the X and Y axes. The observation field references the object's position in the tree, as does the action field, but this can change its value.

When I try to implement the trained model, this error shows up:

"Exception occurred during processing of request from ('127.0.0.1', 50682)
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 11, in do_GET
    self._handle_reply(params)
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 30, in _handle_reply
    action, _states = FlexSimInferenceServer.model.predict(observation)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\base_class.py", line 553, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 363, in predict
    obs_tensor, vectorized_env = self.obs_to_tensor(observation)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 270, in obs_to_tensor
    vectorized_env = is_vectorized_observation(observation, self.observation_space)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 399, in is_vectorized_observation
    return is_vec_obs_func(observation, observation_space)  # type: ignore[operator]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 307, in is_vectorized_multidiscrete_observation
    raise ValueError(
ValueError: Error: Unexpected observation shape () for MultiDiscrete environment, please use (22,) or (n_env, 22) for the observation shape."

ExercicioLayout.fsm

flexsim_env.py

flexsim_inference.py

flexsim_training.py

TreinamentoExercicioLayout.zip

[kavika_faleumu]

Hey @Felipe Capalbo, the error hints that the observation you pass into

action, _states = FlexSimInferenceServer.model.predict(observation)

isn't presented in the right format. The observation space is requesting a (22,) or (n_env, 22) shape. Can you see what kind of shape is getting passed in before the error?

[FelipeCapalbo]

What does this "shape" means? The observations has 22 parameter, as does the actions.

All of them are discrete values.
In the Reinforcement Learning window it is inputed as multidiscrete, because of the 22 parameters.

[kavika_faleumu]

@Felipe Capalbo, here's documentation on the numpy.ndarray.shape property. The example in the documentation shows the following:

>>> x = np.array([1, 2, 3, 4])
>>> x.shape
(4,)

With an array size of 4, x.shape returns (4,). This format matches what the above error is looking for - a shape of (22,). What it says you're providing is (), which seems to be an empty array or just no array at all.

I'm able to train your model, but I'm not sure how you've setup the flexsim-interface.py > main() to work. I can't help you further unless you give me detailed steps on how to send input or results from the RL model to the server it sets up.

[FelipeCapalbo]

By the time, I've been using all the default settings the FlexSim Reinforcement Learning tutorial provide me (the files were uploaded in the main question). I was expecting the trained model to give me a solution on what is the layout that maximizes the model reward (which I think it would be all the objects overlaped together).
Regarding to "flexsim-interface.py > main()", I didn't set it up.

[kavika_faleumu]

@Felipe Capalbo, my mistake about the interface > main(). What are the steps you're taking to get the errors in your original post? What do you mean by "When I try to implement the trained model..."?

[FelipeCapalbo]

Just after finishing training the model, a zip file is generated. When I execute the "flexsim-inference.py", to use the trained model, the error in the original post appears.

[kavika_faleumu]

I can't replicate the errors you're getting. I'm sorry. I trained the model and ran flexsim-inference.py, but I didn't get any errors.

[FelipeCapalbo]

Okay, thank you for your help. I'll see if that's anything wrong with my installation.

[FelipeCapalbo]

I was running the code again and realize that the error appears when you execute "flexsim-inference.py" and open "http://localhost/?observation=3" in the browser tab.

Did you get to do this?

[kavika_faleumu]

Okay, I was able to replicate your error. I then experimented with doing

http://localhost:8080/?observation=[3]

(I changed my port to 8080). I changed the observation to being in an array. It then gave me an error stating that I gave it (1,) and it was looking for (22,). So, based on my assumption, you need to supply an array of 22 items for observation in order for it to run. Those 22 values must be matched up to the 22 parameters in your model.

[jason_lightfoot_adsk]

Hi @Felipe Capalbo,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

[FelipeCapalbo]

Couldn't figure out yet, but I thing the concept used to define the observations could be wrong.

[philboboADSK]

The example is training an AI that takes 1 discrete observation value and makes 1 discrete action value. You've changed your model to take multiple values for the observation, not just 1 value.

So passing just a 3 as an observation doesn't make sense. You need to pass 22 numbers.

That step in the tutorial expects that you were following the rest of the tutorial up to that point, not changing it. Since you are doing something different, the tutorial isn't going to match up exactly the same. Just skip that step (which is just testing if your web service is responding to queries) and move on to the next step.

[FelipeCapalbo]

So im not supose to set an multidiscrete observation field and a discrete action field, for example?

[philboboADSK]

Having a multi-discrete observation space and a discrete action space is fine.

Your problem is that you have a multi-discrete observation space and are passing a discrete observation. The action space is irrelevant.

If you have a multi-discrete observation space, then you need to pass multiple values for the observation in order to receive an action.

You cannot pass a discrete observation if you are using a multi-discrete observation space.

[jason_lightfoot_adsk]

Hi @Felipe Capalbo,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.