topic Re: Error when local server is created for Reinforced Learning in FlexSim Forum

Error when local server is created for Reinforced Learning

FelipeCapalbo — Fri, 01 Dec 2023 15:37:18 GMT

[ FlexSim 23.2.2 ]

I tried to create a reinforcement learning model to optimize a layout by changing the object's position on the X and Y axes. The observation field references the object's position in the tree, as does the action field, but this can change its value.

When I try to implement the trained model, this error shows up:

"Exception occurred during processing of request from ('127.0.0.1', 50682)
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 11, in do_GET
    self._handle_reply(params)
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 30, in _handle_reply
    action, _states = FlexSimInferenceServer.model.predict(observation)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\base_class.py", line 553, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 363, in predict
    obs_tensor, vectorized_env = self.obs_to_tensor(observation)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 270, in obs_to_tensor
    vectorized_env = is_vectorized_observation(observation, self.observation_space)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 399, in is_vectorized_observation
    return is_vec_obs_func(observation, observation_space)  # type: ignore[operator]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 307, in is_vectorized_multidiscrete_observation
    raise ValueError(
ValueError: Error: Unexpected observation shape () for MultiDiscrete environment, please use (22,) or (n_env, 22) for the observation shape."

TreinamentoExercicioLayout.zip

Re: Error when local server is created for Reinforced Learning

kavika_faleumu — Fri, 01 Dec 2023 19:54:29 GMT

Hey @Felipe Capalbo, the error hints that the observation you pass into

action, _states = FlexSimInferenceServer.model.predict(observation)

isn't presented in the right format. The observation space is requesting a (22,) or (n_env, 22) shape. Can you see what kind of shape is getting passed in before the error?

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Sat, 02 Dec 2023 17:58:56 GMT

What does this "shape" means? The observations has 22 parameter, as does the actions.

All of them are discrete values.
In the Reinforcement Learning window it is inputed as multidiscrete, because of the 22 parameters.

Re: Error when local server is created for Reinforced Learning

kavika_faleumu — Mon, 04 Dec 2023 16:43:15 GMT

@Felipe Capalbo, here's documentation on the numpy.ndarray.shape property. The example in the documentation shows the following:

>>> x = np.array([1, 2, 3, 4])
>>> x.shape
(4,)

With an array size of 4, x.shape returns (4,). This format matches what the above error is looking for - a shape of (22,). What it says you're providing is (), which seems to be an empty array or just no array at all.

I'm able to train your model, but I'm not sure how you've setup the flexsim-interface.py > main() to work. I can't help you further unless you give me detailed steps on how to send input or results from the RL model to the server it sets up.

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Mon, 04 Dec 2023 18:17:25 GMT

By the time, I've been using all the default settings the FlexSim Reinforcement Learning tutorial provide me (the files were uploaded in the main question). I was expecting the trained model to give me a solution on what is the layout that maximizes the model reward (which I think it would be all the objects overlaped together).
Regarding to "flexsim-interface.py > main()", I didn't set it up.

Re: Error when local server is created for Reinforced Learning

kavika_faleumu — Mon, 04 Dec 2023 18:35:09 GMT

@Felipe Capalbo, my mistake about the interface > main(). What are the steps you're taking to get the errors in your original post? What do you mean by "When I try to implement the trained model..."?

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Mon, 04 Dec 2023 18:46:24 GMT

Just after finishing training the model, a zip file is generated. When I execute the "flexsim-inference.py", to use the trained model, the error in the original post appears.

Re: Error when local server is created for Reinforced Learning

kavika_faleumu — Mon, 04 Dec 2023 21:16:22 GMT

I can't replicate the errors you're getting. I'm sorry. I trained the model and ran flexsim-inference.py, but I didn't get any errors.

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Tue, 05 Dec 2023 11:37:06 GMT

Okay, thank you for your help. I'll see if that's anything wrong with my installation.

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Wed, 06 Dec 2023 14:35:14 GMT

I was running the code again and realize that the error appears when you execute "flexsim-inference.py" and open "http://localhost/?observation=3" in the browser tab.

Did you get to do this?

Re: Error when local server is created for Reinforced Learning

kavika_faleumu — Wed, 06 Dec 2023 21:44:19 GMT

Okay, I was able to replicate your error. I then experimented with doing

http://localhost:8080/?observation=[3]

(I changed my port to 8080). I changed the observation to being in an array. It then gave me an error stating that I gave it (1,) and it was looking for (22,). So, based on my assumption, you need to supply an array of 22 items for observation in order for it to run. Those 22 values must be matched up to the 22 parameters in your model.

Re: Error when local server is created for Reinforced Learning

jason_lightfoot_adsk — Thu, 14 Dec 2023 16:45:15 GMT

Hi @Felipe Capalbo,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Fri, 15 Dec 2023 00:04:55 GMT

Couldn't figure out yet, but I thing the concept used to define the observations could be wrong.

Re: Error when local server is created for Reinforced Learning

philboboADSK — Fri, 15 Dec 2023 22:16:26 GMT

The example is training an AI that takes 1 discrete observation value and makes 1 discrete action value. You've changed your model to take multiple values for the observation, not just 1 value.

So passing just a 3 as an observation doesn't make sense. You need to pass 22 numbers.

That step in the tutorial expects that you were following the rest of the tutorial up to that point, not changing it. Since you are doing something different, the tutorial isn't going to match up exactly the same. Just skip that step (which is just testing if your web service is responding to queries) and move on to the next step.

Re: Error when local server is created for Reinforced Learning

FelipeCapalbo — Thu, 25 Jan 2024 12:33:58 GMT

So im not supose to set an multidiscrete observation field and a discrete action field, for example?

Re: Error when local server is created for Reinforced Learning

philboboADSK — Thu, 25 Jan 2024 15:12:29 GMT

Having a multi-discrete observation space and a discrete action space is fine.

Your problem is that you have a multi-discrete observation space and are passing a discrete observation. The action space is irrelevant.

If you have a multi-discrete observation space, then you need to pass multiple values for the observation in order to receive an action.

You cannot pass a discrete observation if you are using a multi-discrete observation space.

Re: Error when local server is created for Reinforced Learning

jason_lightfoot_adsk — Wed, 07 Feb 2024 14:08:06 GMT

Hi @Felipe Capalbo,