topic Re: Setting a parameter when resetting the environment in Reinforcement Learning in FlexSim Forum

Setting a parameter when resetting the environment in Reinforcement Learning

arthur_ml — Thu, 16 Mar 2023 08:25:54 GMT

[ FlexSim 22.2.1 ]

Hi everyone,

I am doing Meta Reinforcement Learning for production scheduling. Therefore, I predefined several job lists (n=50) that should be scheduled optimally with RL. In MetaRL, the algorithm regards each list as "task" and the algorithm should be able to select a specific task (i.e. 1 list of the 50 lists) when resetting the environment. I want realize that, by changing the value of a global parameter, which then leads to the usage of the specific list. I.e.: episode is finished -> MetaRL selects. e.g. task=14 for the next episode -> parameter is set to 14 -> flexsim is resetted and job list 14 is used in the model now.

Currently, my code for resetting flexsim looks like this:

def _reset_flexsim(self):
    if self.verbose:
        print("Sending Reset message")
    resetString = "Reset?"
    if hasattr(self, "seedNum"):
        resetString = "Reset:" + str(self.seedNum) + "?"
    self._socket_send(resetString.encode())

Can anyone help me please how I can set the parameter value when resetting flexsim?

Thanks in advance!

Re: Setting a parameter when resetting the environment in Reinforcement Learning

kavika_faleumu — Thu, 16 Mar 2023 21:48:57 GMT

Hey @Arthur Ml, one way you can set the parameter's value on model reset is using the "OnModelReset" trigger found in the Toolbox:

This gives you a code editor that will execute when the model is about to Reset. Here you can grab a reference to the Parameter's value cell and set it with the task for the next run. In my example I simply set it with the output count of a Processor.

set-param-on-reset.fsm

Hope this helps!

Re: Setting a parameter when resetting the environment in Reinforcement Learning

arthur_ml — Fri, 17 Mar 2023 07:29:15 GMT

Dear @Kavika F ,

thanks for the quick response.

Sorry, my question was not specific enough. How can I set a parameter value from Python, when resetting the model/ RL environment? I use a socket connection for connecting Python with Flexsim as presented in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html). So far, with this socket connection, I can launch, reset, and close Flexsim, and I can send actions and receive observation from the model. But how can I set a specific parameter value when resetting the model?

Thanks and best regards,
Arthur

Re: Setting a parameter when resetting the environment in Reinforcement Learning

kavika_faleumu — Sat, 18 Mar 2023 00:02:45 GMT

Oh okay. Once you reset your model, you can use the following command to set a parameter:

controller.setParameter("ParameterName", value)

If you look at the FlexSimPy.cpp, you can see a list of commands you can use from the controller (and within the FlexSim):

Re: Setting a parameter when resetting the environment in Reinforcement Learning

arthur_ml — Mon, 20 Mar 2023 06:48:04 GMT

Is there a way to set a Parameter in Flexsim with the help of the "flexsim_env.py" script you provided in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html)? Because I am using the scripts from the documentation to connect to Flexsim and do the training.

Re: Setting a parameter when resetting the environment in Reinforcement Learning

kavika_faleumu — Mon, 20 Mar 2023 17:27:28 GMT

Sorry I thought you were using the code from the FlexSimPy module on github used to connect to FlexSim externally. That is where I got my code from and reference the FlexSimPy.cpp. I don't know if there's a way to use the flexsim_env.py to change a model parameter; I think the socket connections it deals with are specific to the RL Tool in FlexSim. However, if you use the FlexSimPy module (linked above) to control FlexSim externally, it gives you the capability to set the parameter.

Re: Setting a parameter when resetting the environment in Reinforcement Learning

arthur_ml — Tue, 21 Mar 2023 07:00:10 GMT

Alright, thank you:-)

Re: Setting a parameter when resetting the environment in Reinforcement Learning

Jeanette_Fullmer — Sat, 25 Mar 2023 05:28:34 GMT

Hi @Arthur Ml, was Kavika F's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always unaccept and comment back to reopen your question.