Setting a parameter when resetting the environment in Reinforcement Learning

[arthur_ml]

[ FlexSim 22.2.1 ]

Hi everyone,

I am doing Meta Reinforcement Learning for production scheduling. Therefore, I predefined several job lists (n=50) that should be scheduled optimally with RL. In MetaRL, the algorithm regards each list as "task" and the algorithm should be able to select a specific task (i.e. 1 list of the 50 lists) when resetting the environment. I want realize that, by changing the value of a global parameter, which then leads to the usage of the specific list. I.e.: episode is finished -> MetaRL selects. e.g. task=14 for the next episode -> parameter is set to 14 -> flexsim is resetted and job list 14 is used in the model now.

Currently, my code for resetting flexsim looks like this:

def _reset_flexsim(self):
    if self.verbose:
        print("Sending Reset message")
    resetString = "Reset?"
    if hasattr(self, "seedNum"):
        resetString = "Reset:" + str(self.seedNum) + "?"
    self._socket_send(resetString.encode())

Can anyone help me please how I can set the parameter value when resetting flexsim?

Thanks in advance!

[kavika_faleumu · Accepted solution]

Hey @Arthur Ml, one way you can set the parameter's value on model reset is using the "OnModelReset" trigger found in the Toolbox:

This gives you a code editor that will execute when the model is about to Reset. Here you can grab a reference to the Parameter's value cell and set it with the task for the next run. In my example I simply set it with the output count of a Processor.

set-param-on-reset.fsm

Hope this helps!

[arthur_ml]

Dear @Kavika F ,

thanks for the quick response.

Sorry, my question was not specific enough. How can I set a parameter value from Python, when resetting the model/ RL environment? I use a socket connection for connecting Python with Flexsim as presented in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html). So far, with this socket connection, I can launch, reset, and close Flexsim, and I can send actions and receive observation from the model. But how can I set a specific parameter value when resetting the model?

Thanks and best regards,
Arthur

[kavika_faleumu]

Oh okay. Once you reset your model, you can use the following command to set a parameter:

controller.setParameter("ParameterName", value)

If you look at the FlexSimPy.cpp, you can see a list of commands you can use from the controller (and within the FlexSim):

[arthur_ml]

Is there a way to set a Parameter in Flexsim with the help of the "flexsim_env.py" script you provided in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html)? Because I am using the scripts from the documentation to connect to Flexsim and do the training.

[kavika_faleumu]

Sorry I thought you were using the code from the FlexSimPy module on github used to connect to FlexSim externally. That is where I got my code from and reference the FlexSimPy.cpp. I don't know if there's a way to use the flexsim_env.py to change a model parameter; I think the socket connections it deals with are specific to the RL Tool in FlexSim. However, if you use the FlexSimPy module (linked above) to control FlexSim externally, it gives you the capability to set the parameter.

[arthur_ml]

Alright, thank you:-)

[Jeanette_Fullmer]

Hi @Arthur Ml, was Kavika F's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always unaccept and comment back to reopen your question.