Reinforcement Learning with FlexSim Model. Fps decreases with training time

[nitin_s2]

[ FlexSim 22.0.0 ]

I am following the tutorial (link) on the FlexSim website to integrate RL with an existing FlexSim model. Using the codes provide there too for training and inference.

As I was training the RL agent, I could see that the fps (frames per second) showing a consistent decrease (see graph) with model training time. High fps means more iterations of FlexSim environment (faster training) and it usually should fluctuate around a constant value.
I was wondering if this is a known case and if someone could point me to a way to fix is.

fps-drop.PNG

[Jacob_White · Accepted solution]

Hi @Nitin S2,

In reinforcement learning the fps generally will decrease over time because the algorithm takes more and more time to think about what to do. There is a possibility that we have a bug FlexSim that slows FlexSim down, but you'd probably notice that running the model over and over. Example: run an experiment with 50 replications, only using 1 core. Does the model run slower and slower? If not, then the slowdown must be the RL tool just thinking a lot.

[nitin_s2]

Thanks for the response.

I am not sure that fps decreasing with time is an RL exploration issue. Will run the experiment and find more on it.

[Jeanette_Fullmer]

Hi @Nitin S2, was Jacob W2's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always unaccept and comment back to reopen your question.