FlexSim reinforcement learning

[a9080109]

[ FlexSim 23.0.0 ]

I want to know some details about models and reinforcement learning

For example, episode corresponds to the meaning represented by the model

Or what timestep corresponds to in the model. Because I wanted to understand why I had a spike in rewards at the beginning when I was training.

[joerg_vogel_HsH]

Probably a division by zero or near zero.

[a9080109]

So how do I solve this problem in the model

[joerg_vogel_HsH · Accepted solution]

A quick and dirty way would be to work with a warmup time or you transmit rewards a bit later in your model runtime.

[a9080109]

I don't quite understand the meaning of warm up and what impact it will have on the model.0926.fsm

[jason_lightfootVL7B4]

You can find the warmup description in the online documentation. It could somehow influence the timesteps in question if your rewards are based on model statistics. I'm not sure if it would explain the spike in your graph.

[jason_lightfootVL7B4]

I believe the term 'time-step' comes from the action/reward step in a Markov Decision Process, and their number is aligned to the number of cycles of action->simulate->observe->reward within your episode.

[a9080109]

Thanks but I have other questions,

For example, what is the meaning of the flexsim model corresponding to each timestep?

What does each epoch and episode mean?

[jason_lightfootVL7B4]

As I said above I believe the term 'time-step' comes from the action/reward step in a Markov Decision Process, and their number is aligned to the number of cycles of action->simulate->observe->reward within your episode.

An episode seems to be a series of timesteps resulting in a terminal state. I would expect that's often equivalent to a model run.

In the ML world is seems this doesn't have to be the case in that you could define an episode as 11 hands of cards and the terminal state is having played them all since the target outcome is to win the majority. The same could be true when applied to simulations in that you might define an episode as n replications and you might be interested in achieving a certain kpi in 95% of them. I'm not yet sure if FlexSim supports this.

The definition of epochs seems to vary a little but most talk about one pass through a training dataset.

Note: I'm new to ML/RL and trying to get up to speed, so may need correcting on this. You can probably search the internet and find all the definitions.

[a9080109]

My new problem now is that I used warm up but there was a problem with my sink data.

I guess it’s because I have WARM UP but my SINK calculation is still complete and lateness is required to do the calculation.

current.avgTardiness = (current.lateness/current.stats.input.value);

0926.fsm

[kavika_faleumu]

@mark zhen, where did you get this graph? Did you plot this from python or FlexSim?

[a9080109]

@Kavika F @Jason Lightfoot so can u know how to solve the problems

[Jeanette_Fullmer]

Hi @mark zhen ,

Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.