Written by Jill on Friday February 8th 2013 in Resources, Technical Corner

Successful Simulations – Validation

Today I’m speaking to Liam Hastie, Lean expert and SIMUL8 consultant and Dr. Mark Elder, SIMUL8’s Founder.  I’m asking ‘How do I validate my simulation?’  Here’s what Liam had to say:

“For me validation is a very exciting part of the process because when this is done right, it can really go a long way to reinforcing the Lean principles of staff involvement and encouraging communication.  This is what most people miss about process improvement in general, and what I feel people often overlook in terms of the benefits that simulation provides.

Validation is really just checking that what you are seeing in your simulation is what you see happening in the real process. It is necessary to assess the ‘as-is’ or ‘current state’ process in this way so that we can have greater confidence in the accuracy of the simulation when we start experimenting with new scenarios and our ‘to be’ or ‘future state’ processes.  It is also worth a quick mention that validation should also be done in cases where there is no existing ‘as-is’ process, i.e. we are creating a process that is completely new.  In these cases validation would be broken down to assessing the accuracy of individual steps in order to validate the system as a whole.

Validation  is an extremely effective way of engaging teams and ensuring that meaningful and sustainable improvements are made to a process. This is because validation is clearly a win-win activity; if you find that the simulation passes validation, you can now progress to testing new scenarios and have confidence that you will be able to create a more efficient process. Alternatively, if you’re unable to validate the simulation this can be a good thing too!

Tracking down and determining exactly which part of a process did not run as expected compels teams to communicate and re-assess previously held assumptions, which is an often over-looked but hugely significant benefit that simulation offers.  There are three direct benefits of conducting this sort of re-evaluation;

1. This will inevitably promote the team into creating process improvements that are inherently more sustainable and robust (and more likely to be accepted by staff!)
2. This can promote entirely new ideas or thinking to emerge.
3. And this is purely derived from experience, in 9 cases out of 10, the part of the process that was not matching expectations upon validation turns out to be the point in the system where the most efficiency savings ultimately end up being made (this was the part of the process that was not previously properly understood).

There are three ways to validate your simulation:

1. Confirm KPIs. When you run the simulation do you get similar results for all of your KPIs? Have you further validated this stochastically by checking repeated runs and trials where the randomness of the process will be mirrored?
2. Communicate! SIMUL8 offers huge benefits in being able to communicate with staff and this is really the most effective way achieving sustainable processes improvements.
3. Test the simulation by making changes, even crazy ones! In SIMUL8 it is very easy to make quick modifications and to use this as a way of validating your simulation. When you make changes to the simulation, the system should behave in a similar way to how these changes would affect the real process.

So, in a nutshell – validation adds value and will make your simulations more successful!”

Thanks Liam!  Mark, do you agree?

“I agree, it’s really important – you must validate your simulation.  There are two ways to do this – and you have to do one or the other, each has its advantages but sometimes for practical reasons one way just can’t be done.

Most books suggest the traditional method: testing your simulation against some past data from the real system.  If you use this method make sure you don’t forget that you are using statistical distributions in your simulation but your real data was a one-off case – so you can’t expect them to be identical – just statistically close: use the confidence intervals on SIMUL8’s Results Manager to help with this.

The other method involves your client much more, which is normally a big advantage.  Make sure your client knows you are in a validation phase and they should not expect results to be right yet (otherwise they will immediately lose confidence) – validation is, after all, because you can expect to have some things wrong on the first pass.  The value with this method is, you will find your client learning huge chunks of interesting things about their process even when helping validate a currently incorrect simulation, so it is really worth getting them involved.

Ask the client questions about detailed expectations – “how long would you expect it take to get product X from A to B”, run the simulation and see if it matches – if not – who is wrong, the simulation or the client, yes that’s what I mean about the client learning.  Dig in deeper and fix where necessary. Repeat that until you are both comfortable that the simulation is producing usefully valid information.  Remember it’s only a model, it is, by definition, a simplification of reality.  Don’t worry if it is not perfect – it needs to be good enough to get the decision making right.”

Thanks Mark!  We’ll be back in March with more experts, and more ways to make your simulations successful.