Finally, in the ending steps of the method, we conduct the statistical analysis from the data of the study cases, evaluating them according with the obtained results. Or in other words, we intend to extract the useful information from the simulation results.
As we saw in the last step, we should have by now a detailed experimental design for each factor combination and for n samples. Following this, we apply the same statistical technique utilized in the statistical validation process, building a confidence interval for each parameter and each parameter combination.
Using the same formulation as before, we use the mean and variance for each parameter (or parameter combination) j as in (1) and (2).
(1)
(2)
We use these values to build a confidence interval (3) the same way as before, using t-student distribution (less then 30 samples). The t value can be obtained from the table from step 6 (or this link) value based as function of n-1 samples and the 1-α/2 critical points from 100*(1-α) desired percent confidence, as α being an auxiliary constant.
(3)
Again, we have 3 resulting possibilities, as show bellow:
However, here there are not undesirable intervals, the non-zero intervals may be more informative actually. When the CI have zero included, all we can say is that the chosen parameter value (or algorithm) for "+", in the experimental design, did not have any impact in comparison with the "-" value (which is our original standard simulation).
But when we have a CI which did not included zero, we now known that not only it had an impact on the results, but how much it was in average, meaning that we can determina if its scale is significant or not.
Moreover, when we see the impact between two or more factors at the same type, we can determine if factors interact. A measure of interaction is the difference between the average effect of factor A when factor B is at its "+" level (and all other factors other than A and B are held constant) and the average effect of factor A when factor B is at its "-" level [1].
_________________________________________________________________________________
Example: In the graph bellow we indicate all CI for a 3 factor experimental design about rendez-vous convergence time. Consider an average convergence time of 60 seconds of the standard simulation ("-" level) and that we build it with 20 samples. From the graph we can obtain the following information:
Factor 1 (e1) : have a heavy negative impact on the final result, as it may double the convergence time (approximately between 60+60 ~ 60+75). It also have a big variance interval, meaning that it is affected by the initial conditions.
Factor 2 (e2): it shows as statistical significant but almost did not have impact on the results when we observe its value (less then 1 second), meaning it is not significant for itself to our simulation.
Factor 3 (e3): here we have a significant variance on the results, probably due initial conditions choice, but results are not statistical significant as it include zero. So we may conclude that this factor does not have a impact on the results.
Paired Factors Combination (e12, e13, e23): we can observe the interaction between factors1-3 and 2-3 cases, which means that their values will depend of the values of the paired parameters.
This is especially important in these both cases, as factor 2 (e2) did not have a significant impact because of its value (e2) but shows a significant higher value depending on the values of factor 1(e1).
This also happens for the factor 3, which was statistically insignificant but having a significant effect depending of the values of factor 1.
And for the case of factor 2-3, as it includes zero and had low values, we can say that they do not interact to each other.
We can also conclude this for the 3 way comparison.
_________________________________________________________________________________
If you need more information and examples about this statistical process, please take a look at chapter 12.2 from [1].
The last step is to take all the information build so far and check if it is what was expected from the study, look for possible analysis and determine if there is any future works to be made taking in account a behavior or information that was not explored in the current study cases.
If the complete study is not what we expect, than we start again from the beginning. However, if it was the expected result, than we present the results in form of graphs, charts, publications, presentations, etc.
References
[1] Law, A. M. “Simulation Modelling and Analysis”, McGraw-Hill, 5th Edition, pp. 804, 2015.
[2] Klugl, F. et al. “A validation methodology for agent-based simulations”, in Proceedings of the 2008 ACM Symposium on Applied Computing (SAC), DOI: 10.1145/1363686.1363696, 2008.
[3] Balci, O. "Introduction to Modeling and Simulation". Class Slides, ACM SIGSIM, Available in <http://www.acm-sigsim-mskr.org/Courseware/Balci/introToMS.htm>, 2013.
[4] Sargent, R. G. “Verification and validation of simulation models”, in Journal of Simulation, 7, 12-24, 2013.
[5] Siegfried, R. "Modeling and Simulation of Complex Systems", Springer Vieweg, ISBN 978-3-658-07528-6, 2014.
[4] Sargent, R. G. “Verification and validation of simulation models”, in Journal of Simulation, 7, 12-24, 2013.
[5] Siegfried, R. "Modeling and Simulation of Complex Systems", Springer Vieweg, ISBN 978-3-658-07528-6, 2014.
Page Release: 11/05/17
Last Update: 11/05/17
