differences in EvaluateMVA

Tomas Davidek

2014-09-15 21:30:50 UTC

Dear Helge,
thanks a lot for the feedback. As I said in my previous mail, the
difference is not big, but of course it implies some troubles when we
were checking the results between several people. The BDT output should
be between -1 and 1 by definition (or we are using the most common BDTs ?).

I attached a figure with two plots - the scatter plot between BDT scores
calculated in the old (5.34.18) and new (5.34.20) version of ROOT and
the relative difference between the two. You can see that in most of the
cases the difference is very small, but in few cases the difference can
be as large as 50%... Overall, the RMS of the relative difference
amounts to ~5%...

It seems the plot is perfectly in line with what you wrote. Also, since
the BDT was originally trained in ROOT 5.34.14, I believe it is better
to stick to the older version (5.34.18) before the change to be fully
compatible... I did not evaluate the impact on the final physics results
(hopefully negligible) due to lack of time, we are in the final stage of
the paper approval...

Cheers,
Tomas

Dear Tomas,
oh .. that is indeed, as you said, not to be expected. Looking through
the code changes I noticed that I had made a change that became
necessary in the decision of whether an event goes to the 'left' or ot
the 'right' when going down the tree, from > to >= , otherwise the
handling of integer values (introduced in root 34.20) doesn't work
correctly. Now I expected this to be of negligible effect on 'real'
valued data. Now I see however, that also accidentally a 'trial'
version of the >= compariison, which doesn't belong there creeped into
the current code.
In both cases however, I would expect the differences to be small, and
not substantial as suggested in your email.
As this makes me worried, could you give me more detail about whot the
output distributions compare? You said that for 5.34.18, it is between
-1,1 with RMS=5%, and for 5.34.20 ?? Do you have some plots for me ?
Cheers,
Helge
Hi Helge,
Can you, or Eckhard answer this message in the roottalk mailing list
Thank you
Lorenzo

*Subject: **differences in EvaluateMVA*
*Date: *10 Sep 2014 22:00:52 GMT+2
Hello,
when working with BDT scores, I noticed that these scores are
(slightly) different when evaluating via
TMVA::Reader::EvaluateMVA between different ROOT versions. I
double-checked that the inputs to the BDT scores are perfectly
the same, but the BDT output is different. I exercised that on
lxplus with three different ROOT versions, 5.34.18, 5.34.20 and
5.34.21. While the latter two versions result in the exactly same
BDT score, 5.34.18 is different (the BDT score ranges from -1 to
+1, the difference is centered at zero and shows RMS of ~5%).
The xml BDT files were obtained by training with MVA ROOT version
5.34.14....
I understand that there might be a difference between ROOT
versions when training the BDT, but I thought that the xml must
give the same output regardless the ROOT version.
Is such behaviour expected and/or is there a known bug/problem in
5.34.18 regarding TMVA?
Cheers,
Tomas