pyroot and TFile deletion

Hi Corey,

Change the line below

f = ROOT.TFile.Open("test.root","recreate")

to

f = ROOT.TFile("test.root","recreate")

will give the same result as you call the destructor explicitly.

The member function of TFile class is a static member function.

The difference between them is for the first case, 'f' would be treated as a class attribute.
while for the second case, it will be treated as an instance of class, so when that instance is no longer needed, the destructor would be called automatically.

Cheers,
Pengfei

Original Message
Sender: Corey Reed<***@nikhef.nl>
Recipient: roottalk<***@cern.ch>
Date: Friday, Aug 2, 2013â22:38
Subject: pyroot and TFile deletion

Hi,

I believe I've tracked a memory leak in a complicated script of mine to
the following effect: after using ROOT.TFile.Open to open a file, the
file does not get destructed when all python references to it are gone.

I'm hoping someone can explain this to me.

Take the following simple python script, using ROOT 5.34.03:

---
import ROOT,gc,objgraph

f = ROOT.TFile.Open("test.root","recreate")
print objgraph.by_type('TFile')

del f
gc.collect()

#f.IsA().Destructor(f)

print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().ls()
---

The resulting output shows:

[<ROOT.TFile object ("test.root") at 0x9597dd8>]
[]
OBJ: TList Files Doubly linked list : 0
TFile** test.root
TFile* test.root

(Using gObjectTable also shows the TFile is still around).

The file is still open! Why?? There are no more python references to the
file, as evidenced by the empty array printed by objgraph.

Changing to ROOT.kMemoryStrict has no affect on this behavior.

Alternatively, calling the destructor explicitly with the macro:

---
import ROOT,gc,objgraph

f = ROOT.TFile.Open("test.root","recreate")
print objgraph.by_type('TFile')

#del f
#gc.collect()

f.IsA().Destructor(f)

print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().ls()
---

Gives the expected result:

[<ROOT.TFile object ("test.root") at 0xa71bde8>]
[]
OBJ: TList Files Doubly linked list : 0

- Corey

Corey Reed

2013-08-02 22:24:31 UTC

Thanks Pengfei, I'm so used to calling the static function in C++ that I
didn't think it might lead to issues in PyROOT.

I'm confused about what you say that "f would be treated as a class

Post by Corey Reed
f = ROOT.TFile.Open("test.root","recreate")
f

<ROOT.TFile object ("test.root") at 0x90f7ab8>

Post by Corey Reed
g = ROOT.TFile("t.root","recreate")
g

<ROOT.TFile object ("t.root") at 0x90f17a8>

Both variables are clearly objects; the first one is not class attributes.

Maybe instead: the use of TFile::Open creates the file on the ROOT side,
so PyROOT does not own it and won't delete it? Whereas with the normal
constructor, the object is created on the Python side and so PyROOT will
delete it.

- Corey

Post by Pengfei Ding
Hi Corey,
Change the line below
f = ROOT.TFile.Open("test.root","recreate")
to
f = ROOT.TFile("test.root","recreate")
will give the same result as you call the destructor explicitly.
The member function of TFile class is a static member function.
The difference between them is for the first case, 'f' would be treated
as a class attribute.
while for the second case, it will be treated as an instance of class,
so when that instance is no longer needed, the destructor would be
called automatically.
Cheers,
Pengfei
Original Message
*Date:* Friday, Aug 2, 2013 22:38
*Subject:* pyroot and TFile deletion
Hi,
I believe I've tracked a memory leak in a complicated script of mine to
the following effect: after using ROOT.TFile.Open to open a file, the
file does not get destructed when all python references to it are gone.
I'm hoping someone can explain this to me.
---
import ROOT,gc,objgraph
f = ROOT.TFile.Open("test.root","recreate")
print objgraph.by_type('TFile')
del f
gc.collect()
#f.IsA().Destructor(f)
print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().ls()
---
[<ROOT.TFile object ("test.root") at 0x9597dd8>]
[]
OBJ: TList Files Doubly linked list : 0
TFile** test.root
TFile* test.root
(Using gObjectTable also shows the TFile is still around).
The file is still open! Why?? There are no more python references to the
file, as evidenced by the empty array printed by objgraph.
Changing to ROOT.kMemoryStrict has no affect on this behavior.
---
import ROOT,gc,objgraph
f = ROOT.TFile.Open("test.root","recreate")
print objgraph.by_type('TFile')
#del f
#gc.collect()
f.IsA().Destructor(f)
print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().ls()
---
[<ROOT.TFile object ("test.root") at 0xa71bde8>]
[]
OBJ: TList Files Doubly linked list : 0
- Corey

Pengfei Ding

2013-08-03 00:19:40 UTC

Hi Corey,

For the line f = ROOT.TFile.Open("test.root", "recreate"),
two things happened for f = ROOT.TFile.Open("test.root", "recreate"),
1) TFile.Open creates an TFile object. This object belongs to an class attribute of the TFile class. The TFile class has the ownership.
2) at the same time, it returns an instance of TFile class to f, but actually f doesn't have the ownership of that TFile object.

del f would not affect that object.

Below is an small piece of code illustrate this.

===================code start ========================

class TFile:
gfile = []
def __init__(self, _a):
self.tfile = []
self.tfile.append(_a)
@staticmethod
def Open(_a):
TFile.gfile.append(_a)
return TFile(_a)
def __del__(self):
self.tfile = []

g = TFile.Open('g')
print g #output: <__main__.TFile instance at 0x1060d2170>
attrg = vars(g)
print ', '.join("%s: %s" % item for item in attrg.items()) # output tfile: ['g']

gg = TFile('gg')
print gg # output: tfile:['gg']
attrgg = vars(gg)
print ', '.join("%s: %s" % item for item in attrgg.items()) # output: <__main__.TFile instance at 0x1060d21b8>

del g
print TFile.gfile # output: ['g']

===================code end ========================

Here is the output

<__main__.TFile instance at 0x1060d2170>
tfile: ['g']
<__main__.TFile instance at 0x1060d21b8>
tfile: ['gg']
['g']

Cheers,
Pengfei

Post by Corey Reed
f = ROOT.TFile.Open("test.root","recreate")
f

<ROOT.TFile object ("test.root") at 0x90f7ab8>

Post by Corey Reed
g = ROOT.TFile("t.root","recreate")
g

<ROOT.TFile object ("t.root") at 0x90f17a8>
Both variables are clearly objects; the first one is not class attributes.
Thanks Pengfei, I'm so used to calling the static function in C++ that I
didn't think it might lead to issues in PyROOT.
I'm confused about what you say that "f would be treated as a class

Post by Corey Reed
f = ROOT.TFile.Open("test.root","recreate")
f

<ROOT.TFile object ("test.root") at 0x90f7ab8>

Post by Corey Reed
g = ROOT.TFile("t.root","recreate")
g

<ROOT.TFile object ("t.root") at 0x90f17a8>
Both variables are clearly objects; the first one is not class attributes.
Maybe instead: the use of TFile::Open creates the file on the ROOT side,
so PyROOT does not own it and won't delete it? Whereas with the normal
constructor, the object is created on the Python side and so PyROOT will
delete it.
- Corey

Noel Dawe

2013-09-22 08:55:22 UTC

Hi Corey,

I realize this is an old thread, but the subject line caught my attention
and I learned something new!

from ROOT import TFile
from rootpy.memory.getownership import GetOwnership # <=== this would

be a nice addition to PyROOT

f1 = TFile('test1.root', 'recreate')
GetOwnership(f1)

True

# ^ Python owns f1
f2 = TFile.Open('test2.root', 'recreate')
GetOwnership(f2)

False

# ^ ROOT owns f2

But you can use ROOT.SetOwnership to tell PyROOT that Python will own this

import ROOT
from rootpy.memory.getownership import GetOwnership
f = ROOT.TFile.Open('test.root', 'recreate')
GetOwnership(f)

False

ROOT.SetOwnership(f, True) # tell PyROOT that Python will own f
GetOwnership(f)

True

del f
import gc; gc.collect()

len(ROOT.gROOT.GetListOfFiles())

0

I committed this change to rootpy [1] today so that
rootpy.io.root_open/rootpy.io.File.Open/rootpy.ROOT.TFile.Open calls

import rootpy.ROOT as ROOT
f1 = ROOT.TFile.Open('test1.root', 'recreate')
del f1
import gc; gc.collect()

len(ROOT.gROOT.GetListOfFiles())

0

Cheers,
Noel

[1] https://github.com/rootpy/rootpy

Post by Corey Reed
Hi,
I believe I've tracked a memory leak in a complicated script of mine to
the following effect: after using ROOT.TFile.Open to open a file, the file
does not get destructed when all python references to it are gone.
I'm hoping someone can explain this to me.
---
import ROOT,gc,objgraph
f = ROOT.TFile.Open("test.root","**recreate")
print objgraph.by_type('TFile')
del f
gc.collect()
#f.IsA().Destructor(f)
print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().**ls()
---
[<ROOT.TFile object ("test.root") at 0x9597dd8>]
[]
OBJ: TList Files Doubly linked list : 0
TFile** test.root
TFile* test.root
(Using gObjectTable also shows the TFile is still around).
The file is still open! Why?? There are no more python references to the
file, as evidenced by the empty array printed by objgraph.
Changing to ROOT.kMemoryStrict has no affect on this behavior.
---
import ROOT,gc,objgraph
f = ROOT.TFile.Open("test.root","**recreate")
print objgraph.by_type('TFile')
#del f
#gc.collect()
f.IsA().Destructor(f)
print objgraph.by_type('TFile')
ROOT.gROOT.GetListOfFiles().**ls()
---
[<ROOT.TFile object ("test.root") at 0xa71bde8>]
[]
OBJ: TList Files Doubly linked list : 0
- Corey

w***@lbl.gov

2013-09-22 21:44:28 UTC

Noel,

Post by Noel Dawe
I committed this change to rootpy [1] today so that
rootpy.io.root_open/rootpy.io.File.Open/rootpy.ROOT.TFile.Open calls

there's a short-cut to that, if this is generally true for a method:

import ROOT
ROOT.TFile.Open._creates = True

that way also, the ownership rules of the method propagate. Certain methods
who's behavior is obvious from the name, such as Clone(), are automatically

Post by Noel Dawe

print ROOT.TFile.Clone._creates

1

This is a problem with automatic bindings: it can not in general be inferred
from the signature alone what the behavior is. In C++11, with solid smart
pointers as part of the standard library and move constructors that make
return-by-value efficient, all of this my one day be resolved at the interface
level. Until then, some intelligence, as is being added by rootpy, is needed
for most methods that return pointers.

Best regards,
Wim

[1] https://github.com/rootpy/rootpy

--
***@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

Noel Dawe

2013-09-24 02:09:35 UTC

Hi Wim,

Thanks for the tip! I'll use ROOT.TFile.Open._creates = True instead. From
what tag of ROOT is _creates available?

Thanks,
Noel

Post by Noel Dawe
Noel,
I committed this change to rootpy [1] today so that

rootpy.io.root_open/rootpy.io.**File.Open/rootpy.ROOT.TFile.**Open calls

import ROOT
ROOT.TFile.Open._creates = True
that way also, the ownership rules of the method propagate. Certain methods
who's behavior is obvious from the name, such as Clone(), are automatically

print ROOT.TFile.Clone._creates

1
This is a problem with automatic bindings: it can not in general be inferred
from the signature alone what the behavior is. In C++11, with solid smart
pointers as part of the standard library and move constructors that make
return-by-value efficient, all of this my one day be resolved at the interface
level. Until then, some intelligence, as is being added by rootpy, is needed
for most methods that return pointers.
Best regards,
Wim
[1] https://github.com/rootpy/**rootpy <https://github.com/rootpy/rootpy>
--

w***@lbl.gov

2013-09-24 05:47:30 UTC