Moore Memory Consumption

Final presentation (16.04.2012)

11.04.2012

• Moore v13r3
• CMTCONFIG=x86_64-slc5-gcc46-opt
• tcmalloc
• Start profiling after 5000 events, stop at 80000 events. Total memory usage counter at 5000 events == 0.
• Dump the memory usage data each 1Gb memory allocation/deallocation cycle. Totaly I have 124 dumps.
• At the each dump analyze a total memory usage and select top 3 functions for the plot.
• At the plot below x-axis is a dump number and y-axis is memory usage (Mb).

(Full version)

What I see at the log – it’s a huge peak between 75600 and 75700 events

The problem can be:

1. in HltSelReports (I don’t know what is the meaning of the warning on the image above)
2. somewhere inside call chain with LoKi::Hlt1::HelperTool::_create. In this chain a huge memory was allocated.
3. or I need more statistic (may be after 80000 events the memory will be free)
4. in google perfomance tools (with lower probability). Now I’m running the same test in debug mode.

(Full version of call chain (pdf))

10.04.2012

Top Memory Usage by Function

• Moore v13r2p5
• CMTCONFIG=x86_64-slc5-gcc46-dbg
• Start profiling after 5000 events, stop at 10000 events. Total memory usage counter at 5000 events == 0.
• Dump the memory usage data each 1Gb memory allocation/deallocation cycle. Totaly I have 63 dumps.
• At the each dump analyze a total memory usage and select top 3 functions for the plot.
• At the plot below x-axis is a dump number and y-axis is memory usage (Mb).
• If needed, it' possible to build a call chain graph for each function.
• Full version of the plot
• Call chain stack for the last dump (#63) (look to the big squares there).

09.04.2012

• export CMTCONFIG=x86_64-slc5-gcc46-opt
• SetupProject Moore v13r2p5
• I use google profiling auditor from Gaudi package.
• I’ve run profiling from 5000 to 10000 events.
• I’ve a lot of error messages like:
  • ERROR LoKi::L0Filter:: LHCb::L0DUReportObject is invalid, return ‘false’

Results

After 5000 events

After 10000 events:

Call chain

Full version (pdf)

08.04.2012

Preliminary results

• Every 100 events memory consumption increases by 1 to 2 Mb, but it increases constantly.
• All the memory consumption increased in the following functions:
  1. boost::pool::malloc_need_resize
  2. std::vector::_M_insert_aux

  I don’t know (yet) who calls this functions.

How to use profiler

• Run (use your option file with Google auditor):

  $> LD_PRELOAD=/afs/cern.ch/sw/lcg/external/tcmalloc/1.7p1/x86_64-slc5-gcc46-opt/lib/libtcmalloc_and_profiler.so gaudirun.py ~/leak/moore.py | tee log.txt

• Analyze (use your dump file):

  pprof --text `which python` FULL-Events7001To12001.0068.heap

Option file