Debugging memory problems on solaris with libumem

pmap -x showed an increasing memory usage of an application.

After searching, I found  a lot of hints using libumem for debugging
memory issues in Solaris / OpenSolaris.

Hints are here , here and here and also a lot more.

On Build 134 I started the application with:

# libumem
LD_PRELOAD=libumem.so.1
export LD_PRELOAD
##UMEM_DEBUG=default
UMEM_DEBUG=default,firewall=1
export UMEM_DEBUG
UMEM_LOGGING=transaction
export UMEM_LOGGING
UMEM_OPTIONS=backend=mmap
export UMEM_OPTIONS
After starting the application, I attached the mdb with the command:
mdb -p
>::sysbp _exit
>::cont

After this I let the application run and  shut down. During shutdown the
debugger stopped short before finishing:


mdb: stop on entry to _exit
mdb: target stopped at:
0xfe8c3f68:     nop  

Now we trapped in the exit function of the process and verify the
recordings of libumem:
 
> ::umem_verify
Cache Name                      Addr     Cache Integrity    
umem_magazine_1                 fe826010 clean
umem_magazine_3                 fe826390 clean
umem_magazine_7                 fe826710 clean
umem_magazine_15                fe826a90 clean
umem_magazine_31                fe827010 clean
umem_magazine_47                fe827390 clean
umem_magazine_63                fe827710 clean
umem_magazine_95                fe827a90 clean
umem_magazine_143               fe828010 clean
umem_slab_cache                 fe828390 clean
umem_bufctl_cache               fe828710 clean
umem_bufctl_audit_cache         fe828a90 clean

Here every buffer is clean and no memory overwriting occured, otherwise
there would be a message like:
 umem_alloc_1152                  8098a90 1 corrupt buffer
Show open memory allocations:
> ::findleaks
BYTES             LEAKED VMEM_SEG CALLER
8192                   1 fef39000 MMAP
------------------------------------------------------------------------
           Total       1 oversized leak, 8192 bytes

CACHE     LEAKED   BUFCTL CALLER
fe7d3010       1 fae39eb8 libldap50.so`ldap_x_calloc+0x2d
fe7d0710       6 fae39c60 libldap50.so`ldap_x_malloc+0x29
fe7d0710       1 fae39f30 libldap50.so`ldap_x_malloc+0x29
fe7d0a90      10 fae39e40 libldap50.so`ldap_x_malloc+0x29
fe7d0390       3 fab805f0 libstdc++.so.6`_Znwj+0x29
fe7d0390      17 f9c4e480 libstdc++.so.6`_Znwj+0x29
fe7d0390       2 fab80500 libstdc++.so.6`_Znwj+0x29
fe7d0390      21 fb026ad0 libstdc++.so.6`_Znwj+0x29
fe7d0390       3 fab80398 libstdc++.so.6`_Znwj+0x29
fe7d0390      28 fa91d168 libstdc++.so.6`_Znwj+0x29
fe7d0390      22 fa66df28 libstdc++.so.6`_Znwj+0x29
fe7d0710       1 fa081878 libstdc++.so.6`_Znwj+0x29
fe7d0710       1 faae2148 libstdc++.so.6`_Znwj+0x29
fe7d9a90       1 fde2ce40 lib
XXXXXX.so403`scf_XXXXXX+0x3c
fe7d9a90       1 fd97b078 lib
XXXXXX.so403`scf_XXXXXX+0x3c
fe7d0710       1 fb0842f8 lib
XXXXXX.so403`scf_malloc+0x48
fe7d1010       1 fb084370 lib
XXXXXX.so403`scf_malloc+0x48
fe7dda90       1 fa82f4c0 lib
XXXXXX.so403`scf_malloc+0x48
fe7df390       1 fa0818f0 lib
XXXXXX.so403`scf_malloc+0x48
fe7d3710       1 fe7b48f0 lib
XXXXXX.so403`init_libXXXXXX+0x18
------------------------------------------------------------------------
   Total     123 buffers, 25112 bytes
Analyse a specific buffer:
> fb0842f8::bufctl_audit
            ADDR  BUFADDR  TIMESTAMP    THREAD
                    CACHE    LASTLOG    CONTENTS
        fb0842f8  f9f16000
a926cc9ecf6a     1
                  fe7d0710     fe8033e8     0
                 libumem.so.1`umem_slab_alloc+0x1b7
                 libumem.so.1`umem_cache_alloc+0xe5
                 libumem.so.1`umem_alloc+0xcd
                 libumem.so.1`malloc+0x2a
                 lib
XXXXXX.so403`scf_malloc+0x48
                 libvsmsgCsi.so403`scf_
XXXXXX+0x46b
                 libvsmsgCsi.so403`bcf_
XXXXXX+0x1a1
                 lib
XXXXXX.so403`ctrl_XXXXXX+0x106
                 lib
XXXXXX.so403`ctrl_XXXXXX+0x1bc
                 lib
XXXXXX.so403`ctrl_XXXXXX+0xb2
                 lib
XXXXXX.so403`ctrl_XXXXXX+0x7dd
                 lib
XXXXXX.so403`ctrl_XXXXXX+0x1bc
                 lib
XXXXXX.so403`prog_XXXXXX+0x9a
                
XXXXXX?main+0x140
                 lib
XXXXXX.so403`main+0x196

The real memory pointer in the executable is f9f16000 + 8.

The length of the data can be calculated by dumping the memory:


> f9f16000/10X
0xf9f16000:     2579            3a109a87        2f6a392f        51414134
                4a5a6b53        42416752        41455141        4c4


and calculate 2579-8 in hex seems to be the size of the allocated buffer.

If the leak occured more times, there is only the last entry in the
::findleaks output.  In the column LEAKED is the number of
occurrences of the leak.


To get all buffers issued the following:


::walk leak 
or
::walk leakbuf

and this shows you all leaking buffers, which can be inspected with
::bufctl_audit