Sep 28, 2010

Debugging memory leaks with DTrace and mdb

Some time ago I wrote about debugging memory problems with libumem.

Now I've a different approach using dtrace and mdb to find the source code position.

First a let the program execute and finish running under dtrace.
The dtrace-Script is from http://blogs.sun.com/sanjeevb/.


# cat ./memleak.d

#!/usr/sbin/dtrace -s

pid$1:libc.so.1:malloc:entry
{
        self->trace = 1;
        self->size = arg0;
}
pid$1:libc.so.1:malloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Size=%d", arg1, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:realloc:entry
{
        self->trace = 1;
        self->size = arg1;
        self->oldptr = arg0;
}

pid$1:libc.so.1:realloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Oldptr=0x%p Size=%d", arg1, self->oldptr, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:calloc:entry
/self->trace == 1/
{
        self->trace = 1;
        self->size = arg1;
}

pid$1:libc.so.1:calloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Size=%d", arg1, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:free:entry
{
        printf("Ptr=0x%p ", arg0);
}

The result output from the dtrace-Script is postprocessed with a
Perl-Script checking if alloced memory is freed again:


#!/usr/bin/perl

# findleaks.pl

use Data::Dumper;

my %hash = ();

while (<>) {
        if ((/malloc:return Ptr=([^ ]*) Size=(.*)/) || 
            (/calloc:return Ptr=([^ ]*) Size=(.*)/)) {
                $hash{$1} = { size => $2 };
  while (<>) {
   last if /^$/;
   $hash{$1}->{stack} .= $_;
  }
        }
        elsif (/free:entry Ptr=([^ ]*)/) {
                if (exists $hash{$1} and $hash{$1}) {
                        $hash{$1} = '';
                }
        }
 elsif (/realloc:return Ptr=([^ ]*) Oldptr=([^ ]*) Size=(.*)/) {
  if ($1 eq $2) {
   if (exists $hash{$1} and $hash{$1}) {
    $hash{$1} = { size => $3 };
    $hash{$1}->{stack} = '';
    while (<>) {
                          last if /^$/;
                           $hash{$1}->{stack} .= $_;
                  }
   } 
  } else {
    $hash{$1} = '';
    $hash{$2}= { size => $3 };
    $hash{$2}->{stack} = '';
    while (<>) {
                          last if /^$/;
                           $hash{$2}->{stack} .= $_;
                  }
  }
 }

}

foreach my $key (keys %hash) {
 next if not $hash{$key}->{size};
 print "Ptr=$key Size=", $hash{$key}->{size}, "\n";
 print $hash{$key}->{stack}, "\n---------\n";
}

Now we get a list of memory allocations which are not freed after closing
the application.

Such an entry might look like
---------
Ptr=0x81c64f8 Size=20
              libumem.so.1`malloc+0x3e
              libstdc++.so.6`_Znaj+0x1b
              libxerces-c.so.28`_ZN11xercesc_2_818IconvLCPTranscoder9transcodeEPKc+0xc4
              libxerces-c.so.28`_ZN11xercesc_2_89XMLString9transcodeEPKc+0x22
              libvsEAI.so404`readFunction+0x39
              libvsEAI.so404`bcfEAIRepositoryInit+0x5a9
              libvsopCsiBdOOPool.so404`ctrl_call_bcf+0x106
              libvsopCsiBdOOPool.so404`ctrl_call_list+0x1bc
              libvsopCsiBdOOPool.so404`ctrl_gosub_list+0xb2
              libvsopCsiBdOOPool.so404`ctrl_call_bcf+0x7dd
              libvsopCsiBdOOPool.so404`ctrl_call_list+0x1bc
              libvsopCsiBdOOPool.so404`ctrl_gosub_list+0xb2
              libvsopCsiBdOOPool.so404`ctrl_call_bcf+0x7dd
              libvsopCsiBdOOPool.so404`ctrl_call_bcf+0x795
              libvsopCsiBdOOPool.so404`ctrl_call_bcf+0x795
              libvsopCsiBdOOPool.so404`ctrl_call_list+0x1bc
              libvsopCsiBdOOPool.so404`prog_control+0x9a
              progvs_prog`vsop_main+0x5a2
              libvszwflx2Bd.so404`main+0x196
              progvs_prog`_start+0x80

To get the position in the readFunction  method, make a core of the application by using
gcore [-F]

and start in mdb with mdb
mdb <core file name >
> libvsEAI.so404`readFunction::dis
libvsEAI.so404`readFunction:    pushl  %ebp
libvsEAI.so404`readFunction+1:  movl   %esp,%ebp
libvsEAI.so404`readFunction+3:  pushl  %edi
libvsEAI.so404`readFunction+4:  pushl  %esi
libvsEAI.so404`readFunction+5:  pushl  %ebx
libvsEAI.so404`readFunction+6:  call   +0x5    
libvsEAI.so404`readFunction+0xb:popl   %ebx
libvsEAI.so404`readFunction+0xc:addl   $0x1527d,%ebx
libvsEAI.so404`readFunction+0x12:       subl   $0xd8,%esp
libvsEAI.so404`readFunction+0x18:       leal   0xfffef969(%ebx),%eax
libvsEAI.so404`readFunction+0x1e:       pushl  %eax
libvsEAI.so404`readFunction+0x1f:       movl   0x8(%ebp),%edi
libvsEAI.so404`readFunction+0x22:       call   -0x620e 
libvsEAI.so404`readFunction+0x27:       movl   %eax,-0x88(%ebp)
libvsEAI.so404`readFunction+0x2d:       leal   0xfffef7a1(%ebx),%eax
libvsEAI.so404`readFunction+0x33:       pushl  %eax
libvsEAI.so404`readFunction+0x34:       call   -0x6220 
libvsEAI.so404`readFunction+0x39:       movl   %eax,-0x8c(%ebp)
libvsEAI.so404`readFunction+0x3f:       leal   0xfffef972(%ebx),%eax
libvsEAI.so404`readFunction+0x45:       pushl  %eax
libvsEAI.so404`readFunction+0x46:       call   -0x6232 
libvsEAI.so404`readFunction+0x4b:       movl   %eax,-0x90(%ebp)
libvsEAI.so404`readFunction+0x51:       leal   0xfffef978(%ebx),%eax
libvsEAI.so404`readFunction+0x57:       pushl  %eax
libvsEAI.so404`readFunction+0x58:       movl   %eax,-0xb0(%ebp)
libvsEAI.so404`readFunction+0x5e:       call   -0x624a 
libvsEAI.so404`readFunction+0x63:       leal   0xfffef97b(%ebx),%edx
Because of the offset from the stack trace  libvsEAI.so404`readFunction+0x39 we see in the disassembly, that the second call to libxerces-c.so.28`_ZN11xercesc_2_89XMLString9transcodeEPKc+0x22 is the problem.


The method called is
>echo "_ZN11xercesc_2_89XMLString9transcodeEPKc" | c++filt
xercesc_2_8::XMLString::transcode(char const*)





Sep 21, 2010

SFTP withing jedit doesn't work in OpenIndiana

I use a lot the FTP plugin from jEdit.
This plugin allows you to open a SFTP connection to a remote host via ssh and
work with files from the remote host, as if they are local. The plugin is
available via the file browser.

You can load and save the files.

Within OpenIndiana this just doesn't work anymore.
The activity.log shows:
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: Remote version string: SSH-2.0-Sun_SSH_1.0.1
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: Local version string: SSH-2.0-JSCH-0.1.43
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: aes256-ctr is not available.
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: aes192-ctr is not available.
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: aes256-cbc is not available.
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: aes192-cbc is not available.
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: arcfour256 is not available.
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: SSH_MSG_KEXINIT sent
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: SSH_MSG_KEXINIT received
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: kex: server->client aes128-cbc hmac-md5 none
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: kex: client->server aes128-cbc hmac-md5 none
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: SSH_MSG_KEXDH_INIT sent
8:39:58 AM [jEdit I/O #3] [message] SftpLogger: expecting SSH_MSG_KEXDH_REPLY
8:39:59 AM [jEdit I/O #3] [message] SftpLogger: ssh_rsa_verify: signature false
8:39:59 AM [jEdit I/O #3] [message] SftpLogger: Disconnecting from onbahaspa port 22
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest: java.io.IOException: com.jcraft.jsch.JSchException: verify: false
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at ftp.SFtpConnection.(SFtpConnection.java:105)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at ftp.ConnectionManager.getConnection(ConnectionManager.java:320)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at ftp.FtpVFS.getConnection(FtpVFS.java:451)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at ftp.FtpVFS._canonPath(FtpVFS.java:194)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at org.gjt.sp.jedit.browser.BrowserIORequest.listDirectory(BrowserIORequest.java:158)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at org.gjt.sp.jedit.browser.BrowserIORequest.run(BrowserIORequest.java:90)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at org.gjt.sp.util.WorkThread.doRequest(WorkThread.java:213)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at org.gjt.sp.util.WorkThread.doRequests(WorkThread.java:180)
8:39:59 AM [jEdit I/O #3] [error] BrowserIORequest:  at org.gjt.sp.util.WorkThread.run(WorkThread.java:154)
8:39:59 AM [jEdit I/O #3] [error] ErrorListDialog$ErrorEntry: sftp://hartter@onbahaspa:22/~/:
8:39:59 AM [jEdit I/O #3] [error] ErrorListDialog$ErrorEntry: Cannot list directory: java.io.IOException: com.jcraft.jsch.JSchException: verify: false
The verification fails. I had written to the OpenIndiana Mailinglist and 
the successful suggestion from Onno Molenkamp was, to remove the 
sun.security.pkcs11.SunPKCS11 provider from /usr/jdk/latest/jre/lib/security/java.security

After removing and renumbering the providers there, the SFTP Plugin works again in OpenIndiana. 

 

Sep 16, 2010

Upgraded to OpenIndiana 147 from OpenSolaris Build 134

Now I upgraded from OpenSolaris Build 134 to OpenIndiana Build 147.

Because just testing in VirtualBox fails. The ISO boots and I can install the
system, but during the startup from disk, the kernel likes to update the
microcode and fails.

The upgrade works as usual see the Wiki for more information.

I had the same problem as describe in the wiki with the download of
thunderbird.

Just uninstalled thunderbird and added the strange hostname
 ~0~pkg-origin.openindiana.org with an IP from nslookup to my local hosts files.

After this the upgrade worked with some repetitions because of Error 503 Service unavailble.

After restarting, I didn't get my german keyboard layout.

After searching, I rememberd the fact, that Build 145 changed the keyboard
to be a SMF service.
See the BugIusse @ Illumos

# svccfg
svc:> select keymap:default
svc:/system/keymap:default> setprop keymap/layout = German
svc:/system/keymap:default> refresh
svc:/system/keymap:default> exit




Sep 10, 2010

Changing the message of a mercurial Patch before commiting into the repository

When using Mercurial Queues to work with patches, you can set a message,
when starting a patch.

When you like to change the commit message, when you qfinish the patch,
you have no chance currently.

The only way is to just push the patch you like.
Prepare a file or add the text directly to the qrefresh command.
hg qrefresh -l /path/to/msgfile
or
hg qrefresh -e "my commit text"

Sep 9, 2010

Python pygtk based MergeTool for Mercurial

I've found a merge tool named gpyfm, which is a rewrite of the old TeamWare file merge tool used by Sun Engineers.

Excerpt from the Homepage: 

gpyfm was originally written by David Marker (assitant ON Nevada gatekeeper). Richard Lowe, one of the project leads for SCM Migration.

Here is a picture from the tutorial of the merge tool:

Sep 8, 2010

Compiling Qt 4.5.3 on OpenSolaris

Because the tortoisehg  is changing to PyQt instead of PyGtk, I attempted to
compile the Qt Package from Nokia ( former Trolltech ).

I used the Sun Studio Compilers, but ran into a compile problem, where I had
to set an additional compile/link flag see here for the message.

The configure option I used was:

CC=/opt/sunstudio12.1/bin/cc CXX=/opt/sunstudio12.1/bin/CC CXXFLAGS="-library=stlport4" LDFLAGS="-library=stlport4" ./configure --prefix=/tools/qt/4.5.3 -opensource

After this, just a gmake and gmake install and everything was fine.

Sep 3, 2010

Moving source changes to a previous patch in Mercurial with Mercurial Queues ( MQ )

I started to use Mercurial Queues  to collect my changes in a patch.
Now I have more patches applied in my local repository.

When detecting an error, I changed the error in the code, but it belongs
to an earlier patch than the topmost which is now active.

To get the changes there, I do now

hg diff > /tmp/patch
hg qgoto targetpatch
hg import --no-commit /tmp/patch
hg qrefresh

This allows me to move source changes to a different patch .

Sep 2, 2010

Getting information in screen when something happens on an other shell

Some time ago, I started to use screen in my terminal instead of opening
multiple registers or gnome-terminals. See my starting entry here.

Now, I've running some simulation scripts which receive messages. When you
are on a different  shell in screen and something happens on an other screen-shell, you can get notified by a special key sequence.

Go the the shell in screen, where you like to receive notifications and press
-


After this you get a hint, when something changes in the output.