During my first week at my new job, I had the opportunity to teach some of my new coworkers about
gdb breakpoint commands and conditional breakpoints. I had a lot of fun teaching these techniques my friends here and thought others might find the story enjoyable as well.
The first question I had was: where is our server doing reads? To answer this question, I used an often overlooked feature of
gdb: breakpoint commands. At a high level, these allowed us to run an arbitrary set of
gdb command automatically when a break point is hit. In my case, I wanted to see what stack trace was causing the reads:
(gdb) break read Breakpoint 2 at 0x7ffff7382960: file ../sysdeps/unix/syscall-template.S, line 82. (gdb) commands End with a line saying just "end". >backtrace >continue >end (gdb)
I put a breakpoint on the
read function call and automatically do two things: print the backtrace of the thread that hit the read and then continue execution. The overall effect of this is that
gdb runs the program as normal but prints a backtrace every time the program reads:
Breakpoint 1, read () at ../sysdeps/unix/syscall-template.S:82 82 ../sysdeps/unix/syscall-template.S: No such file or directory. #0 read () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007ffff7324ed8 in _IO_new_file_underflow (fp=0x7ffff76386c0) at fileops.c:606 #2 0x00007ffff73265be in _IO_default_uflow (fp=0x0) at genops.c:440 #3 0x00007ffff731da9b in _IO_getc (fp=0x7ffff76386c0) at getc.c:41 #4 0x00007ffff7b6e63d in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::underflow() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff7b58c17 in std::istream::sentry::sentry(std::istream&, bool) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007ffff7b59b4b in std::istream::operator>>(int&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x0000000000400905 in main ()
I also noticed it was opening an interesting file, so I wanted break into a debugger to inspect the program to figure out why. Unfortunately, it opens a lot of files (logs, etc), so I needed a way to filter out only the interesting calls to
open. To do this, I used
gdb conditional breakpoints. The example below creates a breakpoint on
open that only triggers if
/home/alex is a substring in the filename:
(gdb) break open if strstr($rdi, "/home/alex") Breakpoint 1 at 0x7ffff7382770: file ../sysdeps/unix/syscall-template.S, line 82. (gdb) continue Breakpoint 1, open64 () at ../sysdeps/unix/syscall-template.S:82 82 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) x/s $rdi 0x400b7c: "/home/alex/a.out"
This example takes advantage of two more pieces of
gdbcan call an arbitrary function (in this case, we call the
strstrfunction to compute substring).
gdbcan directly access the values of register and then cast them to common C types (in this case,
$rdiis the first argument, which we know is a
Putting it all together
We can use these tricks together to get ad-hoc dynamic tracing of our server!
This logs the filename and the stack trace from any open call that isn't to our log directory, but otherwise runs the server as normal:
(gdb) break open if !strstr($rdi, "/var/log") (gdb) commands End with a line saying just "end". >print/s $rdi >backtrace >continue >end (gdb) continue