During my first week at my new job, I had the opportunity to teach some of my new coworkers about gdb
breakpoint commands and conditional breakpoints. I had a lot of fun teaching these techniques my friends here and thought others might find the story enjoyable as well.
Breakpoint commands
The first question I had was: where is our server doing reads? To answer this question, I used an often overlooked feature of gdb
: breakpoint commands. At a high level, these allowed us to run an arbitrary set of gdb
command automatically when a break point is hit. In my case, I wanted to see what stack trace was causing the reads:
(gdb) break read
Breakpoint 2 at 0x7ffff7382960: file ../sysdeps/unix/syscall-template.S, line 82.
(gdb) commands
End with a line saying just "end".
>backtrace
>continue
>end
(gdb)
I put a breakpoint on the libc
read
function call and automatically do two things: print the backtrace of the thread that hit the read and then continue execution. The overall effect of this is that gdb
runs the program as normal but prints a backtrace every time the program reads:
Breakpoint 1, read () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
#0 read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007ffff7324ed8 in _IO_new_file_underflow (fp=0x7ffff76386c0)
at fileops.c:606
#2 0x00007ffff73265be in _IO_default_uflow (fp=0x0) at genops.c:440
#3 0x00007ffff731da9b in _IO_getc (fp=0x7ffff76386c0) at getc.c:41
#4 0x00007ffff7b6e63d in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::underflow() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff7b58c17 in std::istream::sentry::sentry(std::istream&, bool) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff7b59b4b in std::istream::operator>>(int&) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x0000000000400905 in main ()
Conditional breakpoints
I also noticed it was opening an interesting file, so I wanted break into a debugger to inspect the program to figure out why. Unfortunately, it opens a lot of files (logs, etc), so I needed a way to filter out only the interesting calls to open
. To do this, I used gdb
conditional breakpoints. The example below creates a breakpoint on open
that only triggers if /home/alex
is a substring in the filename:
(gdb) break open if strstr($rdi, "/home/alex")
Breakpoint 1 at 0x7ffff7382770: file ../sysdeps/unix/syscall-template.S, line 82.
(gdb) continue
Breakpoint 1, open64 () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) x/s $rdi
0x400b7c: "/home/alex/a.out"
This example takes advantage of two more pieces of gdb
functionality:
gdb
can call an arbitrary function (in this case, we call thelibc
strstr
function to compute substring).gdb
can directly access the values of register and then cast them to common C types (in this case,$rdi
is the first argument, which we know is achar *
)
Putting it all together
We can use these tricks together to get ad-hoc dynamic tracing of our server!
This logs the filename and the stack trace from any open call that isn't to our log directory, but otherwise runs the server as normal:
(gdb) break open if !strstr($rdi, "/var/log")
(gdb) commands
End with a line saying just "end".
>print/s $rdi
>backtrace
>continue
>end
(gdb) continue
Comments !