blog/content/2014/03/ctf-armeb-debugging.html

---
title: "CTF + ARMeb + debugging"
description: >
  CTF + ARMeb + debugging
posted: !!timestamp '2014-03-05'
created: !!timestamp '2014-03-05'
time: 7:21 PM
tags:
  - ctf
  - debugging
  - arm
---

I've been working on making the AVILA board work again with FreeBSD.
Thanks to Jim from Netgate for sending me a board to do this work.

I still have a pending patch waiting to go through bde to fix an
unaligned off_t store which gets things farther, but with the patch I'm
getting a: `panic: vm_page_alloc: page 0xc0805db0 is wired` shortly after
the machine launches the daemons.

I did work to get cross gdb working for armeb (committed in r261787 and
r261788), but that didn't help as there is no kernel gdb support on
armeb.  As I'm doing this debugging over the network, I can't dump a
core.

I didn't feel like hand decoding a `struct vm_page`, so I thought of other
methods, and one way is to use CTF to parse the data type and decode the
data.  I know python and ctypes, so I decided to wrap libctf and see
what I could do.

Getting the initial python wrapper working was easy, but my initial test
data was the kernel on my amd64 box that I am developing on.  Now I
needed to use real armeb CTF data.  I point it to my kernel, and I get:
"`File uses more recent ELF version than libctf`".  Ok, extract the CTF
data from the kernel (ctf data is stored in a section named `.SUNW_ctf`)
and work on that directly:
```
$ objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /dev/null
objcopy: /tftpboot/kernel.avila.avila: File format not recognized
```

Well, ok, that's not too surprising since it's an ARMEB binary, lets try:
```
$ /usr/obj/arm.armeb/usr/src.avila/tmp/usr/bin/objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /tmp/test.avila.ctf     
$ ls -l /tmp/test.avila.ctf 
-rwxr-xr-x  1 jmg  wheel  0 Mar  5 17:59 /tmp/test.avila.ctf
```

Hmm, that didn't work too well, ok, lets just use dd to extract the data
using info from `objdump -x`.

Ok, now that I've done that, I get:
```
ValueError: '/tmp/avila.ctf': File is not in CTF or ELF format
```

Hmm, why is that?  Well, it turns out that the endian of the CTF data
is wrong.  The magic is `cf f1`, but the magic on amd64 is `f1 cf`, it's
endian swapped.  That's annoying.  After spending some time trying to
build an cross shared version of libctf, I find that it has the same
issue.

After a bit of looking around, I discover the CTF can only ever read
native endianness, but `ctfmerge` has a magic option that will write out
endian swapped data if necessary depending upon the ELF file it's
putting in.  This means that the CTF data in an armeb object file will
be different depending upon the endian you compiled it on, so the object
file isn't cross compatible.  But, this does mean that the data in the
object files will be readable by libctf, just not the data written into
the kernel.

So, I create a sacrificial amd64 binary:
```
$ echo 'int main() {}' | cc -o /tmp/avila2.ctf -x c -
```

And use `ctfmerge` to put the data in it:
```
$ ctfmerge -L fldkj -o /tmp/avila2.ctf /usr/obj/arm.armeb/usr/src.avila/sys/AVILA/*.o
```

and again use `dd` to extract the `.SUNW_ctf` section into a separate file.

With all this work, I finally have the CTF data in a format that libctf
can parse, so, I try to parse some data.  Now the interesting thing is
that the CTF data does encode sizes of integers, but it uses the native
arch's pointer sizes for `CTF_K_POINTER` types, which means that pointers
appear to be 8 bytes in size instead of the correct 4 bytes.  A little
more hacking on the ctf.py script to force all pointers to be 4 bytes,
and a little help to convert ddb output to a string and finally, I have
a dump of the <code>struct&nbsp;vm_page</code> that I was trying to get all along:
```
{'act_count': '\x00',
 'aflags': '\x00',
 'busy_lock': 1,
 'dirty': '\xff',
 'flags': 0,
 'hold_count': 0,
 'listq': {'tqe_next': 0xc0805e00, 'tqe_prev': 0xc06d18a0},
 'md': {'pv_kva': 3235856384,
        'pv_list': {'tqh_first': 0x0, 'tqh_last': 0xc0805de0},
        'pv_memattr': '\x00',
        'pvh_attrs': 0},
 'object': 0xc06d1878,
 'oflags': '\x04',
 'order': '\t',
 'phys_addr': 17776640,
 'pindex': 3572,
 'plinks': {'memguard': {'p': 0, 'v': 3228376932},
            'q': {'tqe_next': 0x0, 'tqe_prev': 0xc06d1f64},
            's': {'pv': 0xc06d1f64, 'ss': {'sle_next': 0x0}}},
 'pool': '\x00',
 'queue': '\xff',
 'segind': '\x01',
 'valid': '\xff',
 'wire_count': 1}
```

So, the above was produced w/ the final [ctf.py](https://www.funkthat.com/~jmg/ctf.py) script.