Problems Getting The Debugger To Work With Nintendo 64

[flagrama]

I do a fair bit of for-fun software development and reverse engineering on the Nintendo 64 hardware and software. Debugging is always an issue though because only a couple of Windows-only emulators have fleshed out working debuggers. On the other hand, the ares emulator has a gdbstub which works pretty well for debugging your own code, but isn't quite as useful if your code is a modification to an existing binary.

I already use Ghidra for documenting these existing binaries, so I figured using it's debugger with ares' gdbstub would be pretty easy and would let me reference my documentation easier. So far it has been difficult to get the debugger to work properly in the first place.

First, I don't know if this is relevant, but I use the N64LoaderWV extension to import the ROMs and give a decent starting point for documentation. Maybe it is missing information that it should be providing to make this work easier.

Second, when I run the debugger normally I don't get any display in the dynamic view, memory view, regions, etc. Instead in the terminal I get overflow exceptions about converting a negative signed integer to a unsigned integer. This seems pretty obvious to me, it's a sign-extension issue. The PC and general purpose registers of the N64 CPU are all 64-bit sign extended, so for example the beginning of RDRAM at 0x80000000 will be stored as 0xFFFFFFFF80000000 when in the PC or a GPR.

Furthermore, if I manually edit commands.py and mask these addresses with 0xFFFFFFFF the debugger progresses much further. I'm able to actually see values in the memory view, the regions tab has a region at least. However this does not yet solve all my issues. gdb is still quite unhappy being unable to find the beginning of functions. Trying to step through code seems to increment randomly rather than 4 bytes at a time. Specifically it seems to run until a branch or delay slot, I didn't pay super close attention to which. And trying to set breakpoints fails. In the dynamic view they fail saying that the value 0xFFFFFFFF80xxxxxx is out of range. Trying to set them elsewhere, like in my listing with all the documented code it generally says something about not being in the tracepath or the same snap or something.

So I'm wondering what exactly I have to do to improve the situation from here.

Does the N64 loader need to provide NEC VR4300 specific data files? Do data files need to be added to Ghidra's source somehow? Is there something missing in existing data files? Something entirely different?

I'd appreciate any help I can get with this as I believe it would help my workflow immensely.

[d-millar]

@flagrama Just wanted to let you know that your request has not been ignored - there already may be a fix in-play, but I need to verify that. Will let you know when I know.

[d-millar]

@flagrama Apologies for the delayed response - am trying to verify some of our fixes for related issues but having a difficult time getting gdb to work with the Ares emulator at all. Any tips regarding setup? (Am testing running “Sonic the Hedgehog”, but have no registers, no disassembly, and no ability to break.)

[flagrama]

The GDB server is intended for all systems to use eventually, but only the N64 core actually has implemented it currently. (ares-emulator/ares#1223)

[d-millar]

I have yet to find an N64 rom that works "correctly" in gdb proper, so am considering closing this issue until the emulator is in a more stable state. Objections?