So you are interested in Reverse Engineered the PS Jailbreak exploit and know more about the inner workings of PS3? Thanks to phire (aka phiren on EFnet), Matt_P, subdub and others, you can now read more on the theory of this exploit at this wiki page. The wiki page, PS3wiki.lan.st, has been fully updated shortly after the release of immensely popular PSGroove. Here are the intro:
The PSJailbreak dongle is a modchip for the PlayStation3 that allows users to backup and play games off the harddrive. Unlike the modchips of the Previous generation, or the modchips so far for the Xbox360 and Wii, this modchip simply plugs into the USB port on the front of the PS3, avoiding the need for complex soldering and voiding of your warranty.
As the time of writing this document, the final PSJailbreak has not been released, but a number of samples were given out and at least one fell into the hands of someone who owned a USB sniffer. This analysis of the exploit is based on those USB sniffer logs, issues encountered during the development of the opensource PSGroove version of the exploit and a number of educated guesses. It will probably be updated as new information comes in.
The initial analysis by gamefreax.de suggested that it was a Stack overflow attack. After further analist it turns out that this exploit is a Heap Overflow attack. The exploit carefully manipulates the heap by plugging and unplugging fake usb devices with large device descriptors until the device on port 4 which misreports its size to overwrite one of malloc’s boundary tags.