Transmission gears lock together with a series of "dogs” or "dogs and slots”.
Under hard use (such as full throttle clutch-less upshifts - or even sloppy shifting) the gears can begin to separate until the dogs become disengaged, which causes the transmission to jump out of gear. (usually second goes, then fifth)
Undercutting a transmission involves machining angles on the engagement dogs and slots to prevent the transmission from jumping out of gear under load.
See picture below...(courtesy of Jim Hiler)
This can be done by a patient backyard mechanic with a Dremel and steady hands, but it is highly recommended to have a professional do it for you. Dave Dodge offers this service.