When people have problems they will post them on the forum. It is not an indicator of the car's reliability. No one posts thread entitled "No problems so far after 4 years, what should I do??"
Having said that, browsing the forum does tell you what kind of problems you might encounter if you do encounter some (but it won't tell you frequency). M5's will be expensive to fix (and maintain and own) if something breaks, but so will M3s and a GT-R. However, they are some of the most reliable cars in their performance categories, so you can't go wrong with either. Forget about GT-Rs transmission issues, it has been taken care of.
In other words, none of those cars have enough problems for reliability to be your deciding factor between them. Get an extended warranty and don't worry about it.
You should choose based on what you like and what you need (maybe not 'need', more like 'want' - none of us NEED 500hp).
Test drive all three and pick your favorite.
- If speed is your top priority than GT-R is hands down the winner. It has insane torque anywhere on the rev counter, and is an absolute blast to drive.
- If you want to have fun in a lighter car with a manual transmission and RWD than M3 is your pick. M3 would be a blast on the track.
- And if you want a 4 door beast of all trades that does everything well (but does trade some driving dynamics for 4000lbs of luxury and comfort) than M5 is your answer.
I would also decide based on how good your dealers are around you. The Audi service by my house has always found excuses not to do something under warranty, while my BMW dealer always went above and beyond to help me. This is more important to me than reliability. Even if something does die on my M5, I have a piece of mind that I will be treated fairly and that it will be fixed correctly. Nissan dealers fail at this sometimes, but it's on a case by case basis, you can't generalize.