My only knowledge from the M5 is from the streets and my only knowledge from the Z8 is from the track as a passanger so I am not in a possition to discuss in detalil the feelings in the car, but...
The M5 is heavy and have a tendency to understeer in low speed corners, the Z8 is much more direct and goes more towards oversteer in the same corner. I could not notice any irritating body flex on my few laps as I have noticed in other convertibles.
If the definition of handling is, going from one point to another, conected with turns bumps and straights its no question that the Z8 will outrun an M5. That has been proven by the most serious German Auto mag. Sport auto.
( test made by same driver and condition)
The Z8 was 6 sec faster than the M5.(8,28 vs 8,22 on the Nurdburg ring)
This off course also has to do with the fact that the Z8 is a faster car.
If the definition of handling is the speed it can carry around a curve the M5 could be equal or better.
Looking at the test that was made by Sport auto on Nurdburg ring I see that in some curves the M5 was faster and on others the Z8 was faster.
The Z8 was never made to beat the 360 Modena or its like, nor was the M5 made to be a sportscar, but given the right driver it can scare the **** of any of them anytime....
They are quite simply fantastic cars in its own way....
This cars are much much more than just fast cars, if you wanted a BMW that was fast on a track you should buy an AC Schnitzer M Coupe that would drive cirkles around an M5 and the Z8 on a track. ( well.... almost
)