I was blown away when the E60 M5 first came out in the US. The performance and amount of tech/engineering was pretty mind-blowing. I was in my mid 20s at the time and could never even dream of affording it brand new. Having one now, years later, still feels special.
The E46 M3 is the most iconic M3 from BMW. IMO it's what set the standard stage for all subsequent M cars BMW produced. So since BMW enthusiasts will still buy E46s, the price is still locked in that $16-25k range up until now.
As far as the E60 M5, sure it may not have the "legend" price tag to go with it nowadays, but the maintenance cost is not practical for most people.