Is it usually the fastest car on the chunk of road you are on within a few miles? Most of the time yes.
Are there faster cars around town that could get to the next stop light faster than you? These days, many times yes.
Do these other cars have the comfort, luxury amendments, refinement, build quality, and command respect from most car enthusiasts? Usually not.
Factor in that the e39 M5 is a luxury sedan with power and handling. Try to think of an equivalent from a Japanese company and im sure you wont find one on the same level (especially one from the early 2000s). Then compare the other German competitors like Audi and Mercedez. Talk to any previous e39 M5 owner that has left to go to these brands and they will, more times than none say that it was a mistake to leave for those brands.
If you understand that your car is not, and never will be the fastest on the road, then you can start focusing on all the good things that this car has to offer. Then, the next time you are on the freeway and drop into 4th and have fun with other purpose built sports cars (not luxury sedans) and find that you can keep up or sometimes walk over then, you will know why you chose this car.