I have two kids and an ML430. Before this, we had a Jeep Grand Cherokee and a Tahoe.
I'm buying another SUV this summer and while I would love an X5, there is just no cargo room! This is the biggest problem I see, but probably the reason it performs so well. My choices are really the ML500, the MDX, or maybe the new twin turbo Volvo XC90 (bigger than X5).
If you don't buy too much stuff on those trips to Costco, I'd buy the X5. Personally, I fill up the back of our ML all the time and sometimes wish for more room than even it has.
I suppose there are many tradeoffs in vehicles, so just be sure "cargo space" is not on your top 5 list of "must haves", and then go out and get that X5! I hear it's great.