Here's my $.02 and its what I've learned from this site and others........so I don't have first hand knowledge of a 5" exhaust.
If you're not going crazy with hp, a 4" exhaust will suit you just fine. The sound of a 5" is certainly different and a lot of people like it. I haven't heard one yet, so I dunno.
3.5" or 4" downpipe? I just put a 3.5" in my 05 and it was a TIGHT fit. If you want a 4" dp, there's a 90% chance you're going to have to mod your firewall to make clearance. To me, it wasn't worth the time/effort to make clearance for the 4" dp and my understanding is that the gains will be minimal. Gains being better flow and lower EGT's.
About your noise and keeping your wife happy---I'd recommend checking out MBRP
exhausts. They give you the option of deleting your cat and your muffler. The pipes are $40-$50 and if its too loud, you can swap your cat, your muffler, or both back in.......in about 30 min with hand tools. I'm running the cat delete (MBRP
calls it a "cat test" pipe) and the muffler delete and have been very happy with the way my truck sounds. Not too loud at idle, some nice turbo whistle, but if you get on it, it really rumbles.