I think a few people are missing the point - not everybody's wife or girlfriend is as into it as we are and my wife wouldn't dare go in and buy a bike by herself. My brief for buying hers was it simply had to be comfortable and pretty because she'd know full well I'd worry about all the other stuff. (For the record, we went with a '08 Spesh Myka FSR Expert) But I digress...
To the OP, I tend to ride my SS when riding with my wife, not because it makes me any slower but it does make those rides more of a workout and I am less likely to go tanking off too far ahead without realising.
That said, I can't always win, I've told off when on the geared full suss for riding 'with her' in the climbs, yet on she's moaned about me taking the Single because I'm forced to go bombing off on the climbs to stay on top of the gear and she only gets to see me at the top of every hill.
As for buying a downgraded bike cause you might feel silly, nonsense. However, using it as an excuse to get something a bit niche (SS or CX) is fine