Yikes! Definitely don't do that.
Everyone's opinion is different, but I say buy a home that you won't feel financially stressed living in. As DINKs, I'm assuming you want to also live life and do fun things, like go on cool vacations, eat at nice restaurants, go to concerts, sporting events...