Over at Archaic Christianity you can find an old answer to the perennial arguement:
Now Joseph refused, saying "I have sons and I am an old dude, and she is a young woman. This will make me a mockery among the sons of Israel.
This is allegedly Joseph in response to the idea that he should marry Mary. I've no idea of the accuracy. It's from the Protoevangelium of James.