I wish beauty mattered less too. Lots of physicists are very attractive :). I worked at CERN a while ago and I think the women there were definitely pretty.

I remember reading that there is a strong correlation between intelligence and beauty (since life is very unfair). For generations, apparently, the smartest men married the prettiest women and privilege/wealth correlate pretty highly with both intelligence and beauty.