There are other cities booming (Houston, TX) and there is plenty of affordable housing in non-ghetto areas.
There's really no better place to work than SV career-wise, so it isn't a simple question of wanting to live in the area. You're certainly not living in SV for the culture or local attractions.