Castles in the sky

Is mankind damned to remember the palaces erected on behalf of the nobles and rich, not the many workers that built those things?