Before the Commonwealth existed, there was an expanding, multi-galaxy civilization referred to by its descendants/survivors simply as “First Civ”. Due to the combined effects of a too-aggressive policy of expansion, civil unrest, the inequality and abuse of the classes, and the categorizing of augmented humans as a slave class, First Civ disintegrated.
The period of darkness and barbarism that followed is referred to as the Downfall. Various groups of people fled First Civ as they became endangered or more powerful people tried to have them classified as mutants or non-humans, and either sterilized or made them into slaves. Among them were the Khybors, the ancestors of the Leapers.
Some groups of people managed to get hold of ships and flee to distant galaxies.
Learn more about the entire series: