The system starts out as a big cloud of debris orbiting a central point. Each particle is orbiting in its own way, there isn’t any organization. Since there is no organization, most of these particles are going to bump into each other. This changes the speed and direction of the particles, some of them will clump together and most of them ending up gathered at the center of gravity of the entire system. This eventually gets large enough to become your star.

As the remaining particles become clumped together into larger items they will begin exerting gravitational pull on other objects. One plane of orbit will become dominant, and the remaining clumps will gradually settle down into their orbits, in the same general plane, in the same direction. Those that don’t will have their orbital speed reduced enough that the object ends up gathered into the center.