Based on time domain simulation of serious faults using BPA program， this paper introduces the research on setting and coordinating of out-of-step separation system in China southern power grid. The performance of the out-of-step system and the stability of the electrical isolated islands after separation are analyzed， especially considering the impact of under-voltage load shedding and different separation location. The simulation result waves are played back to the out-of-step separation devices by new developed instrumentation to verify the domain analysis results. For this reason， a real-time closed-loop test system is established， which is composed of real-time digital simulation model（RTDS） for AC/DC hybrid power systems of China southern power grid， out-of-step separation system， power amplifier and DC control protection devices. Finally， a suggested setting modification for current out-of-step separation system is given.