System islanding is a very important measure for protecting power systems from wide-area blackouts. It takes at least one oscillation cycle for conventional islanding control devices to detect system out-of-step oscillation conditions， which cannot fully meet the system stability control demands of modern power systems. This paper analyses the trajectory of P/Q at their maximum points when power system is under out-of-step oscillation conditions， and gets the method for detecting critical characteristic point of out-of-step oscillation. Then a new criterion is presented to detect the critical characteristic point of out-of-step oscillation using P-Q-θ trajectory， which can detect out-of-step oscillation quickly and accurately.