This paper proposes an improved Phillips-Heffron model with the single-machine infinite-bus（SMIB） power system and the multi-machine power system including static synchronous series compensator（SSSC）. Based on this improved model， the mechanism of damping low frequency oscillation by the SSSC in SMIB power system and multi-machine power system is analyzed. By renewing the control variables， the affine nonlinear equation of SMIB power system including SSSC is obtained. Combining optimal control theory， the nonlinear transformation and nonlinear feedback theory are used to obtain the nonlinear optimal control law of SSSC. Finally， the electromagnetic transient model of SMIB power system with SSSC is established in RTDS to prove the feasibility of the proposed nonlinear optimal control law in damping out power system low frequency oscillations.