The modulation of the DC converter theoretically based on switching function is deduced. By considering the whole process of harmonic transfer back and forth through DC converters in the AC side and DC side as a whole， a new criterion to determine whether the harmonic instability occurs is put forth from the perspective of the circuit. The ultra high voltage direct current（UHVDC） power transmission project from Nuozhadu to Guangdong is used as an example， an electromagnetic transient model is established. Then the frequency scanning analysis based on the proposed criterion and fault simulation verifications have been carried out. The study results indicate that the proposed criterion can assess the risk of harmonic instability at sending end of HVDC power transmission system.