Based on an unbalance current protection principle for AC filter used in DC transmission system， some shortcomings on the existing principles are pointed out. A new principle suitable for the unbalance current protection is proposed， in which the ratio between the unbalance current and the traversing current flowing through AC filter is used as an unbalance current protection criterion for AC filter. The calculation formula for the unbalance protection setting value is deduced. Based on the concrete parameters of the Gaoling BTB HVDC project， the detailed simulation was completed with electromagnetic transient calculation software EMTDC/PSCAD in order to prove rationality and validity of the new principle. In addition， some useful suggestions are described for implementation and application of the unbalance protection. This method has been put into successful operation in Gaoling BTB HVDC project now， which is in favor of HVDC system design and operation on site.