A new algorithm for evaluating the transformation matrices as smooth functions of frequency is presented. First， the transformation matrix at each frequency point is calculated using the conventional algorithm， then the discontinuities in the transformation matrices are eliminated by a special smoothing procedure. The Newton-Raphson（NR） algorithm is implemented and compared with the new algorithm by a double-circuit three-phase untransposed transmission line example. Experimental results and theoretical analysis demonstrate that the new algorithm is more efficient than the NR algorithm and has better tracing ability. Furthermore， it is more flexible and adaptable to more constraints. Experiments show that more smooth transformation matrices can be obtained with the new method using improved constraints.