Based on least square matrix pencil algorithm, a novel distance protection algorithm in frequency domain for single-phase-to-earth fault of transmission line is proposed, which can accurately calculate the fault distance when faults happen at the remote terminal, and get a positive fault distance error when faults happen around the remote terminal. Assuming that the impedance of zero phase sequence loop after the fault point is pure inductive, a linear distance measure equation with three unknown coefficients is deduced. Simulation results with electromagnetic transient program(EMTP) show that for concentrate parameter model of short-distance transmission lines, this algorithm can measure distance accurately when faults happen at the remote terminal. It can effectively prevent the overreach phenomenon when faults happen around the remote terminal through transient resistance.