As current research methods for determining the optimal routine test interval of the relay protection system are all based on a single indicator, the routine test interval thus determined may not necessarily be optimal. By taking into account both the economic cost and relay reliability and with the dual-redundant relay protection system as the object of research, the corresponding Markov state space model is developed and the expression of annual economic losses is derived. Two new indicators of determining the optimal routine test interval are proposed, i.e. the reliable-coefficient of the relay protection system and reliable-coefficient of economic cost. The former considers the relay availability and reliability while the latter represents the relationship between relay reliability and economic losses. Then, the optimal routine test interval is determined by taking the minimum of reliable-coefficient of economic cost. Finally, the validity and effectiveness of the proposed method are proved by case studies.