Hidden failures in protection system are great threats to the operation of power grids， while there are almost no on- line diagnostic methods for hidden failures so far. To solve this problem， this paper proposes a new method for exploring thehidden failures in protection system. By comparing the calculated values from measurements of wide-area measurement system （WAMS） and from relay protection， this method realizes the real-time monitoring of static characteristics and diagnosis ofhidden failures in protection system. The criterion for failure diagnosis is also developed. As an example， for an optical-fiberlongitudinal differential protection of transmission lines， the setting method for hidden failure criterion's threshold is analyzed.