Internet of Things (IoT) devices are ubiquitous, yet they often present security issues. The research community has invested substantial effort in designing automated methods for identifying these devices through passive network analysis—an essential step in security applications such as anomaly detection, traffic monitoring, and vulnerability scanning. However, despite the promising results reported in laboratory settings, the effectiveness of these methods under realistic conditions remains unclear. In this work, we systematically review the existing literature on IoT device identification by studying the approaches, features, and evaluation environments. We then design and implement a framework to reproduce and evaluate selected identification methods. We re-implement the selected methods and assess their performance, using our framework, under realistic environmental factors, such as non-IoT traffic, dynamic user activity, and unknown devices. Our study reveals several important insights. We demonstrate that the performances of current identification methods significantly decline under realistic conditions. Furthermore, we highlight these methods’ inability to differentiate between known and unknown devices, raising concerns about their effectiveness in security applications such as anomaly detection. We conclude by providing actionable recommendations for future research.