ABSTRACT Nurul Jadid University is a university that implements an Islamic boarding school system. One of the existing regulations at Nurul Jadid University is that it is not allowed for men and women to communicate and interact directly. However, security at Nurul Jadid University still has difficulties in monitoring and controlling this. In this case, a tool or media is needed to identify interactions between men and women so that they are known to break the rules at Nurul Jadid University. The method used in this research is Faster Region Convolutional Neural Network (R-CNN). Faster Region Convolutional Neural Network (R-CNN) is part of a deep learning architecture. The Faster Region Convolutional Neural Network (R-CNN) method can produce automatic detection between men and women through video objects on CCTV. The test results of the Faster R-CNN method used succeeded in detecting video data interacting between men and women. The level of accuracy produced by the video testing process is 89% with 99% precision and 87% recall results. Keywords: Detect, Interaction, Faster R-CNN