Video based target detection is an important research content in intelligent video surveillance, which extracts foreground objects from background images in video sequences. Video based target detection has developed rapidly in recent years. In practical applications, however, detection of small and medium-sized objects in video remains a challenging task as small and medium-sized objects occupy too few pixels, and the obtained information is limited. The demands in aerospace, criminal investigation, face recognition, intelligent transportation and other fields have proved the research value of video based small target detection. This paper first briefly introduces the traditional video based target detection algorithms and the improvements for small target detection. Second, deep learning based models for small target detection in video are summarized in detail, which are categorized into one-stage models and two-stage models according to the detection stages. The network structures and plug-in modules for video based small target detection are also explained. In addition, this paper summarizes the common databases with evaluation criteria. Finally, applications and future research direction in this area are analyzed.